Configuration & Hardware Guide

Configuration System

Defaults are loaded from classpath resources inside src/main/resources/systemprompts/. You can override configuration and prompts at runtime by placing custom files in ./systemprompts/ or an ./application.config file next to the binary.

Example application.config setups:

Managed llama-server (GGUF)

backend.type=managed-llama-server
backend.llama.command=llama-server
backend.llama.modelPath=./models/Gemma4-26B-A4B-QAT.gguf
backend.llama.port=0
backend.llama.startupTimeoutSeconds=120
backend.llama.arguments=--ctx-size 32768 --ctx-checkpoints 32 --parallel 2 --threads 6 --threads-batch 6 --batch-size 1024 --ubatch-size 512 --kv-offload --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --reasoning off

Managed mlx_vlm.server (Apple Silicon MLX)

backend.type=managed-mlx-server
backend.mlx.command=python3
backend.mlx.modelPath=./model/gemma-4-26b-a4b-it-4bit
backend.mlx.port=0
backend.mlx.startupTimeoutSeconds=180
backend.mlx.arguments=-m mlx_vlm.server --max-kv-size 32768 --max-tokens 32768

Hardware Fit At A Glance

This table is a practical fit guide for local storyteller use with quantized models. The context ranges in the hardware columns refer to the model context window size used for this app (recommended values). It is meant as a quick "is this worth trying on my machine?" reference, not as a benchmark table or a hard compatibility guarantee.

Model	Parameters	RTX 3080 Ti 12 GB (16K)	RTX 4090 24 GB (16K to 32K)	Mac 32 GB unified (16K to 32K)	Mac 64 GB unified (32K to 48K)	Practical take
Qwen3-8B	8.2B	Yes	Yes	Yes	Yes	Good entry-level choice
Llama-3.1-8B-Instruct	8B	Yes	Yes	Yes	Yes	Safe and practical
Granite-3.1-8B-Instruct	8B	Yes	Yes	Yes	Yes	Good compact alternative
Gemma-4-12B-QAT	12B	Maybe	Yes	Yes	Yes	Strong option, but 12 GB VRAM is tight
Gemma-4-26B-A4B	26B A4B	No	Maybe	Maybe	Yes	Very strong option on higher-memory Apple Silicon, but too heavy for smaller GPU setups

Interpretation:

• Yes means the setup is generally workable for this app at the recommended context range for that machine.
• Maybe means it can work, but it is more sensitive to quantization, runtime overhead, and context length.
• No means it is usually not a practical match for this storyteller use case.

Note: On an Apple Silicon machine with enough unified memory, especially 64 GB, Gemma-4-26B-A4B will often not only produce better results, but will likely also respond noticeably faster than dense 8B or 12B models.

Practical guidance:

• RTX 3080 Ti 12 GB: mainly suitable for 8B models, and borderline for some 12B setups.
• RTX 4090 24 GB: a strong fit for 8B and 12B models at moderate context sizes.
• Mac 32 GB unified: a good fit for 8B and 12B models.
• Mac 64 GB unified: the most comfortable option here, with much more room for larger models and longer context windows.