Skip to content

Configuration & Hardware Guide

Jump to bottom
Justus Brugman edited this page Aug 25, 2026 · 4 revisions

Configuration

All default configuration values now come from bundled application.config, not hard-coded Java defaults.

By default, model.chat and model.validator are left blank. For an OpenAI-compatible backend the app then omits the model field, so LM Studio, Jan.ai, or another compatible backend can use its currently loaded, selected, or default model automatically. A managed MLX server instead uses backend.mlx.modelPath as the request model automatically.

backend.type=openai-compatible is the default and sends requests to backend.http.url; it works with LM Studio, Jan, Ollama, OpenAI-compatible hosted APIs, and externally started model servers. Set backend.type=managed-llama-server to start a local llama-server process for GGUF, or backend.type=managed-mlx-server to start mlx_vlm.server for a local MLX model. Both managed servers bind only to 127.0.0.1 and stop when the application closes.

For example:

backend.type=managed-llama-server
backend.llama.command=llama-server
backend.llama.modelPath=./models/Gemma4-26B-A4B-QAT.gguf
backend.llama.port=0
backend.llama.startupTimeoutSeconds=120
backend.llama.arguments=--ctx-size 32768 --ctx-checkpoints 32 --parallel 3 --threads 6 --threads-batch 6 --batch-size 1024 --ubatch-size 512 --kv-offload --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --reasoning off

For a local MLX model on Apple Silicon:

backend.type=managed-mlx-server
backend.mlx.command=python3
backend.mlx.modelPath=./model/gemma-4-26b-a4b-it-4bit
backend.mlx.port=0
backend.mlx.startupTimeoutSeconds=180
backend.mlx.arguments=-m mlx_vlm.server --max-kv-size 32768 --max-tokens 32768

Install mlx-vlm for the configured Python interpreter and ensure python3 is available on PATH. The managed MLX server is intended for local use and uses its OpenAI-compatible /v1/chat/completions endpoint.

The bundled defaults do not select a managed-server command, model, or model-specific arguments. Configure those values in a local application.config or start from the examples in configs.example/llama_server and configs.example/mlx_server. An MoE model's active expert count is encoded by the model itself; --n-cpu-moe means something different and would keep MoE layers on the CPU.

Legacy local overrides using lmstudio.url remain accepted and map to backend.http.url.

Configuration is now split into:

Important settings:

  • chat.maxRecentTurns=2
  • recentSummary.maxRecentTurns=12
  • recentSummary.batchMessages=6
  • summary.batchMessages=10
  • canonicalState.batchMessages=2
  • cacheBuster.interval=5 (0 disables periodic cache busters)
  • validation.enabled=true
  • validation.outputMode=auto requests JSON Schema constrained validation when supported and retries as text when the endpoint rejects it; the rejection is remembered for the active session
  • resilience.chat.failureThreshold=3
  • resilience.chat.cooldownSeconds=20
  • resilience.validation.failureThreshold=2
  • resilience.validation.cooldownSeconds=15
  • resilience.background.failureThreshold=2
  • resilience.background.cooldownSeconds=60

If a model behaves badly with the rules engine, disable it with:

validation.enabled=false

When validation is disabled, rules.md is skipped and the raw model answer is returned directly. When validation is enabled, the validator returns an ALLOW or REPLACE decision. By default its request includes an OpenAI-compatible JSON Schema response format so supporting backends constrain that decision during generation. The schema is deliberately used only for validation; story responses remain free text. Older compatible endpoints automatically use the existing tolerant text path when they reject structured output. Set validation.outputMode=text for a legacy endpoint, or json-schema to require constrained validator output.

The app also uses a small built-in resilience layer around LLM calls:

  • foreground chat calls use a short cooldown-based fail-fast guard
  • validation calls use their own guard and remain independent from the foreground response policy
  • background memory refresh calls use a more aggressive cooldown so repeated summary/state updates stop hammering an unavailable backend

Recommended Setup

  • Recommended reference machine: MacBook Pro with M1 Max and 64 GB unified memory
  • Better results usually come from a relatively large mature instruct model, such as google/gemma-4-26b-a4b in a 6-bit MLX quantization with reasoning disabled, using roughly 21.8 GB and a 32K context window
  • Minimum recommended class: Gemma 4 12B QAT or a roughly comparable model
  • If you have less memory available, the default google/gemma-4-26b-a4b-qat 4-bit variant is roughly 16 GB, which makes it practical to experiment with a 32K context window on a machine such as an M1 Mac with 32 GB of unified memory

Workable Setup

  • MacBook Pro with Apple Silicon (M-series) and 32 GB to 48 GB of unified memory
  • google/gemma-4-12b-qat, preferably the 4-bit variant, using roughly 8 GB of model memory
  • A 32K context window, using roughly another 8 GB of memory

The 12B dense model occasionally makes stylistic mistakes, but it is generally good enough to follow story rules and hard constraints. It is a practical model for experimenting with the storyteller app on a smaller machine. Including runtime, context, and general system overhead, total memory usage will usually end up around 18 GB to 20 GB, which is still workable on a 32 GB unified memory machine. The broader takeaway is that Apple Silicon with a moderate amount of unified memory is already a very capable platform for exploring local large language models.

Hardware Fit At A Glance

This table is a practical fit guide for local storyteller use with quantized models. The context ranges in the hardware columns refer to the model context window size used for this app (recommended values). It is meant as a quick "is this worth trying on my machine?" reference, not as a benchmark table or a hard compatibility guarantee.

Model Parameters RTX 3080 Ti 12 GB (16K) RTX 4090 24 GB (16K to 32K) Mac 32 GB unified (16K to 32K) Mac 64 GB unified (32K to 48K) Practical take
Qwen3-8B 8.2B Yes Yes Yes Yes Good entry-level choice
Llama-3.1-8B-Instruct 8B Yes Yes Yes Yes Safe and practical
Granite-3.1-8B-Instruct 8B Yes Yes Yes Yes Good compact alternative
Gemma-4-12B-QAT 12B Maybe Yes Yes Yes Strong option, but 12 GB VRAM is tight
Gemma-4-26B-A4B 26B A4B No Maybe Maybe Yes Very strong option on higher-memory Apple Silicon, but too heavy for smaller GPU setups

Interpretation:

  • Yes means the setup is generally workable for this app at the recommended context range for that machine.
  • Maybe means it can work, but it is more sensitive to quantization, runtime overhead, and context length.
  • No means it is usually not a practical match for this storyteller use case.

Note: On an Apple Silicon machine with enough unified memory, especially 64 GB, Gemma-4-26B-A4B will often not only produce better results, but will likely also respond noticeably faster than dense 8B or 12B models.

Practical guidance:

  • RTX 3080 Ti 12 GB: mainly suitable for 8B models, and borderline for some 12B setups.
  • RTX 4090 24 GB: a strong fit for 8B and 12B models at moderate context sizes.
  • Mac 32 GB unified: a good fit for 8B and 12B models.
  • Mac 64 GB unified: the most comfortable option here, with much more room for larger models and longer context windows.

Note

On the reference machine above, a larger local model such as Gemma 4 26B 6-bit typically uses about 32 to 36 GB of shared memory in practice:

  • about 22 GB for the model itself
  • about 8 to 12 GB for a 32K to 48K context window
  • about 2 GB for runtime overhead

These numbers are practical estimates, not hard guarantees. Actual memory use depends on the exact quantization, runtime, backend, and context length.

Performance & Memory Architecture (Apple Silicon / LM Studio)

Long-context local models, especially larger Mixture-of-Experts models such as Gemma 4 26B-A4B or Qwen 3 Coder 30B, trade context depth against memory pressure and desktop responsiveness. A 64 GB Apple Silicon machine can be a strong long-context setup, but there is no universal safe 65K configuration: model format, quantization, runtime, vision support, and the loaded context length all materially affect memory use.

Sequential derived-memory queue

A completed story turn can schedule three derived-memory refreshes: long-term summary, recent summary, and canonical state. These jobs are deliberately submitted to one daemon-backed DerivedMemoryTaskQueue and run sequentially.

  • This keeps derived-memory requests off the foreground story path.
  • It avoids the application itself starting three simultaneous background inference calls.
  • It reduces peak pressure on the backend, but does not override the backend's own parallelism or resource settings.

Periodic cache busting and drift

Long prompts can make a model less attentive to older instructions or world constraints. The app offers a portable best-effort mitigation: every cacheBuster.interval persisted story turns (default 5), it sends an internal reset request with a unique token prepended to the system prompt. The response is discarded; no extra message is shown to the user. Ctrl-U undo always sends the same reset-with-cache-buster request after removing the last turn.

This is not a documented KV-cache flush and does not guarantee that a backend discards a cache. It deliberately changes the prompt prefix so cache-sensitive OpenAI-compatible backends are less likely to reuse an exact stale prefix. Set cacheBuster.interval=0 to disable periodic requests.

LM Studio starting points for a 64 GB Mac

Use LM Studio's memory estimator before loading a model and verify the actual configuration after loading. The values below are starting points, not guarantees or application requirements. LM Studio supports configuring context length, evaluation batch size, Flash Attention, GPU offload, and model parallelism at load time; support varies by engine and model format.

Setting Suggested starting point Why
Context length Start at 32K; raise incrementally after testing Context length is a primary driver of KV-cache memory use.
GPU offload max when the estimator shows sufficient headroom Maximizes accelerator use; leave room for macOS and other applications.
Flash Attention Enabled when supported Can reduce attention memory use and improve speed on llama.cpp-based models.
Evaluation batch size 512, then tune Smaller values reduce prompt-ingestion peaks at the cost of throughput.
Parallel predictions 1 or 2 for tight memory budgets The Java queue serializes derived-memory work, but backend parallelism can still increase resource use.
KV-cache quantization Test the available option for the loaded engine/model It can substantially change memory use and quality; do not assume one quantization is optimal everywhere.

CPU-thread, physical-batch, and KV-cache settings differ across LM Studio engines and model formats. Prefer LM Studio's current model-specific controls and estimator over fixed thread counts or OS-level memory-limit overrides. See the LM Studio model-loading documentation and CLI resource estimator.

KV-cache quantization

For long contexts, KV-cache quantization is often one of the most effective memory-saving controls because KV-cache use grows with context length. It is distinct from the quantization of the model weights themselves.

  • Start with q8_0 for the K and V caches when it is available: it is usually the more conservative memory-versus-quality trade-off.
  • Try q4_0 for both caches when memory pressure or swapping remains a problem. It can reduce KV-cache memory substantially, but may affect output quality, especially for a particular model or very long context.
  • Keep Flash Attention enabled when the selected engine requires it for quantized K/V caches, and test a representative long story prompt after changing the setting.

These are backend load settings, not OpenAI-compatible request parameters, so Storyteller does not set them automatically. Jan's llama.cpp engine exposes q8_0 and q4_0 K/V-cache options for memory-constrained setups; LM Studio availability depends on the active engine and model format. See the Jan llama.cpp engine guide and LM Studio load configuration reference.

Thread allocation detail (Apple Silicon)

On an M1 Max, the CPU has eight performance cores and two efficiency cores. If LM Studio exposes a CPU-thread control for the active runtime, 8 is a sensible throughput-oriented starting point because it can keep the performance cores busy. 6 is a sensible responsiveness-oriented starting point when IntelliJ, a browser, or compilation should retain more CPU headroom.

Do not assume that 7 is universally the default or that powers of two are inherently optimal: the best value depends on the LM Studio engine, model format, GPU offload level, and what else the machine is doing. Measure generation speed and UI responsiveness with the target model, then keep the lowest value that gives the desired experience.

Clone this wiki locally