LLM Setup

English · 简体中文 · 日本語

LLM Setup

The LLM turns source text into an annotated script and supports review and persona-related tasks. Qwen3-TTS makes the audio. They are separate systems: local TTS does not mean your annotation is local.

Return to Home · Before this: Installation · Next: First Audiobook

Start here for the three-sentence sample

If you already have LM Studio running on the same computer as Alexandria with a loaded instruction-following model, use this one connection path:

1. In Setup, choose LLM Location → Local.
2. Use Base URL http://localhost:1234/v1 only if it is the address your server actually uses. Enter local for API Key only when that server requires no authentication.
3. Refresh Model Name and select the exact model advertised by the running server. Keep How requests are sent → To the server address above and leave failover off.
4. Set Runs on this machine's GPU? to match the loaded model: shared GPU means Yes; CPU-only or another machine means No.
5. Click Test Connection, then Save Configuration. The checkpoint is a successful test and confirmed save. Continue with the original sample, then inspect every sentence.

This is a connection example verified against the application's controls, not a tested model/hardware recipe. The documentation update did not run an LLM or establish a memory minimum. If you have not chosen or loaded a model yet, use your server's model guidance; a model shown in the list is not proof it will annotate reliably. Docker needs an address reachable from inside the container, not this host-local example.

Other routes and detailed troubleshooting are below. You do not need to configure all of them.

Choose a route

Route	What you need	Where the text goes
Local API	An OpenAI-compatible server with a loaded instruction-following model, such as LM Studio or Ollama	To the server address you configure; use a genuinely local endpoint for local processing.
Remote API	A compatible network/hosted endpoint, its model ID, and any required API key	To that provider, including book text, prompts, and relevant cast/script context. Provider charges may apply.
Manual	A chat or script that can answer each prompt, plus your copy/paste work	Alexandria's manual LLM transport does not make HTTP requests for those prompts. You decide where to paste them; a cloud chat still receives the pasted text.

For all routes, use a model that follows structured-output instructions and test a short passage. Model size, reasoning settings, available context, and hardware needs vary. This guide does not promise a particular model's accuracy or speed.

Connect a local or remote API

Open 1. Setup and find LLM Settings (Script Generation).

1. Choose LLM Location:
   • Local for your local profile
   • Remote (Thunder / network) for your remote profile Each location keeps its own connection and provider settings. The label is not a privacy boundary: check the actual URL and any failover destination.
2. Set Base URL to the OpenAI-compatible API base. Common local examples in this project are:
   • LM Studio: http://localhost:1234/v1
   • Ollama: http://localhost:11434/v1 Start the server and load the model separately. Use the actual address/port of your installation.
3. Set API Key. The placeholder local is suitable only for a local server that does not require a real key. For an authenticated provider, use its key or the environment-variable option below.
4. Enter the exact Model Name served by that endpoint. The refresh icon beside the field lists models advertised by the server; select the intended model rather than guessing its display name or filename.
5. Expand Provider request options: transport, GPU sharing, reasoning, headers. Keep How requests are sent on To the server address above.
6. Set Runs on this machine's GPU? accurately. Choose Yes - shares the card with audio generation when the LLM uses the app host's GPU. Choose No - hosted API, CPU-only server, or another machine otherwise. Auto infers from the address, which may be wrong for a forwarded endpoint or a CPU-only local server.
7. Click Test Connection. It lists models and sends a small completion request, so a hosted endpoint may bill even this test. A successful result confirms that probe, not whole-book quality or a sufficient context window.
8. Click Save Configuration. Testing edited settings does not save or apply them. Switching the location also requires a save.
9. Upload a short book excerpt in Script, then use Test this book with the LLM or a short generation before a full run. Inspect the result using Scripts and Review.

Use a Base URL reachable from the machine running Alexandria. In Docker, localhost is the container; for a browser opened on another computer, it is still the Alexandria backend that must reach the LLM.

Keep keys out of configuration where possible

The API Key field accepts env:NAME, for example env:ALEXANDRIA_LLM_KEY. Set that environment variable in the process/container that launches Alexandria, then save the reference in Setup. The server resolves it when making requests.

A directly entered key is stored in the settings file even though the UI masks/redacts it. Protect that file and its backups. Do not publish keys, custom authorization headers, or unredacted diagnostic logs. An environment reference also needs the variable restored after a migration; copying config.json alone does not copy its secret value.

Manual copy/paste transport

Manual mode is useful without a supported API, or when you want to handle every model request yourself. It is labor-intensive for long books: one job can involve many prompts and retries.

1. In Setup, expand Provider request options: transport, GPU sharing, reasoning, headers.
2. Set How requests are sent to Manual - I supply every reply myself. Keep a valid Base URL value in the form if needed for saving; the manual pipeline does not use that address to send prompts.
3. Leave automatic failover off if you want the run to remain manual. Click Save Configuration. Skip Test Connection and Test this book with the LLM: these probes are not supported by the current manual workflow.
4. In Script, upload a short excerpt and click Generate Annotated Script to test the ordinary workflow. When it needs an answer, the Your turn panel appears.
5. Click Copy prompt. Send the complete prompt to the model/chat/script of your choice, including its formatting instructions.
6. Paste only the model's final answer into Model reply, then click Submit reply. Do not paste its thinking, a discussion of the task, or an unrelated conversation transcript.
7. Repeat for each request. The normal validation still runs; an unusable reply produces another prompt. Read the run's Plan: output to understand the work ahead.

If the browser blocks automatic copying, the Copy manually dialog lets you select and copy the text yourself.

Manual transport applies to LLM requests. It does not make external TTS local or prevent model-weight downloads. If you paste a prompt into a cloud service, its privacy and retention rules apply to that text.

Timeouts, retries, and optional failover

The profile exposes these controls:

• Request timeout (seconds): how long an individual request may take. Raising it can help a slow but functioning server; it cannot fix an incorrect URL or a model that never answers.
• Connect timeout (seconds): time allowed to establish the connection.
• Minimum interval (seconds): spacing between requests, useful for provider rate limits.
• API retries and the backoff fields: how failures are retried and how the delay grows.
• When retries run out: Fail the chunk or Pause and wait for me. The pause choice is intended to wait for recovery so Resume can retry the same request. Process pause/resume is unavailable on Windows; see Scripts and Review for checkpoint-based cancellation/resume guidance.

Fail over to the other profile when this one gives up lets the remaining run switch profiles after exhausted retries or a content-policy refusal. Configure and test the alternative beforehand. Local-to-remote failover can disclose text and incur charges; read the confirmation before starting. Disable failover if the alternate destination is not acceptable. A provider warning is not a cost estimate or a spending cap.

Reasoning and provider-specific options

• Reasoning effort is sent as a provider request setting. Available choices include Server default, none — answer directly, low, medium, high, and xhigh. Support and behavior depend on the provider/model; a stronger setting is not automatically better for this task.
• Custom headers (JSON object) applies to requests for that profile. Custom request body (JSON object) supplies provider-specific completion options. Use only settings your server supports and valid JSON objects.
• An explicit reasoning_effort in Custom request body overrides the dropdown. Remove conflicting settings when debugging.
• Banned Tokens in the advanced generation controls is not a universal “turn reasoning off” switch. Do not rely on banning <think> to make a reasoning model produce valid answers.
• Keep prompt presets/defaults intact until the connection and a short annotation work. For script behavior and prompt editing, continue to Scripts and Review.

Optimize LM Studio settings is optional advanced configuration. It can reload/reconfigure the model. The remote path needs an already configured Remote SSH host alias; filling in a hosted API URL does not establish SSH access. Avoid changing this during active work unless you understand the reload and memory implications.

Bundled Start LLM launchers

Pinokio includes Start LLM: Gemma 4 (Voice Tools) and Start LLM: Qwen 2.5 14B (Voice Tools). These are optional alternatives to an independently managed API server.

• Run the core install and Install Voice Tools first.
• Obtain a GGUF model compatible with the selected launcher and place it in the checkout root under the exact filename configured in pinokio.js. The launcher does not download model weights.
• The launcher uses llama-cpp-python[server], binds loopback on an available port, and writes the local profile into the default app/config.json.
• If you use ALEXANDRIA_CONFIG_PATH or a custom data root, do not assume that launcher edit reaches your active settings file. Check Setup, its URL/model, and save the intended profile yourself.
• Its GPU offload and context settings still have to fit your hardware. Starting a local LLM can consume memory needed by TTS.

Connection problems: Troubleshooting. Audio configuration and outputs: Rendering and Export.

Sources

Verified against shipped main at 8c7a33cb310ab9264594a09ca27e48271585094f: Setup and manual UI, save/test UI behavior, connection probe, provider options and key resolution, manual requests and book preflight, bundled LLM launcher.