Agent Context

Agent Context

简体中文 · Home

Agent context is an opt-in, start-time terminology snapshot. It is not a stop-time assistant-message excerpt, and it is not limited to LLM refinement. One immutable snapshot can supply bounded terminology views to Qwen-Audio-3 Streaming and Refine.

Enable it

[llm]
agent_context_enabled = true
agent_context_max_chars = 6000

Terminology is built when either Refine is enabled or the selected ASR provider is alibaba-qwen-audio3. Audio3 can therefore receive Session Context even when llm.enabled = false. The public configuration keeps the feature disabled.

Start-time snapshot

When a recording starts, Voice Input freezes:

• the focused Kitty window and its focused foreground pi or codex process;
• the resolved Pi or Codex session;
• the latest completed assistant source available at that moment.

The source is read before the terminology worker starts. A later Kitty tab change, focus change, or new assistant response cannot change that operation's snapshot. Audio capture is prepared before potentially slow discovery so start-time session lookup does not discard post-hotkey audio.

The source is processed locally in this order:

1. validate the process, session identity, canonical path, and file identity;
2. read at most the last 8 MiB of the session JSONL;
3. select the latest completed assistant text and ignore tool-call blocks;
4. redact known sensitive lines and token-shaped secrets;
5. clamp the source budget to 500–12,000 characters;
6. segment locally with Jieba, filter unsuitable or secret-like candidates, and deduplicate case-insensitively;
7. count occurrences in the redacted source and order unique terms from low frequency to high frequency, with stable source order for ties;
8. retain an immutable, operation-local terminology snapshot.

The complete source text and term frequencies remain local.

Two remote views

The consumers select independent, whole-term views from the same snapshot:

Consumer	Sent data	Limits and timing
Qwen-Audio-3 Streaming	Newline-separated plain-text terms in the initial run-task input context	At most 400 characters, including newlines. This applies even when LLM refinement is disabled.
Refine	reference_context.agent plus a terminology array accompanying the transcript	At most 96 terms and 1,500 terminology characters.

Voice Input never sends the original assistant message, frequency counts, session paths, process IDs, window titles, application metadata, or raw desktop metadata in either view. Audio3 receives no agent label. Refine receives only the canonical Pi or Codex label associated with the start-time snapshot.

Audio3 does not send continue-task. If one recoverable pre-finish interruption creates a replacement task, the replacement run-task receives exactly the same bounded context string. Terms and transcripts from the old and replacement tasks are not merged.

Before Refine and again after the model response, Voice Input locally normalizes high-confidence technical variants that differ from selected snapshot terms only by ASCII case or separators. Boundary and eligibility checks prevent broad replacement of ambiguous subterms. The snapshot and normalization rules do not persist across recordings.

Stop-time focus has a separate role

When dictation stops and LLM refinement is enabled, the focused destination determines only the refinement presentation style:

• a focused Pi or Codex destination selects compact coding-agent Markdown;
• a recognized native messaging client selects conversational message punctuation;
• other destinations use the default lightly formal style.

Stop-time focus does not reselect or refresh the start-time terminology source. Conversely, start-time terminology does not force coding-agent Markdown if the destination at stop is different.

Pi discovery

make install copies:

~/.pi/agent/extensions/voice-input-session-registry.ts

The extension publishes a mode-0600 registry file in:

$XDG_RUNTIME_DIR/voice-input/agent-sessions/pi-<pid>.json

It updates on Pi session lifecycle events and every five seconds, and removes its own registry on shutdown. Voice Input validates:

• registry schema version, PID, and Linux process start ticks;
• that the canonical session path is below ~/.pi/agent/sessions;
• that the session header ID matches the registry;
• that process identity, file device, and inode still match when the source is loaded.

Pi must be restarted or its extensions reloaded after installation or update. An already running Pi process does not automatically load the copied extension.

Registry publications use one serialized write queue and per-runtime generation tokens. Shutdown invalidates its generation, waits for queued writes, and removes only its own registry. A generation check immediately before atomic rename prevents a late heartbeat from replacing a newer entry after /new, /resume, /fork, or /reload.

Voice Input follows the active Pi branch backward to the latest assistant message with stopReason = "stop", collecting text blocks only.

Codex discovery

Codex needs no bundled extension. Voice Input:

1. reads CODEX_HOME from the focused Codex process, falling back to ~/.codex;
2. inspects that process's open file descriptors through Linux /proc;
3. accepts .jsonl files canonically below CODEX_HOME/sessions whose header has source = "cli" and thread_source = "user";
4. proceeds only when exactly one eligible session file is open;
5. selects the latest assistant final_answer, with task_complete.last_agent_message as a compatibility fallback.

Container boundaries, restricted /proc, multiple eligible files, or a non-CLI session produce no terminology snapshot rather than a guessed session.

Kitty requirement

The active window class at recording start must be kitty. Voice Input queries the Kitty OS window through:

unix:/tmp/kitty-<kitty-process-id>

The kitty @ … ls result must identify a focused foreground process whose executable basename is exactly pi or codex. The external query has a one-second timeout.

A launcher can enable the required per-process socket as follows:

#!/bin/sh
exec kitty -o allow_remote_control=yes \
  --listen-on "unix:/tmp/kitty-$$" "$@"

Because exec preserves the shell PID, the socket name matches the Kitty process. Adapt this pattern to your launcher and security policy.

Redaction and trust boundary

Redaction replaces whole lines containing common authorization, API-key, password, secret, token, private-key, or cookie markers. It also replaces known token prefixes and long JWT-like values, then caps the redacted source before extraction. This is heuristic protection: it cannot recognize every credential, private name, proprietary term, or sensitive sentence.

Refine treats every terminology candidate as untrusted vocabulary. The prompt permits terms only to resolve likely ASR spellings and explicitly forbids treating them as instructions or claims. The transcript itself is also text to edit: questions, commands, and requests must be preserved, not answered or executed. Prompt instructions reduce risk but cannot prove perfect behavior by every third-party model.

Do not enable Session terminology for a source whose extracted vocabulary must not reach the configured Alibaba or LLM provider.

Cancellation and failure behavior

Cancellation stops later refinement and output, but it cannot revoke data already read or sent. In particular, start-time source text may already have been read locally, and Audio3 may already have sent its run-task Session Context before record cancel is received.

Terminology is fail-soft:

• unsupported focus, failed Kitty discovery, an invalid or ambiguous session, no completed assistant source, redaction/extraction failure, or a consumer wait timeout produces no terminology view;
• Audio3 waits for at most min(asr.connect_timeout_ms, 5000) ms, then starts without Session Context if no snapshot is ready;
• before refinement starts, the daemon waits for at most min(llm.timeout_ms, 5000) ms, then proceeds without terminology if needed; the later LLM attempts still share their own configured refinement deadline;
• failure to build terminology does not alter recognized text directly;
• LLM failure still fails open to the recognized ASR text.

In Audio3 Adaptive mode, a usable, explicitly finished stream that actually sent Session Context is not replaced by Native solely because the recording reached 30 seconds. Empty, degraded, interrupted, overloaded, or incompletely finished streams still follow normal full-audio recovery policy.

Diagnose selection safely

Start with the support-safe report:

voice-input diagnostics --format text

Schema 4 reports only whether a nonempty Audio3 Session Context was sent; it does not contain terms or source text. Daemon logs can additionally report start-time discovery outcomes, source character count, candidate count, selected term counts/characters, and extraction time without printing the terms:

journalctl --user -u voice-input.service -b \
  | grep -E 'voice-input (agent context|refinement context)'

Check Pi registry creation without opening the files:

ls -l "$XDG_RUNTIME_DIR/voice-input/agent-sessions/"

Check the Kitty endpoint with the actual active-window PID:

hyprctl activewindow -j
kitty @ --to unix:/tmp/kitty-<pid> ls

Do not publish the Kitty response, session command lines, registry contents, session paths, or JSONL files unless you have reviewed them for private data. Do not use voice-input status --extended as a support report because it can contain transcript and tooltip text.

See also: Security and Privacy · Troubleshooting