Skip to content

feat(config): per-operation llm_extra_body overrides - #3159

Merged
nicoloboschi merged 3 commits into
mainfrom
feat/per-operation-llm-extra-body
Aug 4, 2026
Merged

feat(config): per-operation llm_extra_body overrides#3159
nicoloboschi merged 3 commits into
mainfrom
feat/per-operation-llm-extra-body

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Problem

extra_body could only be set globally, via HINDSIGHT_API_LLM_EXTRA_BODY, so retain, reflect and consolidation were forced to share one dict — unlike every other LLM param, which already has a per-operation override.

That's limiting for knobs that are genuinely per-operation. The motivating case: disabling a model's thinking mode for retain extraction while leaving it on for reflect (vLLM chat_template_kwargs). Same for a different max_tokens or top_p per operation.

Change

Three new env vars, following the reasoning_effort pattern exactly:

Variable Falls back to
HINDSIGHT_API_RETAIN_LLM_EXTRA_BODY HINDSIGHT_API_LLM_EXTRA_BODY
HINDSIGHT_API_REFLECT_LLM_EXTRA_BODY HINDSIGHT_API_LLM_EXTRA_BODY
HINDSIGHT_API_CONSOLIDATION_LLM_EXTRA_BODY HINDSIGHT_API_LLM_EXTRA_BODY

Each is parsed into an optional HindsightConfig field and resolved in MemoryEngine as config.<op>_llm_extra_body or config.llm_extra_body, so an operation left unset keeps using the global value. Static server-level config (not added to _CONFIGURABLE_FIELDS), matching the global flag.

# thinking off for extraction, on everywhere else
HINDSIGHT_API_LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": true}}'
HINDSIGHT_API_RETAIN_LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'

A per-operation value replaces the global dict rather than merging with it. Extra-body params are provider-native, so a partial override merged onto a global dict aimed at a different provider would mix two providers' field names — and replacement is how every other per-operation override already behaves.

Scope notes

  • No behaviour change when the new vars are unset: each per-op config resolves to config.llm_extra_body, exactly as before.
  • Indexed multi-LLM members (..._LLM_<n>_EXTRA_BODY) already have their own per-member extra_body and are untouched.
  • No API/schema surface, so no OpenAPI or client SDK regeneration.

Tests

New TestPerOperationExtraBody in tests/test_per_operation_llm_config.py, mirroring TestPerOperationReasoningEffort one-for-one:

  • test_per_operation_extra_body_from_env — each override parses, global untouched
  • test_per_operation_extra_body_fallback_to_global — unset stays None
  • test_memory_engine_applies_per_operation_extra_body — engine threads the reflect override through and retain/consolidation fall back to the global value

Verified locally: 78 passed across test_per_operation_llm_config.py + test_llm_extra_body.py + test_multi_llm_config.py; ./scripts/hooks/lint.sh and uv run ty check hindsight_api/ clean.

Docs

Three rows added to the per-operation table in hindsight-docs/docs/developer/configuration.md.

Extra request-body params could only be set globally via
HINDSIGHT_API_LLM_EXTRA_BODY, so every operation shared one dict. That
forces a single choice on knobs that are genuinely per-operation — e.g.
disabling a model's thinking mode for retain extraction while leaving it
on for reflect (vLLM chat_template_kwargs), or setting a different
max_tokens per operation.

Add the same per-operation override the other LLM params already have:

  HINDSIGHT_API_RETAIN_LLM_EXTRA_BODY
  HINDSIGHT_API_REFLECT_LLM_EXTRA_BODY
  HINDSIGHT_API_CONSOLIDATION_LLM_EXTRA_BODY

Each follows the reasoning_effort pattern exactly: parsed into an
optional HindsightConfig field, resolved in MemoryEngine as
`config.<op>_llm_extra_body or config.llm_extra_body`, so an unset
operation keeps using the global value. Static server-level config (not
per-bank configurable), matching the global flag.

A per-operation value replaces the global dict rather than merging with
it — extra-body params are provider-native, and this is how every other
per-operation override behaves.
Pre-existing drift, unrelated to this PR's feature: the source doc
hindsight-docs/docs-integrations/coding-agents.md was updated by
17b7f46 / d238d2f, but skills/hindsight-docs/ has not been
regenerated since 4278f09. verify-generated-files is therefore red on
main, and stays red on any PR that runs it until the copy is resynced.

Purely the output of ./scripts/generate-docs-skill.sh — no hand edits.
@nicoloboschi
nicoloboschi merged commit cac55fe into main Aug 4, 2026
206 of 208 checks passed
@nicoloboschi
nicoloboschi deleted the feat/per-operation-llm-extra-body branch August 4, 2026 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant