feat(config): per-operation llm_extra_body overrides - #3159
Merged
Conversation
Extra request-body params could only be set globally via HINDSIGHT_API_LLM_EXTRA_BODY, so every operation shared one dict. That forces a single choice on knobs that are genuinely per-operation — e.g. disabling a model's thinking mode for retain extraction while leaving it on for reflect (vLLM chat_template_kwargs), or setting a different max_tokens per operation. Add the same per-operation override the other LLM params already have: HINDSIGHT_API_RETAIN_LLM_EXTRA_BODY HINDSIGHT_API_REFLECT_LLM_EXTRA_BODY HINDSIGHT_API_CONSOLIDATION_LLM_EXTRA_BODY Each follows the reasoning_effort pattern exactly: parsed into an optional HindsightConfig field, resolved in MemoryEngine as `config.<op>_llm_extra_body or config.llm_extra_body`, so an unset operation keeps using the global value. Static server-level config (not per-bank configurable), matching the global flag. A per-operation value replaces the global dict rather than merging with it — extra-body params are provider-native, and this is how every other per-operation override behaves.
Pre-existing drift, unrelated to this PR's feature: the source doc hindsight-docs/docs-integrations/coding-agents.md was updated by 17b7f46 / d238d2f, but skills/hindsight-docs/ has not been regenerated since 4278f09. verify-generated-files is therefore red on main, and stays red on any PR that runs it until the copy is resynced. Purely the output of ./scripts/generate-docs-skill.sh — no hand edits.
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
extra_bodycould only be set globally, viaHINDSIGHT_API_LLM_EXTRA_BODY, so retain, reflect and consolidation were forced to share one dict — unlike every other LLM param, which already has a per-operation override.That's limiting for knobs that are genuinely per-operation. The motivating case: disabling a model's thinking mode for retain extraction while leaving it on for reflect (vLLM
chat_template_kwargs). Same for a differentmax_tokensortop_pper operation.Change
Three new env vars, following the
reasoning_effortpattern exactly:HINDSIGHT_API_RETAIN_LLM_EXTRA_BODYHINDSIGHT_API_LLM_EXTRA_BODYHINDSIGHT_API_REFLECT_LLM_EXTRA_BODYHINDSIGHT_API_LLM_EXTRA_BODYHINDSIGHT_API_CONSOLIDATION_LLM_EXTRA_BODYHINDSIGHT_API_LLM_EXTRA_BODYEach is parsed into an optional
HindsightConfigfield and resolved inMemoryEngineasconfig.<op>_llm_extra_body or config.llm_extra_body, so an operation left unset keeps using the global value. Static server-level config (not added to_CONFIGURABLE_FIELDS), matching the global flag.A per-operation value replaces the global dict rather than merging with it. Extra-body params are provider-native, so a partial override merged onto a global dict aimed at a different provider would mix two providers' field names — and replacement is how every other per-operation override already behaves.
Scope notes
config.llm_extra_body, exactly as before...._LLM_<n>_EXTRA_BODY) already have their own per-memberextra_bodyand are untouched.Tests
New
TestPerOperationExtraBodyintests/test_per_operation_llm_config.py, mirroringTestPerOperationReasoningEffortone-for-one:test_per_operation_extra_body_from_env— each override parses, global untouchedtest_per_operation_extra_body_fallback_to_global— unset staysNonetest_memory_engine_applies_per_operation_extra_body— engine threads the reflect override through and retain/consolidation fall back to the global valueVerified locally: 78 passed across
test_per_operation_llm_config.py+test_llm_extra_body.py+test_multi_llm_config.py;./scripts/hooks/lint.shanduv run ty check hindsight_api/clean.Docs
Three rows added to the per-operation table in
hindsight-docs/docs/developer/configuration.md.