Skip to content

Fix: Optimize token usage for scratchpad tool - #245

Merged
ea-rus merged 8 commits into
stagingfrom
fix/eng-626-memory-consolidation-inflates-scratchpad
Jul 15, 2026
Merged

Fix: Optimize token usage for scratchpad tool#245
ea-rus merged 8 commits into
stagingfrom
fix/eng-626-memory-consolidation-inflates-scratchpad

Conversation

@ea-rus

@ea-rus ea-rus commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Fixes ENG-626 — recall_scratchpad_wisdom() had no size cap and injected content unconditionally into the scratchpad tool description, which is resent on every LLM call. On a heavy real user this grew to ~103K chars, becoming the largest single component of the per-call payload and burying the tool's own safety-critical timeout/progress() contract.

Also fixes two related bugs found while investigating: a shared-singleton mutation bug that leaked memory content across sessions, and a duplication issue where the same procedural knowledge was billed twice per call (system prompt + tool description).

Changes

Static contract ordering (tool_defs.py)

  • Moved the IMPORTANT timeout/progress() block right after the Actions list (where exec/cell are defined) and before the capabilities paragraphs — previously it sat right before the unbounded memory injection, so growth pushed it further from view on every turn.

Token budget for recall_scratchpad_wisdom() (hippocampus.py)

  • "when" rules are now pulled via get_rules() (clean Engram objects) instead of scraping recall_rules()'s raw markdown, which was leaking <!-- confidence:... ts:... --> metadata comments verbatim into the injected text.
  • Rules + lessons are sorted by confidence tier (highmedium/unset → low) then recency within a tier, and trimmed to token_budget (default 2000/scope, ~4 chars/token) via a shared _filter_by_token_budget() helper (also used by get_lessons()).
  • Fixed a correctness bug: the method's own docstring says "relevant to scratchpad execution," but it pulled every when rule regardless of topic. Added the same "scratchpad" in text/topic filter already used for lessons (_is_scratchpad_related()).

Fixed SCRATCHPAD_TOOL singleton mutation (session.py)

  • _build_core_tools() aliased the module-level SCRATCHPAD_TOOL singleton and mutated .description in place (scratchpad_tool = SCRATCHPAD_TOOL). In a process serving multiple sessions/users, each session's injected wisdom compounded on top of the previous one's, forever, until process restart — a real risk of leaking one user's memory into another user's tool description within the same worker.
  • Fixed with scratchpad_tool = replace(SCRATCHPAD_TOOL) (shallow copy via dataclasses.replace).
  • Added a regression test that builds tools for two sequential sessions and asserts the singleton's .description is untouched and no cross-session leakage occurs.

Removed duplication between system prompt and tool description (cortex.py, hippocampus.py)

  • build_memory_context() (system prompt) and get_scratchpad_context() (tool description) were both pulling from the same get_rules()/get_lessons() data with no coordination — any scratchpad-related when rule or lesson was billed in both places on every call.
  • Added exclude_scratchpad / exclude_scratchpad_when parameters to get_lessons() / get_rules() (default False, so recall_scratchpad_wisdom() and the existing recall_rules() caller in session.py — an ACC dedup check that needs the full rule set — are unaffected).
  • recall_lessons() now always excludes scratchpad-related entries; build_memory_context() calls get_rules(exclude_scratchpad_when=True) / recall_lessons(token_budget=1000) directly, keeping the LLM-based rule-relevance filtering and episodic-memory read-logging paths intact.

Testing

  • tests/test_hippocampus.py, tests/test_cortex.py, tests/test_chat_scratchpad.py — new coverage for: budget truncation, confidence/recency ordering, the scratchpad-relatedness filter on rules, singleton-mutation regression, and system-prompt/tool-description exclusion in both directions.
  • Full suite: 1170 passed (one pre-existing e2e flake, test_large_input_no_crash, reproduces identically on main — unrelated to this change, confirmed via git stash).

Not in this PR

  • Relocating long-tail lessons to recall_skill (pull-based) instead of push-injecting into the tool description — larger change, left as follow-up.
  • Filtering user/task-specific trivia (e.g. campaign names) at write time in the consolidator/cerebellum — separate from the read-time budget/sort work done here.

Fixes: https://linear.app/mindsdb/issue/ENG-626/memory-consolidation-inflates-scratchpad-tool-description-without

@ea-rus
ea-rus marked this pull request as ready for review July 13, 2026 09:07
@ea-rus
ea-rus merged commit 3b991fe into staging Jul 15, 2026
5 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 15, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants