Add context source ledger and prompt observability

[cbusillo]

Finish Line

Every Code has a visible, testable prompt/context assembly contract: each context source has an identity, prompt-size contribution is observable before requests are sent, and duplicate/bloated context producers are caught by tests or diagnostics before they silently burn tokens.

Current Status

State: Core request ledger and first /context TUI view are merged; #93 is complete.
Next action: Use /context on real sessions to choose the next observability/budgeting increment: duplicate grouping, richer visual bars, harness request-shape assertions, or budgeting thresholds.
Blocked by: None.
Waiting for: Next implementation slice selection.
Last verified: 2026-05-22; PR #96 merged as f09fa68, local overlay was fast-forwarded, just local-code-rebuild completed, and PATH code reports 0.6.98.

Acceptance Criteria

• Prompt assembly has named source/category buckets such as base instructions, developer messages, user/project instructions, skills catalog, explicit skill bodies, environment context, memories, history, pending input, tool outputs, and status/browser items.
• Before API submission, Every Code logs or emits a compact prompt composition summary with item counts and byte/token estimates by bucket.
• The same ledger powers a user-facing /context command or view in the TUI, similar in spirit to Claude Code / Antigravity context views but tailored to Every Code's sources.
• /context shows at least total estimated context, per-source budget bars/counts, whether each source is persisted or request-only, and the largest contributors.
• /context can highlight likely duplicate contextual fragments or repeated source identities before tokens are spent.
• Contextual fragments have stable identity or markers so duplicate user instructions, skills catalog, environment context, explicit skills, and status fragments can be detected intentionally rather than by ad hoc string checks.
• Regression tests assemble prompts with multiple producers active and assert each one-shot contextual source appears once.
• The code-exec harness has fake Responses coverage proving the ledger/view matches the real executable request shape, not only lower-level helper internals.
• The design supports future budget enforcement for Preserve and budget Every Code memory mode #47 and Add auto-review and auto-drive budget guardrails #50 without requiring every context producer to reinvent its own limit logic.
• The design supports Support manual-only skills outside default context #91 manual-only skills by separating "discoverable/installed", "included in default prompt context", and "explicitly injected for this turn".

Evidence / Context

This is the systemic follow-up to two concrete duplication bugs:

• Duplicate tool outputs were found in rollout/session files and fixed in commit 75710e00fd by filtering already-recorded pending tool outputs.
• De-duplicate prompt and skill injection overhead #46 is now tracking a likely composed-instructions duplication path where project docs and skills are folded into user_instructions, then recomposed.

Read-only agent review found the common pattern:

• Several context producers append directly into request/history without a shared ledger.
• Some producers dedupe locally, but there is no final assembled-prompt invariant.
• Upstream codex-rs has useful reference patterns around contextual fragments and skill config/policy that can guide Every Code without wholesale porting.

Proposed Phases

1. De-duplicate prompt and skill injection overhead #46/Support manual-only skills outside default context #91 immediate fixes: keep raw/effective instructions separate, avoid duplicated project-doc/skills context, and support manual-only skills plus explicit request-only $skill injection.
2. Ledger data model: add a small internal prompt/context source record with source id, category, persistence class, byte/token estimate, and duplicate key.
3. Request assembly instrumentation: populate the ledger from the same places that assemble the real request, then log or emit it immediately before API submission.
4. /context first view: expose a compact TUI command that shows total estimate, per-source bars/counts, persisted vs request-only context, top contributors, and duplicate warnings.
5. Harness truth test: add fake Responses scenarios that assert /context or emitted ledger data matches the captured real request body shape.
6. Budget guardrails: use the same source categories to warn or enforce limits for memories, auto-review/auto-drive, history, status/browser items, and explicit skills.
7. UX polish: add drilldowns, before/after compaction views, and selected-source inspection once the ledger is proven reliable.

Open Questions

• Should the first implementation slice be log/event-only, or should it include a minimal /context command from the start?
• Should /context inspect the last submitted request, the next request estimate, or both?
• Which persistence labels should be canonical: persisted history, contextual/session, request-only, generated-per-attempt, and tool-result?
• Should token counts start with approximate byte/token estimates and graduate to provider/model-specific tokenizer estimates later?
• Should duplicate detection be fail-fast in tests only, warn at runtime, or surface directly in /context?
• Should the first TUI view be text/table only, or include proportional bars immediately?
• How should compacted history and previous-response-id/ZDR modes be represented so the view stays truthful across providers?

Relationships

Parent context: #43
Related: #46, #47, #50, #76, #90, #91

• related: Plan Every Code token efficiency and local LLM sandboxing #43 - Plan Every Code token efficiency and local LLM sandboxing #43
• related: De-duplicate prompt and skill injection overhead #46 - De-duplicate prompt and skill injection overhead #46
• related: Preserve and budget Every Code memory mode #47 - Preserve and budget Every Code memory mode #47
• related: Add auto-review and auto-drive budget guardrails #50 - Add auto-review and auto-drive budget guardrails #50
• related: Make Auto Review snapshot-aware and preserve superseded results #76 - Make Auto Review snapshot-aware and preserve superseded results #76
• related: Triage rollout/session friction audit findings #90 - Triage rollout/session friction audit findings #90
• related: Support manual-only skills outside default context #91 - Support manual-only skills outside default context #91

[cbusillo]

Harness policy update relevant to prompt observability:

tools/code-exec-harness should become the black-box proving ground for this issue once the context source ledger exists. Fake /v1/responses scenarios can already assert raw request shape; the ledger should make those assertions more precise by exposing prompt/context sources such as base instructions, project docs, skills, memories, compaction summaries, resume history, and tool outputs.

Acceptance criteria this issue should preserve:

• harness scenarios can detect duplicate prompt sources without brittle raw-string spelunking,
• request/body evidence can be tied back to source categories and approximate token budgets,
• fake model runs remain the first validation layer for deterministic prompt-shape bugs,
• live/local model harness runs can compare effectiveness when behavior depends on model judgment.

This complements #46's first concrete dedup fix and keeps the broader systemic fix focused on observability rather than one-off string filtering.

[cbusillo]

Additional harness finding while adding dedup scenarios:

A candidate scenario with CODE_HOME/config.toml containing instructions = "Harness base instruction marker." showed that the marker did not appear in either the Responses API instructions field or the user-input context for code exec.

Observed evidence:

• ConfigToml has an instructions field.
• code exec builds ConfigOverrides { base_instructions: None, ... }.
• Config::load_from_base_config_with_overrides appears to populate base_instructions only from the override or experimental_instructions_file; I did not find cfg.instructions wired into the effective config.

I removed the failing scenario from the current harness change because it was proving a separate stale/ambiguous config-surface issue, not the project-doc/skill dedup fix. For #92, this is a good example of why the context ledger should expose source categories and effective prompt fields directly: future harness runs should make it obvious whether a config field is unused, routed to system instructions, or routed to user context.

[cbusillo]

Implemented first #92 slice locally on feat/context-source-ledger: core request-source ledger plus fake Responses executable proof.

What changed:

• Added a core ContextLedger model with source kind, persistence class, byte estimate, approximate token estimate, labels, and duplicate keys.
• The Responses request paths now compute the ledger from the final formatted input plus base instructions and tool schemas, then emit a debug log at code_core::context_ledger.
• The ledger distinguishes user/project instructions, skills manifest, explicit request-only skills, generated status/browser items, tool outputs, tool schemas, base/developer instructions, and history-like input.
• Added focused unit tests for source classification and skills-manifest splitting.
• Added code-exec-harness stderr_contains expectations and scenario tools/code-exec-harness/scenarios/context-ledger-request-summary.json to prove the real executable path logs ledger categories while fake Responses captures the actual request body.

Validation:

• cargo test -p code-core client_common::tests::context_ledger
• cargo test -p code-core --test prompt_context_dedup
• python3 tools/code-exec-harness/harness.py tools/code-exec-harness/scenarios/context-ledger-request-summary.json --code-bin /Users/cbusillo/Developer/code/code-rs/target/debug/code
• ./build-fast.sh
• python3 tools/code-exec-harness/harness.py tools/code-exec-harness/scenarios/context-ledger-request-summary.json --code-bin /Users/cbusillo/Developer/code/.code/working/_target-cache/code/feat-context-source-ledger-be2596d583c8-2af88e04f956/code-rs/dev-fast/code

This intentionally stops before a TUI /context view. The next slice should expose the ledger through protocol/session state and render it in #93 rather than reconstructing prompt shape in the TUI.