Headroom as a baseline in a study on learned field projections for agent tool outputs

[alepom]

Hi, and thanks for Headroom.

I've just published an exploratory study and an open-source library, tracetrim, on a question close to Headroom's: which fields of a tool output does an agent actually reuse in what it writes, and can the rest be removed by learning that from the agent's own traces? Headroom is the main baseline in the paper, so I'd like you to see how we used it before anyone reads the comparison.

• Paper: https://doi.org/10.5281/zenodo.23953934
• Code and data: https://github.com/alepom/tracetrim (archived at https://doi.org/10.5281/zenodo.23954125)

How Headroom was used. headroom-ai 0.39.1, run once per conversation (stateless, so TOIN had no history of these tools), with CompressConfig(protect_recent=0, protect_analysis_context=False, target_ratio=0.2, compress_user_messages=True). The input was the conversation's user message, tool calls and tool messages, with no system prompt, and model="gpt-4o". We measured tokens only (tiktoken o200k_base), not the effect on answers. The script is experiments/headroom_baseline.py.

What we saw, on the same conversations. Headroom removed 17–28% of tool-output tokens on two agents with real APIs and 31–33% on two simulated ones (Stripe- and GitHub-shaped). A projection learned from seven runs of the same task removed 46–53% and 57–61%. A plain JSON minifier, which removes no field, removed 16–24% and 18–33%. That last baseline is the one I'd most like your read on. In this stateless, one-shot setting Headroom's result was often close to minification, which makes me think our configuration didn't let it do what it is designed for. TOIN learning over repeated calls and CCR retrieval are close in spirit to what we do (CCR is very much like our expand), and the paper's related-work section says so.

Questions:

1. Is there a configuration you would consider fair for this setting (repeated runs of the same agent task, structured JSON from MCP tools)? I'm happy to rerun the baseline with it and publish an update.
2. Would a per-tool list of fields learned from what the agent writes, final answer included, be useful as a signal for TOIN or SmartCrusher? Our failure analysis may be relevant: values the agent transforms, fields used only to reason, and agents inventing identifiers when an id field is removed.

Everything is reproducible from the repository, and any correction to how we described or ran Headroom is welcome.

Alessio Pomaro