toolseen

Your tool returned 50KB. Your harness forwarded only 9KB. toolseen shows what changed between the two.

A harness can truncate, filter, summarize, or alter a tool result before passing it downstream. toolseen compares the original value with the downstream value you measured.

That is the whole problem in eleven seconds. A fake tool returns ~50KB of JSON with "delayed": true in the middle. A fake harness keeps only the head and the tail. The agent checks whether the word delayed reached it, finds nothing, and reports that the order is on schedule.

Nothing errored. Nothing warned. The fake agent was answering from text that no longer contained the key fact, and toolseen report is what makes that visible.

Install

pip install toolseen

# from a checkout
pip install -e ".[dev]"

Requires Python 3.10+. No runtime dependencies. Comparison is local and deterministic. No network calls or model API.

Compare two values you already have

If you can get both values—two log lines, two captured payloads, or two request bodies—compare() is the whole API. For files, use the CLI below.

from toolseen import compare

verdict = compare(original_text, seen_text)
print(verdict.kind)

Optional arguments: threshold=64 (minimum dropped characters for drop/truncation kinds), ignore_ws=False.

toolseen compare original.txt seen.txt [--threshold 64] [--ignore-ws]

CONTENT_DROPPED (middle)
original: 50,000 chars
seen:      9,000 chars
dropped:  41,000 chars
cause:    unknown
mode:     raw
keys not observed in seen (approximate): delayed, carrier_notes

Instrument a Python harness

If you control the code that calls your tools, wrap the call instead of carrying the strings around.

from toolseen import watch_call, saw

call_id, result = watch_call(check_order)      # records what the tool returned
delivered = my_harness(result)                 # whatever your pipeline does
saw(call_id, delivered)                        # records what went downstream

toolseen report

check_order#1  CONTENT_DROPPED (middle)
original 50,115 -> seen 9,000
41,115 chars dropped
keys not observed in seen (approximate): delayed, items

By default, toolseen report reads the newest session under .toolseen/ in the current working directory.

report() is available from Python too, returning (call_id, Verdict) pairs:

from toolseen import report

for call_id, verdict in report():
    print(call_id, verdict.kind)

watch_call preserves the wrapped function's return value and exception behavior. A str return is recorded as-is; other allowed types are serialized with json.dumps(..., sort_keys=True). Events go to .toolseen/session-<uuid>.jsonl, created 0700/0600 with a .gitignore alongside, and toolseen refuses to record when that is not the case.

Phase 2 requires O_NOFOLLOW and is unavailable on platforms that do not provide it. compare() and toolseen compare are unaffected.

Run the demos

# the recording above
python examples/demo_instrumented.py
toolseen report

# file comparison, no instrumentation needed
python examples/demo_agent.py
toolseen compare examples/original.txt examples/seen.txt

These demos do not claim to model any commercial product's truncation rules.

Exit codes

Code	Meaning
0	Every verdict is OK
1	One or more non-OK verdicts
2	Input / session / schema error

Verdict kinds

Kind	Meaning (short)
OK	Exact, JSON-semantic, or (optional) whitespace match
EMPTY_SEEN	Original non-empty-like; seen empty-like ("", null, [], …)
SUSPECTED_STRINGIFIED	Seen fully matches closed patterns like [object Object]
TRUNCATED	Exactly one known truncation marker; pieces fit original
CONTENT_DROPPED	Large head/tail/middle drop without a marker; cause unknown
SCHEMA_MISMATCH	Both sides JSON objects; top-level key sets differ
MODIFIED	Any other difference (redaction insert, rewrite, …)

CONTENT_DROPPED is not always truncation—it may be filtering or intentional removal.

Scope

toolseen compares the two boundaries you provide or measure. It does not guarantee what the model ultimately received, and it does not auto-hook Codex, Claude Code, Gemini CLI, or any other harness.

Limits

• If you pass the wrong two values, the verdict is still correct for those two values.
• keys not observed in seen (approximate) is a substring heuristic on top-level JSON keys, not proof a field was deleted.
• The capture layer records one process and does not share a session across processes.
• Phase 2 is intended for ordinary local development, not as a tamper-evident audit log.

Security

Tool results can contain secrets, PII, or tokens, and session files hold what your tools returned. Keep .toolseen/ and production payloads out of public commits and issues.

License

MIT