Skip to content

cli: a malformed trace file crashes trace/metrics/viz with a traceback instead of the exit-2 report #15

Description

@Shashankss1205

Summary

grapharc trace, grapharc metrics and grapharc viz crash with a raw pydantic traceback when the trace file contains a line that is not a valid TraceEvent — exit code 1, nothing on stdout even in --json mode. The CLI's own contract says this exact situation is an exit-2 report: grapharc/cli/output.py defines

# The command could not run at all: a component that is not installed, an
# unreadable trace, a model spec that names no backend.
EXIT_UNAVAILABLE = 2

Reproduced against the current tree:

$ printf '{"not": "a trace event"}\n' > bad.jsonl
$ grapharc trace bad.jsonl --json; echo EXIT=$?
Traceback (most recent call last):
  ...
  File ".../grapharc/cli/main.py", line 483, in _cmd_trace
    events = recorder.read_events(args.run_id)
  File ".../grapharc/observe/trace.py", line 192, in read_events
    ev = TraceEvent.model_validate_json(line)
pydantic_core._pydantic_core.ValidationError: 6 validation errors for TraceEvent
ts
  Field required [type=missing, ...]
...
EXIT=1

The same happens for a line that is not JSON at all (ValidationError: Invalid JSON: expected ident at line 1 column 2), and for grapharc metrics bad.jsonl r1 and grapharc viz bad.jsonl r1viz catches ReplayError but the ValidationError fires inside read_events before replay ever runs.

Why this matters

Traces are advertised as "human-readable, greppable, git-versionable" JSONL that other processes append to. A truncated final line (a process killed mid-write), a hand-edited file, or any stray line makes three reading commands crash instead of report. In --json mode this breaks the CLI's stated promise that "the failure is the document": a script gets an empty stdout, a traceback on stderr, and exit 1 — the code reserved for "the command ran and the answer was negative", so the script concludes the run differed or failed rather than that the file was unreadable. Meanwhile grapharc replay and grapharc diff on the same file answer politely (replay failed: …), so sibling commands disagree about the same broken file.

Where in the code

  • grapharc/cli/main.py:483_cmd_trace: events = recorder.read_events(args.run_id), unguarded
  • grapharc/cli/main.py:513_cmd_metrics: metrics = summarize(recorder, args.run_id)summarize calls read_events, unguarded
  • grapharc/cli/main.py:534-540_cmd_viz catches only ReplayError; ValidationError escapes from the same call
  • grapharc/observe/trace.py:190-193read_events calls TraceEvent.model_validate_json(line) per line and lets ValidationError propagate with no file/line context
  • grapharc/cli/output.py:25-27 — the exit-code contract that names "an unreadable trace" as EXIT_UNAVAILABLE = 2
  • grapharc/cli/replay.py:54-64 — the pattern the fix should match: the sibling commands wrap the engine call and route the failure through fail(...)

Confirm it yourself:

printf '{"not": "a trace event"}\n' > /tmp/bad.jsonl
uv run python -m grapharc.cli.main trace /tmp/bad.jsonl --json; echo "exit=$?"   # traceback, exit 1
uv run python -m grapharc.cli.main metrics /tmp/bad.jsonl r1; echo "exit=$?"     # traceback, exit 1
uv run python -m grapharc.cli.main viz /tmp/bad.jsonl r1; echo "exit=$?"         # traceback, exit 1

What to change

  1. In grapharc/observe/trace.py, raise a named error from read_events when a line does not parse — e.g. a TraceReadError(path, line_number, cause) — so the message can say which line of which file is bad instead of dumping a pydantic field listing. Keep raising (do not silently skip lines): a partially-read audit trail presented as complete would be worse than a refusal.
  2. In grapharc/cli/main.py, catch that error in _cmd_trace, _cmd_metrics and _cmd_viz and return fail(...) with the default EXIT_UNAVAILABLE, following the shape _existing_trace and _cmd_viz's existing ReplayError handler already use. Text mode: error: … on stderr, empty stdout. JSON mode: one failure document on stdout.
  3. Make the message name the path and the 1-based line number, e.g. unreadable trace file: /tmp/bad.jsonl: line 1 is not a trace event.

Out of scope: changing replay/diff (they already report, via their own Exception guard and exit 1 — whether that should also be 2 is a separate discussion); tolerating/skipping bad lines; the server's trace endpoints; thread_summary's incremental index (it reads raw JSON, not TraceEvent, and is a runtime path other agents cover).

How to verify

uv run pytest -q
uv run ruff check .

Add tests (e.g. in tests/test_cli.py) that a trace file containing one invalid line makes trace, metrics and viz exit 2 with an error: line naming the file — and, in --json mode, print exactly one document with "ok": false and an empty stderr. Revert the source edit and watch each go red: today all three raise ValidationError.

Acceptance criteria

  • grapharc trace|metrics|viz <file-with-a-bad-line> exits 2 with a message naming the file and line, no traceback
  • In --json mode the failure is a single JSON document on stdout and stderr is empty
  • A well-formed trace file behaves exactly as before (the pinned README/cookbook transcripts are unchanged)
  • uv run pytest stays green and uv run ruff check . is clean
  • Any README or cookbook sentence this changes is updated in the same pull request (several are byte-compared against real output by the test suite)

Skill level

good first issue — well bounded: one new exception class, three small except blocks, and two live patterns to copy in the same file (_existing_trace at grapharc/cli/main.py:238 and the ReplayError handler at grapharc/cli/main.py:536). The one judgement call — raise versus skip a bad line — is decided above (raise). Questions welcome on the issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions