fix(opencode): tool-call telemetry captured NOTHING — 2,699/2,699 rows read text='unknown'

[ZacxDev]

The bug

All 2,699 source='opencode' kind='tool-call' rows on the workbench carry text='unknown', and a payload that is not valid JSON. OpenCode tool distribution has never been measurable, so we cannot say whether opencode has the same Bash-over-Grep bias Claude Code demonstrably has.

Root cause — two independent defects

1. The tool name was never read. tool.execute.after did input?.tool?.name || input?.name || "unknown". The installed contract (~/.config/opencode/node_modules/@opencode-ai/plugin/dist/index.d.ts:249) is:

"tool.execute.after"?: (input: { tool: string; sessionID: string; callID: string; args: any },
                        output: { title: string; output: string; metadata: any }) => Promise<void>;

input.tool is a string, so .name is undefined on every single call and the literal fallback won 100% of the time.

2. The payload was shell-mangled. emitEvent ran execSync with the arguments joined into one shell string, values unquoted. Bash brace-expanded the unquoted JSON object into three separate b64:payload= arguments; emit keeps the last. Reproduced byte-for-byte in test_harness_negative_control_sees_the_mangled_payload:

b64:payload=duration_ms:0
b64:payload=success:true
b64:payload=args_summary:{"name":"customize-opencode"}     <- exactly the stored value

Values with spaces were word-split too: b64:cwd=/tmp/dir with space → 3 argv entries.

The fix

• execFileSync(emit, argv) — no shell, no splitting, no quote removal.
• extractToolName() reads the string form (tolerating a {name} object form for contract drift).
• A capture failure is now distinguishable from a real tool. unknown was a plausible tool name; failures now write the namespaced sentinel __name_capture_failed__ plus name_captured:false and name_capture_shape:<typeof>. A tool genuinely called unknown still records name_captured:true.
• duration_ms was output.duration_ms — a field that does not exist on this hook — so it was always 0. Now measured across tool.execute.before → .after via a bounded callID map, and absent (not 0) when unmeasurable.
• success:true was hard-coded; the hook cannot observe a failure at all (OpenCode throws past it). Replaced with outcome:"completed".
• session came from currentSession, set only by a session.created handler the contract never calls. Now input.sessionID.
• args_summary was a raw .slice(0,200) of a JSON string — invalid JSON whenever it truncated. Now degrades to {"_truncated":true,"keys":[…],"bytes":N}.

Why this shipped: the suite was never gated

scripts/collector/opencode/tests was not in run-tests.sh's target list — 166 tests that no gate has ever run. Added. This is the only line of run-tests.sh this PR touches: the pytest dir list (~line 172). MIN_TESTS deliberately untouched.

The pre-existing test_plugin.py could not have caught either bug: it tests a copy of emitEvent re-typed inside the test string and builds emit's argv itself with subprocess.run([...]), so neither the real function nor the real hook handler is ever executed. Both of its tool-call tests pass unchanged against the broken code.

Harness validation + control pair

Negative controls (harness must go RED on known-bad code) — a vendored reproduction of the two pre-fix mechanisms:

control	asserts	result
test_harness_negative_control_sees_the_lost_tool_name	pre-fix extraction yields text == 'unknown'	passes → harness can observe the name
test_harness_negative_control_sees_the_mangled_payload	pre-fix payload is unparseable and equals the live value verbatim	passes → harness can observe the corruption

Positive control for the reassuring zero — corrupted_arg_count() returns 0 for the fixed code:

	corrupted argv entries
positive control (buggy emitter)	3 (proj, with, space) + 3 b64:payload= fragments
under test (fixed)	0

⚠ Worth recording: my first positive control measured 0 and failed. It asserted on a spacey value inside args_summary, but JSON.stringify wraps string values in ", so the shell preserves those spaces. The splitting hits the fields the plugin does not quote (project, cwd). Without that control, test_no_argv_corruption's 0 would have been read as proof while the counter was blind to the case it exists for.

⚠ Correction — the first push was RED on the authoritative gate

The original version of this section reported 4568 collected / 4567 passed / 0 failed as "the full gate". That was the dev-host tier (nix-shell with the flake's package set). It is not the gate. The authoritative gate is nix build .#checks.x86_64-linux.pytests, and on the first push it was RED.

I reproduced it independently before fixing anything, and ran the discriminating control the coordinator also ran: the branch fails alone, so it is not a cross-PR interaction.

Why the dev host could not see it

Both write_mock_emit (pre-existing) and write_argv_recorder (new) write a stub script at runtime with #!/usr/bin/env bash.

• NixOS dev host — /usr/bin/env exists (lrwxrwxrwx /usr/bin/env -> /nix/store/…-coreutils-9.10/bin/env) → the stub execs, suite green.
• nix build sandbox — no /usr/bin/env → execve fails ENOENT.

nixpkgs' patchShebangs src/scripts (flake.nix) rewrites shebangs of files in the source tree — which is why the real scripts/collector/emit works in the sandbox — but it cannot touch a file a test creates while running. The defect was structurally unobservable in the tier I measured.

It was also mostly silent: emitEvent swallows every error by design, so node still exits 0 when the stub cannot exec; tests then failed on an empty list (IndexError), pointing at the wrong thing.

The fix — deterministic, one place, assertions untouched

New tests/_mockbin.py owns the shebang (/bin/sh, POSIX bodies — the convention four other suites in this repo already carry as hand-written comments) and raises if a call site supplies its own. Both call sites go through it. No assertion was weakened.

Added so the class cannot return:

• test_the_mock_helper_can_actually_exec — named tier precondition, runs in both tiers, never skips.
• test_no_runtime_written_shebang_uses_usr_bin_env — structural scan of the suite's sources, with a positive control proving the scan can flag an offender. Its needles are assembled from char codes: the first version matched its own source lines (the pgrep -f self-match).
• drive_tool_call now names the rival mechanism on an empty log instead of letting IndexError misreport it.

Two further defects the sandbox exposed — neither fixed by weakening

1. _mockbin.py was untracked, so the flake source omitted it (flakes only see tracked files) and the run collapsed to opencode collected=2 errors=2 — 168 tests silently not collected, while the global floor (2850) was nowhere near tripping. Caught by comparing collected counts across runs, not by an exit code. Staged.

2. The corruption's SHAPE is shell-dependent. The pre-fix execSync bug has two effects and only one is universal:

• word-splitting + quote removal — every POSIX shell, both tiers;
• brace expansion of the unquoted JSON object into three b64:payload= args — bash only, and it is what produced the exact stored value args_summary:{"name":"…"} in the 2,699 live rows (the workbench runs bash). The sandbox's /bin/sh does not brace-expand.

The negative controls now assert the universal half unconditionally and gate the bash-only signature on _mockbin.shell_does_brace_expansion() — a measured probe, with test_brace_expansion_probe_agrees_with_the_shell_node_actually_uses pinning that the probe describes the same shell node's execSync actually runs. Both shapes are asserted; neither branch is a skip.

Both tiers, both numbers

tier	command	result
dev host	nix-shell … python -m pytest scripts/collector/opencode/tests/	170 passed, 0 skipped
authoritative	nix build .#checks.x86_64-linux.pytests	opencode collected=170 passed=170 skipped=0 TOTAL collected=4543 passed=4542 skipped=1 failed=0 · RESULT: PASS

Progression on the authoritative gate:

revision	opencode	TOTAL	verdict
first push 8306719	166 / 152 passed / 14 failed	4539 / 4524 / 14 failed	FAIL
helper added but untracked	collected=2 / errors=2	4375 / 4372 / 2 failed	FAIL (collection collapse)
403f65f (this branch)	170 / 170 passed	4543 / 4542 / 1 skipped / 0 failed	PASS

origin/main baseline in this tier = 4373 collected (the opencode suite was not in the target list at all).

Red/green matrix — SANDBOX TIER

Base ref origin/main = 73f5ec0. Method: at 403f65f, git checkout origin/main -- scripts/collector/opencode/activity-plugin.js, run nix build .#checks.x86_64-linux.pytests, restore.

	plugin at 73f5ec0	plugin at 403f65f
scripts/collector/opencode/tests	collected=170, 159 passed, 11 failed	collected=170, 170 passed
TOTAL	4543 / 4531 / 11 failed · FAIL	4543 / 4542 / 0 failed · PASS

The 11 red-at-base, in the authoritative tier:

test_tool_name_captured_from_string_input
test_tool_name_captured_from_object_form
test_capture_failure_is_distinguishable_from_a_real_tool
test_a_tool_literally_named_unknown_is_not_a_failure
test_no_argv_corruption
test_payload_is_valid_json_with_spaces_and_quotes
test_free_text_fields_with_spaces_survive
test_args_truncation_keeps_payload_parseable
test_duration_is_measured_not_hardcoded_zero
test_duration_absent_when_before_hook_never_fired
test_roundtrip_through_real_emit_and_collector

Green at both are the 2 harness negative controls + 2 positive controls (they drive the vendored buggy code, so they must be green in both — correct by construction), and 4 tier/structural guards + test_b64_values_are_real_base64, all labelled in-file as preconditions/invariant guards, not regression coverage.

Crucially, the harness negative controls now pass in the tier that gates merges. On the first push they were 2 of the 14 failures — so the harness had never been validated where it counts, and every surviving green there proved nothing.

Control pairs re-confirmed in the sandbox tier

All are pinned equalities that passed inside nix build:

counter	positive control	under test
corrupted_arg_count (argv entries with no =)	3 — universal, asserted unconditionally in both tiers	0
b64:payload= fragments from the buggy emitter	3 on the dev host (bash brace-expands) / 1 in the sandbox — branch chosen by the measured probe, both asserted	n/a
payload parses as JSON	every fragment raises JSONDecodeError (tier-independent)	valid JSON

Adjacent finding (not fixed here — would re-block this PR)

scripts/dl-router/tests is also absent from run-tests.sh's target list, and test_setup_script.py:134 writes a #!/usr/bin/env bash stub and execs it — the same latent sandbox failure. Adding that suite is a separate change; flagging it rather than expanding this PR.

🤖 Generated with Claude Code

Existing rows

activity.events is append-only with a 180d TTL. No backfill, no DELETE, no ALTER was attempted. The 2,699 malformed rows stay until they expire. Readers must tolerate both shapes: treat text='unknown' with a non-JSON payload as pre-fix garbage. The clean discriminator is payload.name_captured, which only exists post-fix.

Not verified

The fixed plugin has not been exercised by a real OpenCode session — that needs a home-manager switch plus an opencode run, and this task was explicitly told not to switch or restart anything. Every claim above rests on the real hook handlers being driven under node through the real emit into the real collector.parse_line, against the hook shape taken from the installed typings.