Skip to content

feat: add cost-aware workflow telemetry - #5

Open
Daaviddev wants to merge 6 commits into
Team-Volt:mainfrom
Daaviddev:codex/session-cost-report
Open

feat: add cost-aware workflow telemetry#5
Daaviddev wants to merge 6 commits into
Team-Volt:mainfrom
Daaviddev:codex/session-cost-report

Conversation

@Daaviddev

Copy link
Copy Markdown

What changed

  • normalize Windows fingerprint paths
  • enforce Terra-high routing for routine review agents, with explicit Sol exceptions
  • add metadata-only whole-session cost reporting for prompts, tools, delegation, validation, and review activity
  • record marketplace installation provenance

Why

This makes routine reviews cheaper by default and provides honest session-level usage proxies without claiming access to hidden token, billing, quota, or reasoning telemetry. The report stores counts and byte sizes only, not prompt, tool, or reviewer content.

Validation

  • node --test test/voltflow.test.mjs: 47 passed, 0 failed, 2 Windows capability skips
  • marketplace metadata parsed successfully as JSON
  • two independent Terra-high review lanes found no blocking issues

@davidrobertson

Copy link
Copy Markdown
Contributor

Thanks @Daaviddev for opening this PR and for the work on the cost-aware workflow telemetry. The direction makes sense, and the metadata-only reporting is a useful addition.

I pulled the exact head (c9a70b6) and ran the full test suite. The existing tests are green, but I found two issues that should be fixed before merge:

  1. hooks/hooks.json only runs PostToolUse for Bash, exec, and edit tools. Delegation, discovery, GitHub, and browser tools never reach recordToolUsage, so the new report undercounts the activity it says it measures. The telemetry test calls handleHook directly, which bypasses the installed hook matcher and misses this integration gap.

  2. routineReviewSpawnViolation() decides whether a task is a review from free-text keywords. The same work described as a "checker" or "inspection" skips the Terra-high requirement, and the eventual receipt is not bound to the spawned model or reasoning effort. This needs a durable binding between the review assignment, spawn metadata, and receipt instead of more synonyms in the regex.

Validation:

  • node --test test/voltflow.test.mjs: 49/49 passed
  • npm test: 49/49 passed
  • syntax checks, JSON parsing, and git diff --check: passed
  • targeted installed-hook matcher assertion: failed, confirming the telemetry wiring problem
  • no reportable security vulnerabilities found

I do not think this is ready to merge until those two paths are fixed and covered through the real hook configuration. Thanks again for putting the PR together and working on the new feature.

@Daaviddev
Daaviddev force-pushed the codex/session-cost-report branch from c9a70b6 to dc40229 Compare July 16, 2026 09:48
@Daaviddev

Copy link
Copy Markdown
Author

Resolved in ce17761.

  • hooks/hooks.json now uses PostToolUse matcher *, and the suite asserts the installed configuration so telemetry sees every tool category.
  • Reviewer dispatch now requires an explicit VOLTFLOW_REVIEW_ASSIGNMENT: <lane> <token> marker. The successful spawn binds its agent ID plus requested model and reasoning effort; only a matching receipt from that agent is accepted. The regression coverage includes non-keyword task names, wrong-agent rejection, and split-review concurrency.

Validation: node --test test/voltflow.test.mjs — 52 passed, 0 failed, 2 skipped.

@davidrobertson

Copy link
Copy Markdown
Contributor

Thanks for the quick follow-up. I pulled the exact head (ce177613) and confirmed the PostToolUse: "*" fix through the installed hook configuration; npm test is green (54/54), as are syntax, JSON, and diff checks.

I found three new blockers before this is merge-ready:

  1. bindReviewSpawn() accepts only tool_response.agent_id / agentId, but the ordinary v2 agents.spawn_agent result is { task_name, nickname }. Reproducing that real response shape leaves the assignment unbound, so the later SubagentStop receipt is ignored and the gate cannot pass. The new tests fabricate agent_id, which masks the production mismatch.
  2. SOL_EXCEPTION: direct user request is trusted from the agent-authored spawn message without checking user-originated state. A routine reviewer can therefore self-authorize Sol despite a neutral user prompt, bypassing the documented cost boundary.
  3. The whole-session report loses review waves between turns: start resets reviewHistory, while sessionReport() derives reviewRounds and repeated-review advice from that array. A completed bound review followed by a new prompt/start reports reviewRounds: 0.

Please also bind the existing final-line and failed-review regression tests before asserting their outcomes; they currently return early at the new bound-agent guard, so those named behaviors are no longer exercised.

Thanks again—the wildcard telemetry gap is fixed, but the reviewer lifecycle needs the actual v2 response shape and the session report needs durable review-wave accounting.

— Feedback provided by Codex (AI assistant)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants