feat(trace-review): /devflow:trace-review per-skill Langfuse regression report - #57
Merged
Conversation
…port Component (a) of the Langfuse trace-improvement loop spike (docs/plans/2026-06-30-langfuse-trace-improvement-loop.md §5). A READ-only weekly per-skill regression report over the local Langfuse trace store, built through the create-skill TDD flow (RED baseline -> GREEN -> determinism -> tessl). What it does: - Compares this week vs last week (rolling 7d) per skill: invocation count, per-execution error rate, p50/p95 latency, token cost, and score trend. - Flags regressions against balanced thresholds (error rate +10pp, score -0.05, cost +25%, p95 latency +25%; all overridable via TRACE_REVIEW_* env). - Emits a severity-tagged markdown report grouped by skill with exemplar trace links, following the alert-report/daily-report tone. Renders gracefully when there is no prior-week baseline (NEW rows) or no scores yet (no false "blocked"). Engine (deterministic, no AI judgement): - lib/trace-review.py joins observations to skills by traceId via the skill_name attribute Claude Code emits on claude_code.tool.execution spans (NOT a skill.name trace field), and attributes Langfuse's already-computed per-trace totalCost/latency to each skill. This fixes the two baseline failures the RED phase surfaced: wrong attribution key, and inflated cost/latency from naively summing mixed span types. - Error rate is per tool execution (failed claude_code.tool.execution spans over total), so one failed command in a long session is a small fraction, not 100%. Permission gates (blocked_on_user, decision=reject) are never errors. - Excludes the analysis skill's own activations (skill_name == devflow:trace-review) + an optional session id, since service.name is identical for every CC session. CLI + extensibility: - devflow trace-review run [--json|--output|--window] and a provider-agnostic scheduler (devflow trace-review schedule --backend cron|claude). cron is the portable default (works for Claude Code and OpenCode, which has no native scheduler); claude emits a cloud-routine spec. New seam in lib/utils.sh: detect_agent_provider + agent_invoke_cmd, mirroring detect_vcs_provider. A future provider slots in with one arm in each. The skill offers scheduling on first run via AskUserQuestion. Architectural truth stated in the skill: Claude Code skills load from disk, never from Langfuse at runtime. This is the READ half of the loop; the OTel collector is the WRITE half. They compose, not duplicate. Tests + gates: - tests/unit/trace-review.bats: provider seam, scheduler dispatch, and the deterministic regression math across all flag branches (15 tests, network-free). - skills/trace-review/determinism.promptfooconfig.yaml: 6 output-shape asserts against the engine directly (6/6 pass). Dev-only harness kept canonical-only, not mirrored into the plugin. - tessl review score: 97 (gate 85). Skill lives in the 3 synced devflow locations (skills/, devflow-plugin/skills/, devflow-plugin/commands/) + registry.json (observability category). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ds, self-tests Addresses the multi-perspective review of PR #57 (correctness + test-adequacy + prose agents). All changes are lib/tests only; no SKILL.md touched. Correctness / output-trust: - Correct the docstring/comments: skill_name lands on `claude_code.tool` spans, NOT `claude_code.tool.execution`. The join already scans every observation so behavior was right; the comment was factually wrong (verify-first against live Langfuse: skill_name observed on a claude_code.tool span). - Hold the `(unattributed)` bucket OUT of the per-skill regression ranking and surface it as a footnote instead. It is ordinary non-skill tool activity, not a skill; previously it dominated the table with a meaningless mixed cost/latency row (31/33 traces, $11.88, p95 301s on the live store). - _die() now writes the error JSON to stderr, not stdout, so a failed run under the `--output FILE` redirect no longer leaves a bogus report file behind. - Guard `t.get('id')` / timestamp in _aggregate and the noise loop: one malformed trace no longer KeyErrors the whole run (which the top-level except masked as a generic 'Langfuse API error'). - _paginate treats a missing `meta` block as an error instead of silently truncating to page 1, and warns to stderr when the hard cap is hit (raised to 20000). Prevents a partial pull being mistaken for a complete one. - .sh: guard the `--project` flag so a trailing value-less --project does not abort the run under set -euo pipefail. Attribution-coverage caveat added to the engine docstring: per-skill grouping is only as complete as skill_name in the traces, which Claude Code stamps only when a skill is active. Tests (the module's core functions were previously untested): - New unit tests for _build_trace_skill_map (first-skill_name-wins join), _trace_exec_stats (permission-reject is never an error), _aggregate (traceId join + per-trace totalCost/latency passthrough + unattributed counting + window bound), and _flag_regressions holding (unattributed) out of the ranking. - _pct test now asserts pinned nearest-rank values instead of a tautological self-equality. - eval-run.sh comment relabeled: it is a SHAPE smoke test against live data, not a data-reproducible eval (that lives in the bats fixtures + TRACE_REVIEW_NOW). 19/19 trace-review bats + 94/94 full unit suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new /devflow:trace-review capability to generate a deterministic, week-over-week, per-skill regression report from the local Langfuse trace store, plus CLI wiring, scheduling support, and unit tests.
Changes:
- Introduces a deterministic Python aggregation engine (
lib/trace-review.py) and a Bash CLI wrapper (lib/trace-review.sh) with run/schedule/unschedule subcommands. - Adds an agent-provider seam in
lib/utils.sh(detect_agent_provider,agent_invoke_cmd) and wires a newtrace-reviewcommand intobin/devflow. - Adds a new skill (plus plugin/command mirrors), registry entry, and unit tests covering regression flagging + scheduling/provider seams.
Reviewed changes
Copilot reviewed 14 out of 16 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
tests/unit/trace-review.bats |
Adds network-free unit coverage for regression math and wrapper/provider/scheduler behavior. |
skills/trace-review/TEMPLATES.md |
Documents the pinned report format and extension/scheduler seams. |
skills/trace-review/SKILL.md |
Defines the /devflow:trace-review skill contract and operating procedure. |
skills/trace-review/requirements.json |
Declares dependencies for the skill’s preflight/deps checks. |
skills/trace-review/eval-run.sh |
Adds a promptfoo exec provider to run the CLI/engine for shape assertions. |
skills/trace-review/determinism.promptfooconfig.yaml |
Pins deterministic output “shape” assertions for JSON/markdown outputs. |
skills/registry.json |
Registers the new trace-review skill and reformats several existing entries. |
lib/utils.sh |
Adds agent-provider detection and headless invocation command generation. |
lib/trace-review.sh |
Implements the devflow trace-review CLI wrapper plus cron/Claude routine scheduling. |
lib/trace-review.py |
Implements deterministic Langfuse data pulls, attribution, aggregation, regression flagging, and markdown rendering. |
devflow-plugin/skills/trace-review/TEMPLATES.md |
Plugin-mirrored copy of the skill report-format documentation. |
devflow-plugin/skills/trace-review/SKILL.md |
Plugin-mirrored copy of the skill definition/contract. |
devflow-plugin/skills/trace-review/requirements.json |
Plugin-mirrored copy of the skill dependency declaration. |
devflow-plugin/commands/trace-review.md |
Adds the command surface for the plugin command palette. |
bin/devflow |
Wires in the new trace-review command and help output entry. |
.gitignore |
Ignores Python bytecode artifacts for the new Python engine. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+110
to
+115
| local provider; provider="$(detect_agent_provider)" | ||
| local report_dir="${HOME}/.devflow/trace-review" | ||
| mkdir -p "$report_dir" | ||
| local out_file="${report_dir}/\$(date -u +%Y-%m-%d).md" | ||
| local cron_marker="# devflow-trace-review:${project_dir}" | ||
| local cron_entry="${cron_expr} cd ${project_dir} && ${devflow_bin} trace-review run --output ${out_file} >> ${report_dir}/trace-review.log 2>&1" |
Comment on lines
+63
to
+65
| --since-now) now="$2"; shift 2 ;; # ISO timestamp; pins "now" for reproducible runs | ||
| --project) shift; [[ $# -gt 0 ]] && shift ;; # accepted for cron-entry symmetry; value ignored (engine is project-agnostic). Guarded so a trailing --project with no value does not abort under set -euo pipefail. | ||
| *) shift ;; |
Comment on lines
+1
to
+11
| { | ||
| "skill": "trace-review", | ||
| "required": [ | ||
| { "name": "python3", "check": "python3 --version", "why": "runs the deterministic trace aggregation engine (lib/trace-review.py)", "install": "https://www.python.org (usually preinstalled on macOS/Linux)" }, | ||
| { "name": "curl", "check": "curl --version", "why": "Langfuse reachability check + REST queries", "install": "preinstalled on macOS/Linux" } | ||
| ], | ||
| "optional": [ | ||
| { "name": "langfuse", "check": "curl -fsS -o /dev/null --connect-timeout 4 ${LANGFUSE_HOST:-http://localhost:3100}/api/public/health", "why": "the trace store this report reads from", "install": "devflow up", "degrade": "STOP and tell the user to run `devflow up` - there is no data to report on without it" }, | ||
| { "name": "langfuse-keys", "check": "test -n \"$LANGFUSE_PUBLIC_KEY\" -a -n \"$LANGFUSE_SECRET_KEY\" || grep -q LANGFUSE_PUBLIC_KEY \"$HOME/.config/zsh/secrets\"", "why": "Basic-auth to the Langfuse public API", "install": "add LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY to ~/.config/zsh/secrets", "degrade": "STOP - the engine cannot authenticate without keys" } | ||
| ] | ||
| } |
Comment on lines
+1
to
+11
| { | ||
| "skill": "trace-review", | ||
| "required": [ | ||
| { "name": "python3", "check": "python3 --version", "why": "runs the deterministic trace aggregation engine (lib/trace-review.py)", "install": "https://www.python.org (usually preinstalled on macOS/Linux)" }, | ||
| { "name": "curl", "check": "curl --version", "why": "Langfuse reachability check + REST queries", "install": "preinstalled on macOS/Linux" } | ||
| ], | ||
| "optional": [ | ||
| { "name": "langfuse", "check": "curl -fsS -o /dev/null --connect-timeout 4 ${LANGFUSE_HOST:-http://localhost:3100}/api/public/health", "why": "the trace store this report reads from", "install": "devflow up", "degrade": "STOP and tell the user to run `devflow up` - there is no data to report on without it" }, | ||
| { "name": "langfuse-keys", "check": "test -n \"$LANGFUSE_PUBLIC_KEY\" -a -n \"$LANGFUSE_SECRET_KEY\" || grep -q LANGFUSE_PUBLIC_KEY \"$HOME/.config/zsh/secrets\"", "why": "Basic-auth to the Langfuse public API", "install": "add LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY to ~/.config/zsh/secrets", "degrade": "STOP - the engine cannot authenticate without keys" } | ||
| ] | ||
| } |
Comment on lines
+205
to
+210
| """Compare this vs last per skill, return regression flags + severity.""" | ||
| rows = [] | ||
| # (unattributed) is not a skill - it holds ordinary tool activity with no skill_name. | ||
| # Keep it in the totals (see main), but never rank it as a regressing skill. | ||
| skills = sorted((set(this_m) | set(last_m)) - {UNATTRIBUTED}) | ||
| for skill in skills: |
Comment on lines
+56
to
+61
| - `skill_name` is emitted on `claude_code.tool.execution` spans (when a Skill tool | ||
| fires). It is **not** a top-level trace field, and is **not** `skill.name`. | ||
| - Cost / tokens live on sibling `claude_code.llm_request` spans under the same trace. | ||
| - The engine maps `traceId → skill_name`, then attributes each trace's | ||
| Langfuse-computed `totalCost` + `latency` (already aggregated, correct) to that skill. | ||
| Traces with no skill activation fall into `(unattributed)`. |
Comment on lines
+56
to
+61
| - `skill_name` is emitted on `claude_code.tool.execution` spans (when a Skill tool | ||
| fires). It is **not** a top-level trace field, and is **not** `skill.name`. | ||
| - Cost / tokens live on sibling `claude_code.llm_request` spans under the same trace. | ||
| - The engine maps `traceId → skill_name`, then attributes each trace's | ||
| Langfuse-computed `totalCost` + `latency` (already aggregated, correct) to that skill. | ||
| Traces with no skill activation fall into `(unattributed)`. |
…le-5 (+ [1m] variants) Extends docker/langfuse-seed-models.sh beyond the single claude-opus-4-8[1m] entry so /devflow:trace-review's cost column is non-zero for every Claude model Claude Code emits. Two gaps closed: - The 1M-context [1m] suffix. Claude Code appends [1m] on 1M-context requests; Langfuse's built-in price regexes end at $ so <model>[1m] never matches and cost computes to 0. Added a [1m]-anchored entry for opus-4-8 (already present), sonnet-5, and fable-5. - New model IDs. Langfuse's built-in list lags Anthropic releases, so claude-sonnet-5 and claude-fable-5 resolved to no price. Seeded both, plus claude-haiku-4-5. Rates (per-token USD, published Anthropic per-1M / 1e6; cache_read = 0.1x input, cache_write 5m = 1.25x input): opus-4-8 $5/$25, sonnet-5 $3/$15, haiku-4-5 $1/$5, fable-5 $10/$50. Sonnet 5 seeded at STANDARD $3/$15, not the intro $2/$10 (through 2026-08-31): for a cost-regression tool, the lower intro rate would make every sonnet-5 cost jump on 2026-09-01 and read as a false cost-up regression; the standard rate avoids that. The >200K long-context premium tier is not separately modeled (Langfuse uses one flat per-model price), so [1m] entries carry the same base rates as their standard sibling. Also fixed two latent bugs in the existing script's idempotency check: the models-list GET used limit=500 (Langfuse caps limit at 100, so the call errored and the grep silently never matched, forcing every seed to POST and rely on the 400 duplicate fallback), and the substring grep would false-skip claude-fable-5 as a prefix of claude-fable-5-1m. Now paginates the list and matches modelName exactly. Verified against live Langfuse: synthetic 100k-in/100k-out generations for all six IDs compute calculatedTotalCost exactly matching the published rates (sonnet-5 1.80, haiku-4-5 0.60, fable-5 6.00, opus-4-8[1m] 3.00). Idempotent re-run skips all six. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…h -> hook sidecar)
The engine attributed traces by skill_name alone. On real data that covers only
3/57 traces (2 skill_name spans + 1 leading-slash prompt), because Claude Code
stamps skill_name only when a skill is active and does not emit it as a span
attribute for most sessions. Everything else fell into (unattributed), so the
per-skill report was near-empty and could not answer 'which skill regressed'.
Replace _build_trace_skill_map with _attribute_traces, a per-trace precedence
ladder (best signal wins):
1. skill_name - the skill_name attribute on any span (incl. tool_name=='Skill').
2. slash_command - leading-slash user_prompt on the interaction span; first token,
slash stripped (recovers a /devflow:review run that carried no
skill_name span).
3. hook_sidecar - session_id + timestamp window join against an enrichment JSONL
(written by a hook, landed in the next commit). Because one
Langfuse trace equals one Claude Code interaction (verified: one
session fans out to 9 traces), the join binds each trace to the
LATEST skill invoked at-or-before that trace's timestamp in the
same session; a flat session->skill map would mis-attribute every
turn of a multi-skill session. Forward-only: attributes only traces
produced after the hook is installed.
_attribute_traces returns (skill_map, source_map) so the report can print a
per-source coverage breakdown, making thin attribution visible rather than hidden.
The doc gains an 'attribution' block and the markdown shows 'N/M traces mapped
(skill_name X, slash Y, hook Z)'.
Rung 3 code and its windowed-join semantics are proven now by synthetic bats
fixtures (multi-skill-per-session, at-or-before binding, malformed-line tolerance);
the sidecar is inert until the hook writes it. 23/23 unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the WRITE side of the attribution ladder's sidecar rung. A PreToolUse hook
(matcher: Skill) appends {session_id, ts, skill} to ~/.devflow/skill-activations.jsonl
on every Skill-tool invocation, so trace-review can attribute a session's traces to
the skill even when Claude Code emits no skill_name span. devflow init registers it.
Why a hook + sidecar (not a span attribute): a hook is a separate child process and
cannot inject an attribute into Claude Code's in-flight OTel export (resource/env
attrs are fixed at exec). The session_id + timestamp sidecar join (engine rung 3,
prior commit) is the reachable enrichment path.
Why PreToolUse:Skill specifically (precision over recall): tool_input.skill is the
exact skill name at invocation, and it excludes non-skill slash commands (/compact,
/clear) that a UserPromptSubmit capture would wrongly record and mis-attribute
unrelated traces to. A missed row just leaves a trace (unattributed); a wrong row
would corrupt a real skill's numbers. The hook always exits 0 and never blocks the
tool call; it logs only session_id + UTC ts + skill name (no prompt text, no args,
no secrets).
Forward-only: it can only attribute traces produced after install, so the report's
coverage grows over time rather than retroactively. Verified end to end: Skill
payload writes one row, Bash/malformed payloads write none and never fail; the
engine's rung 3 then attributes a trace to that skill via hook_sidecar.
Docs corrected: the command md / SKILL.md / TEMPLATES.md said skill_name lands on
claude_code.tool.execution spans; it lands on claude_code.tool (verified against
live traces). Updated to describe the full ladder. 25/25 unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rowth Join key verified against live data: the CC hook's session_id equals the Langfuse trace sessionId (this session's CC UUID appears as a trace sessionId; all sessionIds are UUID-shaped), so the sidecar rows will join. Record the two honest heuristic limits in the engine docstring: rung 3 propagates a skill forward to later same-session turns rather than backfilling its own invocation turn (rung 1 handles that), and it charges all subsequent turns to the last skill until another fires. Also note the sidecar is read whole and does not self-rotate. Docstring-only; 25/25 tests still pass.
…ls-sync)
Merge main brought in the single-source skills generator (build-skills.sh). Regenerate
the devflow-plugin/{commands,skills}/trace-review copies + plugin.json skills array from
the authored skills/trace-review/ source so the skills-check drift gate passes. Content
is the attribution-ladder edits already made to the source; no behavior change.
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Adds
/devflow:trace-review— a READ-only weekly per-skill regression report over the local Langfuse trace store. Component (a) of the trace-improvement loop spike (§5). Built via the/create-skillTDD flow (RED baseline -> GREEN -> determinism -> tessl). tessl review score: 97 (gate 85).What it does
Per skill, this week vs last week (rolling 7d): invocation count, per-execution error rate, p50/p95 latency, token cost, score trend. Flags regressions (error rate +10pp, score -0.05, cost +25%, p95 +25%) and emits a severity-tagged markdown report grouped by skill with exemplar trace links. Renders gracefully on no-baseline / no-scores (NEW rows, never "blocked").
Change table
lib/trace-review.pytraceIdvia theskill_nameattribute onclaude_code.tool.executionspans; attributes Langfuse per-tracetotalCost/latency; per-execution error rate; window diff + flags + markdown renderlib/trace-review.sh,bin/devflowdevflow trace-review run|schedule|unschedulelib/utils.shdetect_agent_provider+agent_invoke_cmd(sibling ofdetect_vcs_provider)skills/trace-review/{SKILL.md,TEMPLATES.md,requirements.json,determinism.promptfooconfig.yaml,eval-run.sh}+ plugin mirror + command +registry.json/devflow:trace-reviewsurface (3 synced locations)tests/unit/trace-review.batsKey design decisions
skill_namelives on tool-execution spans (not askill.nametrace field); cost/latency on siblingllm_requestspans. The engine joins bytraceIdand uses Langfuse's already-aggregated per-trace totals (the RED baseline got both wrong: assumedskill.name, and inflated totals by summing mixed span types).cronis the portable default (Claude Code + OpenCode, which has no native scheduler);claudeemits a cloud-routine spec. Asked on first run viaAskUserQuestion. A future provider adds one arm to each seam.Verification
make test(smoke) +bats tests/unit/(90 tests incl. 15 new) greenNot in this PR (follow-ups from the spike)
Components (b) auto-mirror skills -> Langfuse prompt-mgmt, (c) trace-derived datasets + eval gate, (d) score-drop automations.
🤖 Generated with Claude Code