Skip to content

feat(telemetry): agent-mode stamping — non-production rows out of the KPIs (ruler-1) - #73

Merged
Cranot merged 1 commit into
mainfrom
feat/ruler-agent-mode
Jul 16, 2026
Merged

feat(telemetry): agent-mode stamping — non-production rows out of the KPIs (ruler-1)#73
Cranot merged 1 commit into
mainfrom
feat/ruler-agent-mode

Conversation

@Cranot

@Cranot Cranot commented Jul 16, 2026

Copy link
Copy Markdown
Owner

Measurement integrity. Compile telemetry (.roam/compile-runs.jsonl) feeds the L1-rate and latency KPIs, but benchmarks, corpus sweeps, the diff tool, and the pytest battery wrote UNstamped (unknown) rows indistinguishable from real traffic — skewing every reported number.

New src/roam/plan/agent_mode.py: an agent_mode(value) context manager (DRYs the cache-warmer's hand-rolled set/restore), NON_PRODUCTION_MODES, is_non_production(). ROAM_AGENT_MODE is dual-purpose but policy only honors VALID_MODES, so a bench/test stamp is inert to policy (same as the pre-existing compile_cache_build stamp).

  • Writers stamped: bench-compile (child env), compiler-corpus / dispatch-trace / envelope-diff (in-process), the floor test (60 compiles into the real repo), and the UPS hook stamps hook via setdefault (an explicit policy mode is preserved) so real traffic leaves the mixed unknown bucket.
  • Reader: compile-stats default-excludes non-production rows from the KPI aggregates + discloses the count; --include-bench keeps them; --by-mode always shows the full split; dispatch-trace's fast-path refuses to serve a non-production row as production data. unknown stays IN (historically mixed — disclosed, not dropped).

12 tests; prepush 7/7 green.

@Cranot
feat(telemetry): agent-mode stamping — keep non-production rows out o…
89540bf 
…f the KPIs (ruler-1)

Measurement integrity (doctrine: currency measured on clean signal). Every
compile appends an agent_mode-stamped row to .roam/compile-runs.jsonl that feeds
the L1-rate and latency KPIs — but benchmarks, corpus sweeps, the diff tool, and
the pytest battery were all writing UNstamped ('unknown') rows indistinguishable
from real traffic, silently skewing every reported number.

New src/roam/plan/agent_mode.py centralizes the stamp: an agent_mode(value)
context manager (DRYs the set/restore the cache-warmer hand-rolled) + the
NON_PRODUCTION_MODES set + is_non_production(). ROAM_AGENT_MODE is dual-purpose
(also the mode-policy signal), but policy only honors VALID_MODES, so a telemetry
stamp like 'bench'/'test' is inert to policy — exactly how the pre-existing
'compile_cache_build' stamp already coexists.

Writers stamped: bench-compile (child env), compiler-corpus, dispatch-trace,
envelope-diff (in-process context), the pytest floor test (writes 60 compiles
into the real repo), and the UPS hook stamps 'hook' (setdefault — an explicit
policy mode is preserved) so real traffic is no longer lumped into 'unknown'.

Reader: compile-stats default-EXCLUDES non-production rows from the KPI
aggregates (summary/L1/latency/by-procedure/top-misses), discloses the excluded
count, and adds --include-bench to keep them. --by-mode always shows the full
split. dispatch-trace's fast-path now refuses to serve a non-production row as
production probe data. 'unknown' stays IN the KPIs (a historically MIXED bucket
— dropping it would hide real traffic; disclosed instead).

12 tests; prepush 7/7 green.
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown

roam-code Analysis

Mode: incremental (changed-only) — base 1b5271f64865de57947452e6414628be2c7d7780, 29 changed+dependent files

Health Score: 79/100 FAIR

health: Fair codebase (79/100) — 48 critical, 7 warnings, focus: god_components
pr-risk: Moderate risk (40/100) — review recommended (risk_level medium) (driver: hotspot_score)

Health Metrics

Metric Value
Health Score 79/100
Tangle Ratio 0%
Propagation Cost 0.0015
Total Issues 65

PR Risk

Metric Value
Risk Score 40/100

Quality Gate: PASSED

Gate expression: health_score>=50

SARIF Upload

Metric Value
Category roam-code-self-analysis/self-analysis/py3.12
Results Uploaded 82
Full analysis output

health

{
  "_meta": {
    "cache_ttl_s": 300,
    "cacheable": true,
    "index_age_s": 3,
    "index_status": {
      "dirty_files": 0,
      "fresh": false,
      "head_commit": "7f8fa3c368b7",
      "hint": "index latest commit 89540bf95ef6 != HEAD 7f8fa3c368b7 — git-derived metrics (commits, churn, co-change, weather) may be stale. Run `roam index --force`.",
      "indexed_commit": "89540bf95ef6"
    },
    "latency_ms": null,
    "response_tokens": 4083,
    "roam_version": "13.9.0",
    "timestamp": "2026-07-16T09:56:13Z"
  },
  "actionable_count": 15,
  "actionable_cycles": 0,
  "agent_contract": {
    "confidence": null,
    "facts": [
      "Fair codebase (79/100) — 48 critical, 7 warnings, focus: god_components",
      "health score 79",
      "tangle ratio 0.0",
      "0.0015 propagation cost findings",
      "issue count 65"
    ],
    "next_commands": [
      "roam debt",
      "roam trends --days 30"
    ],
    "risks": []
  },
  "algebraic_connectivity": null,
  "algebraic_connectivity_available": false,
  "bottleneck_thresholds": {
    "p70": 1146.8,
    "p90": 9460.8,
    "population": 1163,
    "utility_multiplier": 1.5
  },
  "category_severity": {
    "bottlenecks": {
      "critical": 15,
      "info": 0,
      "warning": 0
    },
    "cycles": {
      "critical": 0,
      "info": 0,
      "warning": 0
    },
    "god_components": {
      "critical": 33,
      "info": 10,
      "warning": 7
    },
    "layer_violations": {
      "critical": 0,
      "info": 0,
      "warning": 0
    }
  },
  "command": "health",
  "cycles_actionable": 0,
  "cycles_total": 17,
  "framework_filtered": 0,
  "health_score": 79,
  "ignored_cycles": 17,
  "imported_coverable_lines": 0,
  "imported_coverage_files": 0,
  "imported_coverage_pct": null,
  "imported_covered_lines": 0,
  "index_status": {
    "dirty_files": 0,
    "fresh": false,
    "head_commit": "7f8fa3c368b7",
    "hint": "index latest commit 89540bf95ef6 != HEAD 7f8fa3c368b7 — git-derived metrics (commits, churn, co-change, weather) may be stale. Run `roam index --force`.",
    "indexed_commit": "89540bf95ef6"
  },
  "issue_count": 65,
  "list_counts": {
    "bottlenecks": 15,
    "cycle_break_suggestions": 0,
    "cycles": 17,
    "god_components": 50,
    "layer_violations": 0,
    "next_steps": 2,
    "score_breakdown": 5
  },
  "project": "roam-code",
  "propagation_cost": 0.0015,
  "schema": "roam-envelope-v1",
  "schema_version": "1.1.0",
  "severity": {
    "critical": 48,
    "info": 27,
    "warning": 7
  },
  "summary": {
    "actionable_cycles": 0,
    "algebraic_connectivity": null,
    "algebraic_connectivity_available": false,
    "category_severity": {
      "bottlenecks": {
        "critical": 15,
        "info": 0,
        "warning": 0
      },
      "cycles": {
        "critical": 0,
        "info": 0,
        "warning": 0
      },
      "god_components": {
        "critical": 33,
        "info": 10,
        "warning": 7
      },
      "layer_violations": {
        "critical": 0,
        "info": 0,
        "warning": 0
      }
    },
    "cycles_actionable": 0,
    "cycles_definition": "Cycle counts derived from `roam.graph.cycles.find_cycles(G, min_size=2)` on the symbol graph. `cycles_total` = all SCCs of size >= 2; `cycles_actionable` = SCCs spanning >=2 files AND no test files (same-file and test-only cycles are informational). Run `roam health` for the per-cycle breakdown.",
    "cycles_total": 17,
    "detail_available": true,
    "god_components": 50,
    "god_components_definition": "God components: symbols where `(in_degree + out_degree) > 20` from the `graph_metrics` table, with utility-aware severity bands (standard >50=CRITICAL >30=WARNING; utility >150=CRITICAL >90=WARNING). Run `roam health` for the per-symbol breakdown. Legacy aliases: `god_objects` (fingerprint), `god_classes` (rules).",
    "health_score": 79,
    "health_score_definition": "weighted geometric mean (0-100) of 5 sigmoid health factors: tangle_ratio, god_components, bottlenecks, layer_violations, file_health (+coverage if available).",
    "ignored_cycles": 17,
    "imported_coverage_files": 0,
    "imported_coverage_pct": null,
    "issue_count": 65,
    "partial_success": true,
    "preserved_list_truncations": {},
    "propagation_cost": 0.0015,
    "severity": {
      "critical": 48,
      "info": 27,
      "warning": 7
    },
    "tangle_ratio": 0,
    "tangle_ratio_definition": "fraction of symbols inside non-trivial SCCs; higher = more cyclic coupling.",
    "total_cycles": 17,
    "truncated": true,
    "verdict": "Fair codebase (79/100) — 48 critical, 7 warnings, focus: god_components",
    "warnings_out": [
      "health_algebraic_connectivity_warning:RuntimeWarning:algebraic_connectivity compute failed (ModuleNotFoundError): No module named 'numpy'; returning 0.0 sentinel — value is NOT a legitimate disconnected-graph reading"
    ]
  },
  "tangle_ratio": 0,
  "total_cycles": 17,
  "utility_count": 35,
  "version": "13.9.0",
  "warnings_out": [
    "health_algebraic_connectivity_warning:RuntimeWarning:algebraic_connectivity compute failed (ModuleNotFoundError): No module named 'numpy'; returning 0.0 sentinel — value is NOT a legitimate disconnected-graph reading"
  ]
}

pr-risk

{
  "_meta": {
    "cache_ttl_s": 60,
    "cacheable": true,
    "index_age_s": 5,
    "index_status": {
      "dirty_files": 0,
      "fresh": false,
      "head_commit": "7f8fa3c368b7",
      "hint": "index latest commit 89540bf95ef6 != HEAD 7f8fa3c368b7 — git-derived metrics (commits, churn, co-change, weather) may be stale. Run `roam index --force`.",
      "indexed_commit": "89540bf95ef6"
    },
    "latency_ms": null,
    "response_tokens": 1403,
    "roam_version": "13.9.0",
    "timestamp": "2026-07-16T09:56:15Z"
  },
  "actor": null,
  "agent_contract": {
    "confidence": null,
    "facts": [
      "Moderate risk (40/100) — review recommended (risk_level medium) (driver: hotspot_score)",
      "risk score 40",
      "2 risk rank findings",
      "9 changed files",
      "228 lines added"
    ],
    "next_commands": [],
    "risks": [
      "pr-risk: moderate (40/100) on 1b5271f64865de57947452e6414628be2c7d7780..HEAD — driver: hotspot_score"
    ]
  },
  "author": null,
  "blast_radius_pct": 0.6,
  "bus_factor_risk": 0.86,
  "change_shape": "mixed",
  "changed_files": 9,
  "closest_historical_pattern": [
    "src/roam/commands/cmd_bench.py",
    "src/roam/commands/cmd_compile_stats.py",
    "src/roam/commands/cmd_compiler_corpus.py",
    "src/roam/commands/cmd_dispatch_trace.py",
    "src/roam/commands/cmd_envelope_diff.py",
    "src/roam/commands/cmd_hooks.py",
    "src/roam/plan/agent_mode.py",
    "tests/test_agent_mode_telemetry.py",
    "tests/test_l1_rate_floor.py"
  ],
  "closest_similarity": 1,
  "cluster_spread": 0,
  "clusters_touched": 48,
  "command": "pr-risk",
  "coupling_score": 0.1,
  "dead_code": [
    {
      "file": "src/roam/commands/cmd_hooks.py",
      "kind": "function",
      "name": "install"
    },
    {
      "file": "src/roam/commands/cmd_hooks.py",
      "kind": "function",
      "name": "uninstall"
    },
    {
      "file": "src/roam/commands/cmd_hooks.py",
      "kind": "function",
      "name": "status"
    },
    {
      "file": "src/roam/commands/cmd_hooks.py",
      "kind": "function",
      "name": "claude_setup"
    }
  ],
  "dead_exports": 4,
  "familiarity": {
    "avg_familiarity": 1,
    "files": [],
    "files_assessed": 0
  },
  "findings": [
    {
      "claim": "pr-risk: moderate (40/100) on 1b5271f64865de57947452e6414628be2c7d7780..HEAD — driver: hotspot_score",
      "confidence": "heuristic",
      "evidence": {
        "actor": null,
        "author": null,
        "blast_radius_pct": 0.6,
        "bus_factor_risk": 0.857,
        "changed_files_count": 9,
        "commit_range": "1b5271f64865de57947452e6414628be2c7d7780..HEAD",
        "coupling_score": 0.097,
        "created_at_epoch": 1784195775,
        "diff_id": "5e8a320f52c0",
        "familiarity_risk": 0,
        "file_list": [
          "src/roam/commands/cmd_bench.py",
          "src/roam/commands/cmd_compile_stats.py",
          "src/roam/commands/cmd_compiler_corpus.py",
          "src/roam/commands/cmd_dispatch_trace.py",
          "src/roam/commands/cmd_envelope_diff.py",
          "src/roam/commands/cmd_hooks.py",
          "src/roam/plan/agent_mode.py",
          "tests/test_agent_mode_telemetry.py",
          "tests/test_l1_rate_floor.py"
        ],
        "hotspot_score": 0.752,
        "label": "1b5271f64865de57947452e6414628be2c7d7780..HEAD",
        "lines_added": 228,
        "lines_removed": 11,
        "minor_risk": 0,
        "novelty_score": 0,
        "reductive_change": false,
        "risk_level": "moderate",
        "risk_score": 40,
        "staged": false,
        "test_coverage_pct": 85.7,
        "top_driver": "hotspot_score"
      },
      "finding_id_str": "pr-risk:composite-risk-score:5e8a320f52c0",
      "kind": "pr-risk:composite-risk-score",
      "severity": "medium",
      "source_detector": "pr-risk",
      "source_version": "1.0.0",
      "subject_id": null,
      "subject_kind": "commit"
    }
  ],
  "hotspot_score": 0.75,
  "label": "1b5271f64865de57947452e6414628be2c7d7780..HEAD",
  "layer_spread": 0.29,
  "layers_touched": 9,
  "lines_added": 228,
  "lines_removed": 11,
  "minor_risk": {
    "files": [],
    "files_assessed": 0,
    "minor_files": 0
  },
  "novelty_score": 0,
  "per_file": [
    {
      "blast": 137,
      "churn": 508,
      "is_test": false,
      "lines_added": 28,
      "lines_removed": 4,
      "path": "src/roam/commands/cmd_compile_stats.py",
      "symbols": 12
    },
    {
      "blast": 86,
      "churn": 1730,
      "is_test": false,
      "lines_added": 4,
      "lines_removed": 0,
      "path": "src/roam/commands/cmd_hooks.py",
      "symbols": 39
    },
    {
      "blast": 67,
      "churn": 1027,
      "is_test": false,
      "lines_added": 3,
      "lines_removed": 1,
      "path": "src/roam/commands/cmd_envelope_diff.py",
      "symbols": 44
    },
    {
      "blast": 63,
      "churn": 62,
      "is_test": false,
      "lines_added": 62,
      "lines_removed": 0,
      "path": "src/roam/plan/agent_mode.py",
      "symbols": 11
    },
    {
      "blast": 62,
      "churn": 550,
      "is_test": false,
      "lines_added": 12,
      "lines_removed": 5,
      "path": "src/roam/commands/cmd_dispatch_trace.py",
      "symbols": 13
    },
    {
      "blast": 57,
      "churn": 429,
      "is_test": false,
      "lines_added": 3,
      "lines_removed": 1,
      "path": "src/roam/commands/cmd_compiler_corpus.py",
      "symbols": 8
    },
    {
      "blast": 47,
      "churn": 783,
      "is_test": false,
      "lines_added": 7,
      "lines_removed": 0,
      "path": "src/roam/commands/cmd_bench.py",
      "symbols": 18
    },
    {
      "blast": 4,
      "churn": 103,
      "is_test": true,
      "lines_added": 103,
      "lines_removed": 0,
      "path": "tests/test_agent_mode_telemetry.py",
      "symbols": 9
    },
    {
      "blast": 0,
      "churn": 68,
      "is_test": true,
      "lines_added": 6,
      "lines_removed": 0,
      "path": "tests/test_l1_rate_floor.py",
      "symbols": 5
    }
  ],
  "project": "roam-code",
  "reductive_change": false,
  "reductive_discount_applied": false,
  "risk_level": "moderate",
  "risk_level_canonical": "medium",
  "risk_rank": 2,
  "risk_score": 40,
  "schema": "roam-envelope-v1",
  "schema_version": "1.1.0",
  "suggested_reviewers": [
    {
      "actor": "Cranot",
      "author": "Cranot",
      "lines": 4825
    },
    {
      "actor": "Dimitris",
      "author": "Dimitris",
      "lines": 11
    }
  ],
  "summary": {
    "change_shape": "mixed",
    "changed_files": 9,
    "findings_count": 1,
    "lines_added": 228,
    "lines_removed": 11,
    "partial_success": false,
    "risk_level": "moderate",
    "risk_level_canonical": "medium",
    "risk_rank": 2,
    "risk_score": 40,
    "score_classification": "classified",
    "verdict": "Moderate risk (40/100) — review recommended (risk_level medium) (driver: hotspot_score)"
  },
  "test_coverage_pct": 85.7,
  "total_clusters": 13080,
  "total_layers": 31,
  "version": "13.9.0",
  "warnings_out": []
}

roam-code analysis | Commands: health pr-risk

@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown

Roam Agent Review

Verdict: SAFE (risk_level low)

blast-radius 0/100 · ai-likelihood 28/100 · rule violations 0 · critique high-severity 0

Verdict: SAFE. All structural signals clean at the configured thresholds.

Next steps

  • No structural concerns at the configured thresholds. Standard review still recommended.

Powered by roam-code — Apache 2.0, 100% local. Customize thresholds in .roam/rules.yml. Docs.

@Cranot
Cranot merged commit 19e74bd into main Jul 16, 2026
11 checks passed
@Cranot
Cranot deleted the feat/ruler-agent-mode branch July 16, 2026 10:21
This was referenced Jul 16, 2026
Merged
Merged
Cranot added a commit that referenced this pull request Jul 16, 2026
@Cranot
fix(compile-stats): disclose all-non-production telemetry in human ou…
7c743fb 
…tput (#74)

Fresh-eyes follow-up to the ruler-1 KPI filter (#73). When a repo's telemetry
is 100% non-production (all bench/corpus/trace/test rows), the human path hit
the 'no telemetry yet' early-return and printed '(no .roam/compile-runs.jsonl
under ...)' — doubly wrong: the file exists AND has rows, they were just
filtered out of the KPIs. That silently undercut the very disclosure the filter
was supposed to guarantee. Now the human path distinguishes file-absent from
all-filtered ('no production telemetry; N row(s) present, all non-production;
--include-bench to include them'). The JSON path already carried the
machine-readable excluded_non_production_rows count. 1 test.
Cranot added a commit that referenced this pull request Jul 16, 2026
@Cranot
test(ci): executable compile->envelope->hook invariant smoke (H2c) (#75)
c37556a 
Closes gap #4: CI ran unit tests, a wheel smoke of one unrelated command, and a
health/pr-risk self-analysis — but NEVER executed the production
compile -> envelope -> UserPromptSubmit hook chain with a real compiler. A change
that renamed summary.injection_advice, dropped artifact.plan, or broke the hook's
parse would pass every existing test (the hook unit test hand-BUILDS its
envelope) while silently breaking the deployed injection path — the exact
surface #69/#72/#73 touched.

Three invariants, all against the REAL compiler:
- roam --json compile on a fixture emits the hook-critical shape
  (summary.procedure/injection_advice/artifact_type + artifact.plan).
- a REAL compiled envelope fed through the REAL hook script injects the
  PRE-COMPUTED PLAN on inject advice (true producer<->consumer contract, not a
  hand-built stand-in).
- the hook's startswith('skip') gate drops a skip_* envelope (the invariant that
  lets edit-trim's skip_edit_task work with already-deployed hooks).

Robustness: fail-open SKIP when compile can't run (offline grammar fetch) so it
never falsely reds; and a positive-control guard on the skip test — it asserts
only where our bare-name roam stub actually shadows a real install (POSIX/CI),
skipping on Windows where subprocess won't resolve a roam.bat. Building it
surfaced that the pre-existing hook unit test silently exercises the real roam
on Windows rather than its stub (coincidentally passing); the positive control
makes the shadow explicit and self-checking. Not marked slow — runs in the
matrix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Cranot