test(ci): executable compile→envelope→hook invariant smoke (H2c) - #75
Merged
Conversation
Closes gap #4: CI ran unit tests, a wheel smoke of one unrelated command, and a health/pr-risk self-analysis — but NEVER executed the production compile -> envelope -> UserPromptSubmit hook chain with a real compiler. A change that renamed summary.injection_advice, dropped artifact.plan, or broke the hook's parse would pass every existing test (the hook unit test hand-BUILDS its envelope) while silently breaking the deployed injection path — the exact surface #69/#72/#73 touched. Three invariants, all against the REAL compiler: - roam --json compile on a fixture emits the hook-critical shape (summary.procedure/injection_advice/artifact_type + artifact.plan). - a REAL compiled envelope fed through the REAL hook script injects the PRE-COMPUTED PLAN on inject advice (true producer<->consumer contract, not a hand-built stand-in). - the hook's startswith('skip') gate drops a skip_* envelope (the invariant that lets edit-trim's skip_edit_task work with already-deployed hooks). Robustness: fail-open SKIP when compile can't run (offline grammar fetch) so it never falsely reds; and a positive-control guard on the skip test — it asserts only where our bare-name roam stub actually shadows a real install (POSIX/CI), skipping on Windows where subprocess won't resolve a roam.bat. Building it surfaced that the pre-existing hook unit test silently exercises the real roam on Windows rather than its stub (coincidentally passing); the positive control makes the shadow explicit and self-checking. Not marked slow — runs in the matrix.
Roam Agent ReviewVerdict: SAFE (risk_level low) blast-radius 0/100 · ai-likelihood 18/100 · rule violations 0 · critique high-severity 0 Verdict: SAFE. All structural signals clean at the configured thresholds. Next steps
Powered by roam-code — Apache 2.0, 100% local. Customize thresholds in |
roam-code AnalysisMode: incremental ( Health Score: 79/100 health: Fair codebase (79/100) — 48 critical, 7 warnings, focus: god_components Health Metrics
PR Risk
Quality Gate: PASSEDGate expression: SARIF Upload
Full analysis outputhealth{
"_meta": {
"cache_ttl_s": 300,
"cacheable": true,
"index_age_s": 4,
"index_status": {
"dirty_files": 0,
"fresh": false,
"head_commit": "2837ddd1151a",
"hint": "index latest commit 4ef853623b03 != HEAD 2837ddd1151a — git-derived metrics (commits, churn, co-change, weather) may be stale. Run `roam index --force`.",
"indexed_commit": "4ef853623b03"
},
"latency_ms": null,
"response_tokens": 4083,
"roam_version": "13.9.0",
"timestamp": "2026-07-16T11:17:18Z"
},
"actionable_count": 15,
"actionable_cycles": 0,
"agent_contract": {
"confidence": null,
"facts": [
"Fair codebase (79/100) — 48 critical, 7 warnings, focus: god_components",
"health score 79",
"tangle ratio 0.0",
"0.0013 propagation cost findings",
"issue count 65"
],
"next_commands": [
"roam debt",
"roam trends --days 30"
],
"risks": []
},
"algebraic_connectivity": null,
"algebraic_connectivity_available": false,
"bottleneck_thresholds": {
"p70": 655.2,
"p90": 4404.9,
"population": 2254,
"utility_multiplier": 1.5
},
"category_severity": {
"bottlenecks": {
"critical": 15,
"info": 0,
"warning": 0
},
"cycles": {
"critical": 0,
"info": 0,
"warning": 0
},
"god_components": {
"critical": 33,
"info": 10,
"warning": 7
},
"layer_violations": {
"critical": 0,
"info": 0,
"warning": 0
}
},
"command": "health",
"cycles_actionable": 0,
"cycles_total": 17,
"framework_filtered": 0,
"health_score": 79,
"ignored_cycles": 17,
"imported_coverable_lines": 0,
"imported_coverage_files": 0,
"imported_coverage_pct": null,
"imported_covered_lines": 0,
"index_status": {
"dirty_files": 0,
"fresh": false,
"head_commit": "2837ddd1151a",
"hint": "index latest commit 4ef853623b03 != HEAD 2837ddd1151a — git-derived metrics (commits, churn, co-change, weather) may be stale. Run `roam index --force`.",
"indexed_commit": "4ef853623b03"
},
"issue_count": 65,
"list_counts": {
"bottlenecks": 15,
"cycle_break_suggestions": 0,
"cycles": 17,
"god_components": 50,
"layer_violations": 0,
"next_steps": 2,
"score_breakdown": 5
},
"project": "roam-code",
"propagation_cost": 0.0013,
"schema": "roam-envelope-v1",
"schema_version": "1.1.0",
"severity": {
"critical": 48,
"info": 27,
"warning": 7
},
"summary": {
"actionable_cycles": 0,
"algebraic_connectivity": null,
"algebraic_connectivity_available": false,
"category_severity": {
"bottlenecks": {
"critical": 15,
"info": 0,
"warning": 0
},
"cycles": {
"critical": 0,
"info": 0,
"warning": 0
},
"god_components": {
"critical": 33,
"info": 10,
"warning": 7
},
"layer_violations": {
"critical": 0,
"info": 0,
"warning": 0
}
},
"cycles_actionable": 0,
"cycles_definition": "Cycle counts derived from `roam.graph.cycles.find_cycles(G, min_size=2)` on the symbol graph. `cycles_total` = all SCCs of size >= 2; `cycles_actionable` = SCCs spanning >=2 files AND no test files (same-file and test-only cycles are informational). Run `roam health` for the per-cycle breakdown.",
"cycles_total": 17,
"detail_available": true,
"god_components": 50,
"god_components_definition": "God components: symbols where `(in_degree + out_degree) > 20` from the `graph_metrics` table, with utility-aware severity bands (standard >50=CRITICAL >30=WARNING; utility >150=CRITICAL >90=WARNING). Run `roam health` for the per-symbol breakdown. Legacy aliases: `god_objects` (fingerprint), `god_classes` (rules).",
"health_score": 79,
"health_score_definition": "weighted geometric mean (0-100) of 5 sigmoid health factors: tangle_ratio, god_components, bottlenecks, layer_violations, file_health (+coverage if available).",
"ignored_cycles": 17,
"imported_coverage_files": 0,
"imported_coverage_pct": null,
"issue_count": 65,
"partial_success": true,
"preserved_list_truncations": {},
"propagation_cost": 0.0013,
"severity": {
"critical": 48,
"info": 27,
"warning": 7
},
"tangle_ratio": 0,
"tangle_ratio_definition": "fraction of symbols inside non-trivial SCCs; higher = more cyclic coupling.",
"total_cycles": 17,
"truncated": true,
"verdict": "Fair codebase (79/100) — 48 critical, 7 warnings, focus: god_components",
"warnings_out": [
"health_algebraic_connectivity_warning:RuntimeWarning:algebraic_connectivity compute failed (ModuleNotFoundError): No module named 'numpy'; returning 0.0 sentinel — value is NOT a legitimate disconnected-graph reading"
]
},
"tangle_ratio": 0,
"total_cycles": 17,
"utility_count": 35,
"version": "13.9.0",
"warnings_out": [
"health_algebraic_connectivity_warning:RuntimeWarning:algebraic_connectivity compute failed (ModuleNotFoundError): No module named 'numpy'; returning 0.0 sentinel — value is NOT a legitimate disconnected-graph reading"
]
}pr-risk{
"_meta": {
"cache_ttl_s": 60,
"cacheable": true,
"index_age_s": 5,
"index_status": {
"dirty_files": 0,
"fresh": false,
"head_commit": "2837ddd1151a",
"hint": "index latest commit 4ef853623b03 != HEAD 2837ddd1151a — git-derived metrics (commits, churn, co-change, weather) may be stale. Run `roam index --force`.",
"indexed_commit": "4ef853623b03"
},
"latency_ms": null,
"response_tokens": 845,
"roam_version": "13.9.0",
"timestamp": "2026-07-16T11:17:20Z"
},
"actor": null,
"agent_contract": {
"confidence": null,
"facts": [
"Moderate risk (34/100) — review recommended (risk_level medium) (driver: test_coverage_low)",
"risk score 34",
"2 risk rank findings",
"1 changed files",
"210 lines added"
],
"next_commands": [],
"risks": [
"pr-risk: moderate (34/100) on 19e74bd501797e04f6e5b1cda07770522f9c9c40..HEAD — driver: test_coverage_low"
]
},
"author": null,
"blast_radius_pct": 0,
"bus_factor_risk": 0,
"change_shape": "mixed",
"changed_files": 1,
"closest_historical_pattern": null,
"closest_similarity": 0,
"cluster_spread": 0,
"clusters_touched": 2,
"command": "pr-risk",
"coupling_score": 0,
"dead_code": [],
"dead_exports": 0,
"familiarity": {
"avg_familiarity": 1,
"files": [],
"files_assessed": 0
},
"findings": [
{
"claim": "pr-risk: moderate (34/100) on 19e74bd501797e04f6e5b1cda07770522f9c9c40..HEAD — driver: test_coverage_low",
"confidence": "heuristic",
"evidence": {
"actor": null,
"author": null,
"blast_radius_pct": 0,
"bus_factor_risk": 0,
"changed_files_count": 1,
"commit_range": "19e74bd501797e04f6e5b1cda07770522f9c9c40..HEAD",
"coupling_score": 0,
"created_at_epoch": 1784200640,
"diff_id": "b9c1636a2924",
"familiarity_risk": 0,
"file_list": [
"tests/test_compile_hook_invariants.py"
],
"hotspot_score": 0.221,
"label": "19e74bd501797e04f6e5b1cda07770522f9c9c40..HEAD",
"lines_added": 210,
"lines_removed": 0,
"minor_risk": 0,
"novelty_score": 0,
"reductive_change": false,
"risk_level": "moderate",
"risk_score": 34,
"staged": false,
"test_coverage_pct": 0,
"top_driver": "test_coverage_low"
},
"finding_id_str": "pr-risk:composite-risk-score:b9c1636a2924",
"kind": "pr-risk:composite-risk-score",
"severity": "medium",
"source_detector": "pr-risk",
"source_version": "1.0.0",
"subject_id": null,
"subject_kind": "commit"
}
],
"hotspot_score": 0.22,
"label": "19e74bd501797e04f6e5b1cda07770522f9c9c40..HEAD",
"layer_spread": 0.1,
"layers_touched": 3,
"lines_added": 210,
"lines_removed": 0,
"minor_risk": {
"files": [],
"files_assessed": 0,
"minor_files": 0
},
"novelty_score": 0,
"per_file": [
{
"blast": 4,
"churn": 210,
"is_test": true,
"lines_added": 210,
"lines_removed": 0,
"path": "tests/test_compile_hook_invariants.py",
"symbols": 9
}
],
"project": "roam-code",
"reductive_change": false,
"reductive_discount_applied": false,
"risk_level": "moderate",
"risk_level_canonical": "medium",
"risk_rank": 2,
"risk_score": 34,
"schema": "roam-envelope-v1",
"schema_version": "1.1.0",
"suggested_reviewers": [],
"summary": {
"change_shape": "mixed",
"changed_files": 1,
"findings_count": 1,
"lines_added": 210,
"lines_removed": 0,
"partial_success": false,
"risk_level": "moderate",
"risk_level_canonical": "medium",
"risk_rank": 2,
"risk_score": 34,
"score_classification": "classified",
"verdict": "Moderate risk (34/100) — review recommended (risk_level medium) (driver: test_coverage_low)"
},
"test_coverage_pct": 0,
"total_clusters": 13123,
"total_layers": 31,
"version": "13.9.0",
"warnings_out": []
}roam-code analysis | Commands: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes gap #4 — CI never executed the production compile → envelope → hook chain with a real compiler, so a change renaming
summary.injection_advice, droppingartifact.plan, or breaking the hook parse would pass every test (the hook unit test hand-builds its envelope) while silently breaking the deployed injection path — the exact surface #69/#72/#73 touched.Three invariants against the real compiler: (1)
roam --json compileemits the hook-critical shape; (2) a real compiled envelope fed through the real hook injects oninject; (3) thestartswith('skip')gate drops askip_*envelope (the invariant edit-trim'sskip_edit_taskrelies on).Robust: fail-open skip when compile can't run (offline grammar) so it never flakes red; a positive-control guard on the skip test asserts only where the bare-name
roamstub actually shadows a real install (POSIX/CI). Building it surfaced that the pre-existing hook unit test silently hits real roam on Windows — the positive control makes the shadow explicit. Not slow; runs in the matrix. 3 tests.