fix(calc): adversarial-audit batch — 12 defects across the calc suite - #71
Merged
Conversation
An adversarial audit of the shipped 5-tool suite confirmed 17 findings by
direct execution; this lands the 12 substantive fixes:
CRITICAL — runner partial-oracle hole: a runner answering only some cases read
green (liveness only tripped on wholesale silence). check_with_runner now
counts missing (sent-but-never-answered) cases, surfaces the first ids and any
stderr/exit-code diagnostics, and the check gate fails on missing > 0 in
runner mode. A runner that drops the hard buckets can never read green.
MAJOR:
- tolerance default 0.005 -> 0.001: at 0.005 an UNROUNDED product passes every
tie by definition (|raw - rounded| <= half a cent) — the exact cases the
oracle exists to test. Help text documents the boundary.
- DBF columns: validated case-insensitively but keyed case-sensitively —
'net=netvalue' vs NETVALUE silently extracted 0 cases. Yielded dicts now key
by the requested spelling; extract additionally exits 2 on an empty corpus
from a non-empty source (liveness at the first stage).
- normalize_target now strips underscores: vat_amount and vatAmount group (the
headline snake-backend vs camel-frontend case was invisible); verify's
prefilter searches both spellings.
- rounding_semantic applies LANGUAGE_ALIASES: .tsx/.jsx/.vue and TS get JS
semantics (a React frontend was invisible to divergence detection).
- RULE_SEMANTICS naming bridge: golden's half_up (= ties away from zero) is
now machine-joined to calc-inventory's labels in every audit/check envelope,
and --rule help warns it is NOT JS Math.round's half-toward-+inf.
- nested-call mode poisoning: round(round(b,2,PHP_ROUND_HALF_EVEN)+a, 2) no
longer labels the OUTER call half_to_even (nested call spans are blanked
before the mode regex).
- calc-probe scoped mode selects exact (lang, fn, mode) matches only — no more
probing default-mode variants the code never uses (phantom divergences).
MINOR: --sample uses ceil stride (no untested corpus tail) + IntRange guards;
DBF numeric overflow ('**12.34') decodes to None instead of junk; the reader
respects the header record count (no ghost records from appender debris);
--tolerance parse + malformed-corpus reads sealed to usage errors; --timeout
exposed for slow runners; probe discloses present-but-crashed runtimes
separately from not-installed; audit verdict wording corrected to 'unexplained
by the best per-bucket rule'.
12 new regression tests (81 total across the suite); prepush 7/7 green.
roam-code AnalysisMode: incremental ( Health Score: 79/100 health: Fair codebase (79/100) — 48 critical, 7 warnings, focus: god_components Health Metrics
PR Risk
Quality Gate: PASSEDGate expression: SARIF Upload
Full analysis outputhealth{
"_meta": {
"cache_ttl_s": 300,
"cacheable": true,
"index_age_s": 4,
"index_status": {
"dirty_files": 0,
"fresh": false,
"head_commit": "ac73ea6750f0",
"hint": "index latest commit db11003c54bb != HEAD ac73ea6750f0 — git-derived metrics (commits, churn, co-change, weather) may be stale. Run `roam index --force`.",
"indexed_commit": "db11003c54bb"
},
"latency_ms": null,
"response_tokens": 4083,
"roam_version": "13.9.0",
"timestamp": "2026-07-16T09:29:33Z"
},
"actionable_count": 15,
"actionable_cycles": 0,
"agent_contract": {
"confidence": null,
"facts": [
"Fair codebase (79/100) — 48 critical, 7 warnings, focus: god_components",
"health score 79",
"tangle ratio 0.0",
"0.0013 propagation cost findings",
"issue count 65"
],
"next_commands": [
"roam debt",
"roam trends --days 30"
],
"risks": []
},
"algebraic_connectivity": null,
"algebraic_connectivity_available": false,
"bottleneck_thresholds": {
"p70": 662.9,
"p90": 4193.4,
"population": 2389,
"utility_multiplier": 1.5
},
"category_severity": {
"bottlenecks": {
"critical": 15,
"info": 0,
"warning": 0
},
"cycles": {
"critical": 0,
"info": 0,
"warning": 0
},
"god_components": {
"critical": 33,
"info": 10,
"warning": 7
},
"layer_violations": {
"critical": 0,
"info": 0,
"warning": 0
}
},
"command": "health",
"cycles_actionable": 0,
"cycles_total": 17,
"framework_filtered": 0,
"health_score": 79,
"ignored_cycles": 17,
"imported_coverable_lines": 0,
"imported_coverage_files": 0,
"imported_coverage_pct": null,
"imported_covered_lines": 0,
"index_status": {
"dirty_files": 0,
"fresh": false,
"head_commit": "ac73ea6750f0",
"hint": "index latest commit db11003c54bb != HEAD ac73ea6750f0 — git-derived metrics (commits, churn, co-change, weather) may be stale. Run `roam index --force`.",
"indexed_commit": "db11003c54bb"
},
"issue_count": 65,
"list_counts": {
"bottlenecks": 15,
"cycle_break_suggestions": 0,
"cycles": 17,
"god_components": 50,
"layer_violations": 0,
"next_steps": 2,
"score_breakdown": 5
},
"project": "roam-code",
"propagation_cost": 0.0013,
"schema": "roam-envelope-v1",
"schema_version": "1.1.0",
"severity": {
"critical": 48,
"info": 27,
"warning": 7
},
"summary": {
"actionable_cycles": 0,
"algebraic_connectivity": null,
"algebraic_connectivity_available": false,
"category_severity": {
"bottlenecks": {
"critical": 15,
"info": 0,
"warning": 0
},
"cycles": {
"critical": 0,
"info": 0,
"warning": 0
},
"god_components": {
"critical": 33,
"info": 10,
"warning": 7
},
"layer_violations": {
"critical": 0,
"info": 0,
"warning": 0
}
},
"cycles_actionable": 0,
"cycles_definition": "Cycle counts derived from `roam.graph.cycles.find_cycles(G, min_size=2)` on the symbol graph. `cycles_total` = all SCCs of size >= 2; `cycles_actionable` = SCCs spanning >=2 files AND no test files (same-file and test-only cycles are informational). Run `roam health` for the per-cycle breakdown.",
"cycles_total": 17,
"detail_available": true,
"god_components": 50,
"god_components_definition": "God components: symbols where `(in_degree + out_degree) > 20` from the `graph_metrics` table, with utility-aware severity bands (standard >50=CRITICAL >30=WARNING; utility >150=CRITICAL >90=WARNING). Run `roam health` for the per-symbol breakdown. Legacy aliases: `god_objects` (fingerprint), `god_classes` (rules).",
"health_score": 79,
"health_score_definition": "weighted geometric mean (0-100) of 5 sigmoid health factors: tangle_ratio, god_components, bottlenecks, layer_violations, file_health (+coverage if available).",
"ignored_cycles": 17,
"imported_coverage_files": 0,
"imported_coverage_pct": null,
"issue_count": 65,
"partial_success": true,
"preserved_list_truncations": {},
"propagation_cost": 0.0013,
"severity": {
"critical": 48,
"info": 27,
"warning": 7
},
"tangle_ratio": 0,
"tangle_ratio_definition": "fraction of symbols inside non-trivial SCCs; higher = more cyclic coupling.",
"total_cycles": 17,
"truncated": true,
"verdict": "Fair codebase (79/100) — 48 critical, 7 warnings, focus: god_components",
"warnings_out": [
"health_algebraic_connectivity_warning:RuntimeWarning:algebraic_connectivity compute failed (ModuleNotFoundError): No module named 'numpy'; returning 0.0 sentinel — value is NOT a legitimate disconnected-graph reading"
]
},
"tangle_ratio": 0,
"total_cycles": 17,
"utility_count": 35,
"version": "13.9.0",
"warnings_out": [
"health_algebraic_connectivity_warning:RuntimeWarning:algebraic_connectivity compute failed (ModuleNotFoundError): No module named 'numpy'; returning 0.0 sentinel — value is NOT a legitimate disconnected-graph reading"
]
}pr-risk{
"_meta": {
"cache_ttl_s": 60,
"cacheable": true,
"index_age_s": 5,
"index_status": {
"dirty_files": 0,
"fresh": false,
"head_commit": "ac73ea6750f0",
"hint": "index latest commit db11003c54bb != HEAD ac73ea6750f0 — git-derived metrics (commits, churn, co-change, weather) may be stale. Run `roam index --force`.",
"indexed_commit": "db11003c54bb"
},
"latency_ms": null,
"response_tokens": 1639,
"roam_version": "13.9.0",
"timestamp": "2026-07-16T09:29:35Z"
},
"actor": null,
"agent_contract": {
"confidence": null,
"facts": [
"High risk (59/100) — careful review needed (risk_level high) (driver: blast_pct)",
"risk score 59",
"3 risk rank findings",
"8 changed files",
"420 lines added"
],
"next_commands": [],
"risks": [
"pr-risk: high (59/100) on 1b5271f64865de57947452e6414628be2c7d7780..HEAD — driver: blast_pct",
"High blast radius: 12761 of 41853 symbols affected (30.5%) on 1b5271f64865de57947452e6414628be2c7d7780..HEAD"
]
},
"author": null,
"blast_radius_pct": 30.5,
"bus_factor_risk": 0.47,
"change_shape": "mixed",
"changed_files": 8,
"closest_historical_pattern": [
"src/roam/commands/cmd_calc_golden.py",
"src/roam/commands/cmd_calc_probe.py",
"src/roam/commands/cmd_verify.py",
"src/roam/index/calc_extract.py",
"src/roam/index/golden_calc.py",
"tests/test_calc_golden.py",
"tests/test_calc_inventory.py",
"tests/test_calc_probe.py"
],
"closest_similarity": 1,
"cluster_spread": 0.01,
"clusters_touched": 160,
"command": "pr-risk",
"coupling_score": 0.11,
"dead_code": [
{
"file": "src/roam/commands/cmd_calc_golden.py",
"kind": "function",
"name": "extract"
},
{
"file": "src/roam/commands/cmd_calc_golden.py",
"kind": "function",
"name": "audit"
},
{
"file": "src/roam/index/golden_calc.py",
"kind": "class",
"name": "RuleFit"
},
{
"file": "src/roam/index/golden_calc.py",
"kind": "method",
"name": "pct"
},
{
"file": "src/roam/index/golden_calc.py",
"kind": "method",
"name": "failed"
}
],
"dead_exports": 5,
"familiarity": {
"avg_familiarity": 1,
"files": [],
"files_assessed": 0
},
"findings": [
{
"claim": "pr-risk: high (59/100) on 1b5271f64865de57947452e6414628be2c7d7780..HEAD — driver: blast_pct",
"confidence": "heuristic",
"evidence": {
"actor": null,
"author": null,
"blast_radius_pct": 30.5,
"bus_factor_risk": 0.467,
"changed_files_count": 8,
"commit_range": "1b5271f64865de57947452e6414628be2c7d7780..HEAD",
"coupling_score": 0.107,
"created_at_epoch": 1784194175,
"diff_id": "9b49874b58e6",
"familiarity_risk": 0,
"file_list": [
"src/roam/commands/cmd_calc_golden.py",
"src/roam/commands/cmd_calc_probe.py",
"src/roam/commands/cmd_verify.py",
"src/roam/index/calc_extract.py",
"src/roam/index/golden_calc.py",
"tests/test_calc_golden.py",
"tests/test_calc_inventory.py",
"tests/test_calc_probe.py"
],
"hotspot_score": 1,
"label": "1b5271f64865de57947452e6414628be2c7d7780..HEAD",
"lines_added": 420,
"lines_removed": 48,
"minor_risk": 0,
"novelty_score": 0,
"reductive_change": false,
"risk_level": "high",
"risk_score": 59,
"staged": false,
"test_coverage_pct": 80,
"top_driver": "blast_pct"
},
"finding_id_str": "pr-risk:composite-risk-score:9b49874b58e6",
"kind": "pr-risk:composite-risk-score",
"severity": "high",
"source_detector": "pr-risk",
"source_version": "1.0.0",
"subject_id": null,
"subject_kind": "commit"
},
{
"claim": "High blast radius: 12761 of 41853 symbols affected (30.5%) on 1b5271f64865de57947452e6414628be2c7d7780..HEAD",
"confidence": "structural",
"evidence": {
"affected_symbols": 12761,
"blast_radius_pct": 30.5,
"changed_files_count": 8,
"changed_symbol_ids_count": 522,
"commit_range": "1b5271f64865de57947452e6414628be2c7d7780..HEAD",
"created_at_epoch": 1784194175,
"diff_id": "9b49874b58e6",
"file_list": [
"src/roam/commands/cmd_calc_golden.py",
"src/roam/commands/cmd_calc_probe.py",
"src/roam/commands/cmd_verify.py",
"src/roam/index/calc_extract.py",
"src/roam/index/golden_calc.py",
"tests/test_calc_golden.py",
"tests/test_calc_inventory.py",
"tests/test_calc_probe.py"
],
"label": "1b5271f64865de57947452e6414628be2c7d7780..HEAD",
"staged": false,
"total_symbols": 41853
},
"finding_id_str": "pr-risk:high-blast-radius-symbol-touched:9b49874b58e6",
"kind": "pr-risk:high-blast-radius-symbol-touched",
"severity": "high",
"source_detector": "pr-risk",
"source_version": "1.0.0",
"subject_id": null,
"subject_kind": "commit"
}
],
"hotspot_score": 1,
"label": "1b5271f64865de57947452e6414628be2c7d7780..HEAD",
"layer_spread": 0.52,
"layers_touched": 16,
"lines_added": 420,
"lines_removed": 48,
"minor_risk": {
"files": [],
"files_assessed": 0,
"minor_files": 0
},
"novelty_score": 0,
"per_file": [
{
"blast": 12594,
"churn": 567,
"is_test": false,
"lines_added": 63,
"lines_removed": 9,
"path": "src/roam/index/golden_calc.py",
"symbols": 50
},
{
"blast": 235,
"churn": 9991,
"is_test": false,
"lines_added": 19,
"lines_removed": 1,
"path": "src/roam/commands/cmd_verify.py",
"symbols": 349
},
{
"blast": 179,
"churn": 437,
"is_test": false,
"lines_added": 42,
"lines_removed": 7,
"path": "src/roam/index/calc_extract.py",
"symbols": 37
},
{
"blast": 51,
"churn": 384,
"is_test": false,
"lines_added": 86,
"lines_removed": 18,
"path": "src/roam/commands/cmd_calc_golden.py",
"symbols": 7
},
{
"blast": 47,
"churn": 379,
"is_test": false,
"lines_added": 18,
"lines_removed": 11,
"path": "src/roam/commands/cmd_calc_probe.py",
"symbols": 8
},
{
"blast": 15,
"churn": 478,
"is_test": true,
"lines_added": 148,
"lines_removed": 0,
"path": "tests/test_calc_golden.py",
"symbols": 29
},
{
"blast": 10,
"churn": 302,
"is_test": true,
"lines_added": 29,
"lines_removed": 2,
"path": "tests/test_calc_inventory.py",
"symbols": 31
},
{
"blast": 4,
"churn": 144,
"is_test": true,
"lines_added": 15,
"lines_removed": 0,
"path": "tests/test_calc_probe.py",
"symbols": 11
}
],
"project": "roam-code",
"reductive_change": false,
"reductive_discount_applied": false,
"risk_level": "high",
"risk_level_canonical": "high",
"risk_rank": 3,
"risk_score": 59,
"schema": "roam-envelope-v1",
"schema_version": "1.1.0",
"suggested_reviewers": [
{
"actor": "Cranot",
"author": "Cranot",
"lines": 8177
},
{
"actor": "Dimitris",
"author": "Dimitris",
"lines": 1620
},
{
"actor": "stoa-automerge",
"author": "stoa-automerge",
"lines": 5
}
],
"summary": {
"change_shape": "mixed",
"changed_files": 8,
"findings_count": 2,
"lines_added": 420,
"lines_removed": 48,
"partial_success": false,
"risk_level": "high",
"risk_level_canonical": "high",
"risk_rank": 3,
"risk_score": 59,
"score_classification": "classified",
"verdict": "High risk (59/100) — careful review needed (risk_level high) (driver: blast_pct)"
},
"test_coverage_pct": 80,
"total_clusters": 13134,
"total_layers": 31,
"version": "13.9.0",
"warnings_out": []
}roam-code analysis | Commands: |
Roam Agent ReviewVerdict: SAFE (risk_level low) blast-radius 0/100 · ai-likelihood 13/100 · rule violations 0 · critique high-severity 0 Verdict: SAFE. All structural signals clean at the configured thresholds. Next steps
Powered by roam-code — Apache 2.0, 100% local. Customize thresholds in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An adversarial audit (17 findings, all confirmed by direct execution) of the shipped calc suite; this lands the 12 substantive fixes. Highlights:
missingis now counted, diagnosed (ids + stderr tail), and gates (exit 5) in runner mode.net=netvaluevsNETVALUEsilently extracted 0 cases → requested-spelling keying + exit-2 on empty corpus from non-empty source.--timeout, failed-runtime disclosure.12 new regression tests (81 total); prepush 7/7 green.