feat(clustering): mark un-clusterable-fit rows in the contract_anomaly feed - #76
Merged
Merged
Conversation
…y feed Phase 2 of the un-clusterable-contract fix (Phase 1 broke the re-fit loop). A contract whose shape does not cluster fits as majority DBSCAN-noise, so its outlier detectors flag a high-volume degenerate-window flood into the host contract_anomaly feed. Rather than withdraw any vote (which would demote a genuine single-detector anomaly below the publish bar: a recall regression), stamp an evidence-only unclusterable_fit marker on every published row so the host feed can group and de-prioritize the flood without suppressing anything. - migration 012: additive unclusterable_fit UInt8 DEFAULT 0 on tx_contract_anomaly; - publish_contract_anomaly derives one 0/1 flag from the contract's fit_coverage (the SAME signal Phase 1's scheduler/UI use, so there is no second threshold to diverge) and stamps it uniformly on every row a reconciliation writes (positives and tombstones); - pipeline saves fit_coverage before it publishes, so a fresh fit's rows carry that fit's coverage, not the previous fit's. Recall untouched: no vote, verdict, consensus, or publication decision changes; detect.py's DBSCAN-noise seed is not touched; every flagged row still publishes and stays individually inspectable. The host-side grouping that consumes the marker is a separate follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
elluff
force-pushed
the
fix/clustering-anomaly-unclusterable-marker
branch
from
July 23, 2026 11:57
ce0976c to
04de3be
Compare
This was referenced Jul 23, 2026
elluff
added a commit
that referenced
this pull request
Jul 23, 2026
…y rows (#78) Consumes the sidecar's unclusterable_fit marker (migration 012, PRs #75/#76) on the host attacks feed. An "anomaly" verdict from a structurally un-clusterable fit is DBSCAN-noise from a model that could not cluster its own data, not a distinguishing signal, so the feed should visibly discount it without silencing anything. Backend: - clustering_queries: add unclusterable_fit to the tx_contract_anomaly projection (_FIELDS); it flows through _row_to_dict/_group to every reader. - contract_anomaly.resolve: carry the flag forward, coerced to bool. - ClassScoreResult.contract_anomaly_unclusterable: new field, parallel to contract_anomaly_corroborates; set by _merge_contract_anomaly. - _sort_results: append a final-tie-break de-prioritization so an un-clusterable contract_anomaly row loses only to an equally-scored / equally-dated peer, never on score or recency. Frontend: - Thread the flag through ApiAnalysisResult -> toRiskAlert -> RiskAlert. - AttacksPage: an "unclusterable model" badge on those rows. Recall-safe by construction: score, risk_band, and alert routing are UNCHANGED. A human-labeled `malicious` verdict still floors Critical and wins max_class even when its fit is flagged; the marker is evidence only. Nothing is filtered, hidden, or re-banded. Mirrors the Phase-2 decision to add an evidence marker rather than withdraw the DBSCAN-noise vote. Tests: backend merge/recall-guard/sort tie-break cases (incl. test_malicious_still_fires_when_unclusterable and test_flag_never_beats_a_higher_score); frontend mapping pass-through/default. Note: group-by-contract collapse is deferred to a follow-on (net-new list UI; the row type carries no contract id and the list has no grouping infra yet). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Phase 2 of the un-clusterable-contract fix. Depends on #75 (Phase 1, which adds
fit_coverageand themodel_unclusterablehelper). This PR targetsmain, so its diff currently includes #75's commit (a8c7d1c) as well; review onlyce0976c, and it should be merged after #75 (GitHub will drop #75's commit from the diff once #75 merges).Problem
The same un-clusterable fit that drove the re-fit loop (#75) also floods the host
contract_anomaly"attacks" feed: on a majority-DBSCAN-noise population the outlier detectors flag a high-volume degenerate-window stream.Fix (recall-safe, evidence only)
Rather than withdraw any vote (which would demote a genuine single-detector anomaly below the publish bar — a recall regression, and the batch path is the sole evaluator for backfilled history), stamp an evidence-only
unclusterable_fitmarker on every published row so the host feed can group and de-prioritize the flood without suppressing anything.012: additiveunclusterable_fit UInt8 DEFAULT 0ontx_contract_anomaly(idempotent, validated live);publish_contract_anomalyderives one 0/1 flag from the contract'sfit_coverage(the same signal fix(clustering): break the non-convergent re-fit loop for un-clusterable contracts #75's scheduler/UI use — no second threshold to diverge) and stamps it uniformly on every row a reconciliation writes (positives and tombstones);fit_coveragebefore it publishes, so a fresh fit's rows carry that fit's coverage, not the previous fit's.Recall safety
No vote, verdict, consensus, or publication decision changes;
detect.py's DBSCAN-noise seed is not touched; every flagged row still publishes and stays individually inspectable (a mismarked row is fully visible). All detector locks (test_anomaly) and publish-set tests re-run green.Follow-up (not in this PR)
The host-side feed change that consumes
unclusterable_fitto group/de-prioritize the degenerate-window rows (same area as the #72 Moderate-band cap) is a separate PR. Until it lands, the column is inert (rows still appear, now carrying the flag); no recall impact.🤖 Generated with Claude Code