Skip to content

[Findings] Confidence everywhere; severity derived from signals (C3) #85

@AndresL230

Description

@AndresL230

Summary

Every finding carries confidence and costImpactUsd. Severity is computed from those signals, not hardcoded per detector.

Why

Today, a high severity finding on a free endpoint and a high severity finding on a gpt-4o polling loop look identical. Users can't filter by confidence on local-rule findings (only AI ones). Severity can't be calibrated against benchmark data.

Current state

  • AI findings carry confidence (filtered by eco.aiReview.minConfidence)
  • Local-rule findings have no confidence
  • Severity hardcoded per detector in src/scanner/local-waste-detector.ts

What to do

  1. Add to every detector emission:
interface FindingSignals {
  confidence: number;             // 0..1
  costImpactUsd: number | null;
  frequencyClass: FrequencyClass;
}
  1. Local detectors start with fixed confidence per type, refined by C1 calibration.
  2. Derive severity at one place:
function deriveSeverity(s: FindingSignals): "high" | "medium" | "low" {
  const score = s.confidence * (s.costImpactUsd ?? 0);
  if (score >= 100) return "high";
  if (score >= 10)  return "medium";
  return "low";
}
  1. Tune thresholds against the benchmark.
  2. Add UI: filter by confidence, sort by cost impact.

Acceptance criteria

  • Every FindingNode has confidence + costImpactUsd
  • Severity computed in one place from those signals
  • Filtering by confidence in webview works
  • Two findings of same type on different-cost endpoints get different severities

Depends on

  • C1 (per-detector confidence values)
  • D1 (threshold tuning)

Reference

Full design: https://github.com/recost-dev/extension/blob/main/docs/accuracy/findings.md#c3-confidence-as-a-first-class-field-severity-derived-from-signals

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/findingsdocs/accuracy/findings.md — calibrated waste detector findingsenhancementNew feature or requestwave/1-findings-qualityC2 + C3 + guard-regex cleanup

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions