Skip to content

[codex] add evidence-bound failure clustering - #6

Draft
divo12 wants to merge 7 commits into
codex/mine-admission-snapshotsfrom
codex/mine-diagnosis-clustering
Draft

[codex] add evidence-bound failure clustering#6
divo12 wants to merge 7 commits into
codex/mine-admission-snapshotsfrom
codex/mine-diagnosis-clustering

Conversation

@divo12

@divo12 divo12 commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Stack

Depends on #5. Review against codex/mine-admission-snapshots.

Problem

Mine can now admit trustworthy failures, but a failure list is not an improvement target. OFW needs evidence-bound diagnoses and recurring mechanism clusters without allowing a model or plugin to invent unsupported claims, inspect good/holdout data, hang the process, or mutate the source workspace.

Implementation

  • Adds DiagnosisRun(harness_or_revision, mine_result, PythonDiagnoser).run().
  • Runs a file-backed top-level diagnoser in a disposable, timeout-bound child process.
  • Supports an optional pinned model fingerprint without coupling OFW to one model provider.
  • Sends only privacy-safe verified-failure snapshots to the diagnoser; good/ambiguous/invalid traces are excluded.
  • Requires typed proposed diagnoses or explicit abstention.
  • Validates trace identity, diagnosis structure, confidence bounds, component classes, and every observation/score evidence anchor.
  • Converts timeout, child failure, malformed output, invalid structure, foreign trace identity, or nonexistent evidence into abstention.
  • Verifies snapshot paths stay inside the Mine artifact root and bytes match the recorded content digest before reading.
  • Binds loaded snapshot trace ID, harness revision, and collection digest back to the admission/Mine lineage.
  • Gives each mechanism a stable cluster family ID and content-digested revisions with recurrence, severity, confidence, resolution rate, reopen/resolved state, affected components, evidence, and parent revision references.
  • Includes actual diagnosis payload digest in diagnosis/cluster IDs so nondeterministic model output cannot overwrite prior evidence.
  • Writes owner-only atomic diagnosis manifests.

TDD evidence

  • Two planted tool-schema failures form one recurring cluster; a prompt gap forms another.
  • Good traces are invisible to diagnosis.
  • Explicit abstention is retained.
  • Invalid evidence anchors become abstentions and never create clusters.
  • Slow diagnosers are terminated at their timeout and abstain.
  • Identical inputs and outputs produce identical diagnosis results.
  • Tampered or path-escaping snapshots fail before diagnoser execution.
  • Swapped trace, revision, or collection snapshots fail even when their replacement digest is internally valid.
  • Added failures advance the same cluster family revision; disappearing failures produce a resolved revision.

Verification

  • pytest --cov=ofw --cov-report=term-missing — 94 passed, 93% coverage
  • Ruff and strict mypy across src and tests — clean
  • forbidden construct scan — no Any, raw dict, object, getattr, setattr, lambdas, or __import__
  • Bandit and deep security scan — zero findings
  • pip-audit — no known vulnerabilities
  • source distribution and wheel build successfully

Deliberate scope

  • Exact typed mechanism grouping only; embeddings are unnecessary until measured retrieval pressure exists.
  • No cluster confirmation UI, merge/split operations, exports, benchmark runner, candidate mutation, scheduler, or promotion.

…is-clustering

# Conflicts:
#	src/ofw/__init__.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant