Skip to content

[codex] add exact failure pattern mining - #30

Merged
divo12 merged 6 commits into
freshfrom
ao/openflywheel-1/failure-manifest
Sep 2, 2026
Merged

[codex] add exact failure pattern mining#30
divo12 merged 6 commits into
freshfrom
ao/openflywheel-1/failure-manifest

Conversation

@divo12

@divo12 divo12 commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Context

Builds on #27 and #28, which introduced the authoritative FailureDiagnosis contract and one immutable local diagnosis artifact per failed outcome.

Summary

  • add a bounded mine_failure_patterns module and read-only MCP tool over one to fifty explicit diagnosis artifact IDs
  • convert persisted JSON back into validated domain diagnoses at the file-adapter seam
  • normalize volatile absolute paths, long opaque identifiers, and numbers before stable SHA-256 fingerprinting
  • group supported diagnoses by failure type plus exact normalized root cause
  • return deterministic counts, task/trace/artifact membership, and first/last occurrence times
  • keep inconclusive diagnoses separate instead of forcing attribution
  • add $failure-pattern-miner, route ITSM runs through it after individual diagnoses, and bump the plugin/package to 0.6.0
  • pin the portable plugin runtime to the exact feature commit

Architecture

MCP input -> FailurePatternMiningService -> FailureArtifactReader -> FileFailureWorkspace -> typed pattern observation

The service depends on a narrow reader interface. The file adapter performs bounded no-follow reads and immediately reconstructs FailureDiagnosis objects, so malformed or semantically inconsistent artifacts fail closed before reduction.

Non-goals

  • no semantic clustering, embeddings, vector index, or similarity threshold
  • no unbounded directory scan or implicit artifact discovery
  • no copied Langfuse trace payloads or local trace database
  • no outcome inference, diagnosis mutation, automatic hypothesis, or harness edit
  • no change to trace querying, verifier behavior, benchmark execution, or artifact schema

Local verification

  • changed-file Ruff formatting: clean
  • uv run ruff check src tests plugins/openflywheel/scripts/mcp_server.py: clean
  • uv run mypy src tests plugins/openflywheel/scripts/mcp_server.py: clean across 44 source files
  • uv run pytest --cov=ofw --cov-report=term-missing --cov-fail-under=90 -q: 188 passed, 2 skipped, 95.24% coverage
  • changed production functions: cyclomatic complexity 5 or lower; average A (1.92)
  • trace-query-planner, outcome-recorder, failure-miner, and failure-pattern-miner skill validation: passed
  • plugin validation: passed

All verification above is local evidence, not CI evidence.

Known limitation

This intentionally performs deterministic exact bucketing after normalization. Paraphrased causes may remain separate, and a shared fingerprint does not prove that one repair applies to every occurrence. The skill requires rereading supporting diagnoses before forming a shared hypothesis.


Summary by cubic

OpenFlyWheel previously recorded individual failure diagnoses; it now mines bounded exact patterns from selected diagnosis artifacts, making recurring causes easier to prioritize without scanning workspaces or copying trace data. Reads fail closed for invalid IDs, malformed or oversized artifacts, and non-regular files.

  • mine_failure_patterns accepts 1–50 unique artifact IDs from an explicit scope.
  • The file adapter uses no-follow, non-blocking reads and reconstructs validated diagnoses before mining.
  • Fingerprinting masks volatile paths, supported opaque identifiers, UUIDs, ULIDs, hex digests, and numbers.
  • Results provide deterministic ordering, counts, task and trace membership, artifact IDs, and occurrence times.
  • Inconclusive diagnoses remain separate instead of being assigned to a pattern.
  • Adds $failure-pattern-miner, routes ITSM templates through it after diagnosis recording, and bumps the package and plugin to 0.6.0 with a pinned portable runtime.

Written for commit f3f3cfe. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 17 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread src/ofw/evaluation/failure_workspace.py
Comment thread src/ofw/evaluation/failure_patterns.py Outdated
Comment thread src/ofw/evaluation/failure_patterns.py Outdated
Comment thread src/ofw/evaluation/failure_patterns.py
Comment thread tests/test_failure_patterns.py Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread tests/test_failure_patterns.py
Comment thread src/ofw/evaluation/failure_patterns.py
@divo12
divo12 merged commit 17b3a76 into fresh Sep 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant