RunSift

Evidence-backed CI failure triage for GitHub Actions.

RunSift helps maintainers move from a failed GitHub Actions job to a safe next action. It extracts the smallest useful log evidence, classifies recognizable failure causes, exposes uncertainty, and abstains when the evidence is insufficient.

The project is intentionally CLI-first. The first production-quality slice targets maintainers of small Python repositories that use GitHub Actions and do not have a dedicated Developer Productivity platform.

Problem

A failed CI job can produce thousands of log lines without answering the question that matters:

Should I fix code, dependencies, workflow configuration, environment setup, or investigate a transient infrastructure failure?

Blind reruns waste time and can hide genuine regressions. Manual triage also fails to preserve reusable evidence about recurring incidents.

Current status

RunSift is in foundation implementation. The repository now contains an installable typed Python package, a minimal CLI entry point, locked development tooling, tests, package smoke checks, and progressive CI quality gates.

Only runsift --help and runsift --version are implemented. Failure collection, parsing, classification, and reporting remain contracted or planned; no working triage capability or measured product result is claimed yet.

Development

uv sync --frozen
uv run --frozen runsift --help
uv run --frozen ruff format --check .
uv run --frozen ruff check .
uv run --frozen mypy src
uv run --frozen pytest
uv run --frozen python scripts/check_markdown_links.py
uv build

Python 3.11 is the default development interpreter. CI also verifies the package on Python 3.14.

MVP workflow

offline fixture, downloaded job log, or selected GitHub Actions job
→ validate and normalize input
→ redact secrets before RunSift-owned persistence
→ extract failure windows and structured evidence
→ apply deterministic classification rules
→ abstain when evidence or rule coverage is insufficient
→ render terminal, Markdown, or JSON output

Supported MVP scope

• GitHub Actions only
• public repositories or private repositories explicitly authorized by the user
• Ubuntu-hosted jobs
• Python-oriented workflows, including pytest, packaging, linting, and type checking
• offline fixture mode and local downloaded-log mode
• optional read-only GitHub API collection with a user-provided token
• suggestion-only output

Initial rule categories:

• dependency
• test
• lint_typecheck
• build_compile
• environment_permissions
• network_rate_limit
• workflow_configuration
• unknown

flaky is not a primary single-run cause label. It is a separate temporal assessment that requires comparable prior attempts, rerun evidence, or explicit maintainer confirmation.

Planned CLI

# Repeatable offline development and demo path
runsift analyze --fixture dependency-resolution

# Analyze files already downloaded by the user
runsift analyze --log job.log --metadata job.json --format terminal

# Optional read-only GitHub collection
runsift collect --repo owner/name --run-id 123456789
runsift analyze --input .runsift/runs/123456789 --format json

These command names are maintained implementation contracts, not evidence that the commands already exist. See the CLI MVP contract.

Safety principles

• Never execute commands found in logs.
• Redact before RunSift-owned persistence or report generation.
• Do not claim calibrated probability for deterministic rule scores.
• Keep unknown and abstention as valid outcomes.
• Require evidence references for every non-unknown classification.
• Treat rerun success as weak evidence, not ground truth.
• Never automatically rerun, quarantine, modify code, or suppress a failure.
• Keep private logs out of telemetry, examples, screenshots, and public fixtures.
• Use least-privilege GitHub credentials and never persist tokens.

Evidence-gated build strategy

1. Foundation: define contracts, fixtures, tests, and repository quality gates.
2. Rule-based CLI: deliver safe offline and read-only triage.
3. User validation: test usefulness and taxonomy agreement on reviewed historical failures.
4. ML evaluation: add a reproducible model only when it improves on deterministic rules.
5. GitHub integration: consider a GitHub App only after the CLI workflow proves useful.
6. Controlled operations: add promotion, rollback, and monitoring only when there is a real model and beta workload to operate.

Success evidence

RunSift is useful only when it improves a real maintenance workflow. Relevant evidence includes:

• maintainers repeatedly using it on real failures;
• reduced time to a correct next action;
• fewer blind reruns;
• useful and reproducible evidence selection;
• safe coverage for high-risk recommendations;
• measurable improvement over deterministic rules;
• reproducible evaluation and rollback demonstrations.

Documentation

• Documentation index
• CLI MVP contract
• Product and validation
• Architecture
• Data and ML evaluation
• Delivery roadmap
• Initial research summary

Non-goals

The MVP is not:

• an arbitrary log chatbot;
• an LLM-generated root-cause system;
• an autonomous remediation agent;
• a generic observability platform;
• a multi-CI product;
• a GitHub App;
• a Kubernetes-first portfolio demo.