Skip to content

Home

Jump to bottom
Ícaro Galvão do Nascimento edited this page Oct 5, 2026 · 1 revision

devin-qa-pack

Flagship QA audit for Devin sessions: it checks that what a session claims it delivered is backed by what the tool calls actually did. Verdicts: PASS, PARTIAL, UNVERIFIED.

Why it exists

Agents say "tests pass" and "pushed the fix" whether or not the evidence exists. devin-qa-pack reads sessions.db, tool_call_state and message_nodes, extracts claims (tests, commits, files, pushes, HTTP) and verifies each against recorded evidence and git state.

Install

pipx install devin-qa-pack
# or: pip install devin-qa-pack  (PyPI)
# source: pipx install "git+https://github.com/Icaro0310/devin-qa-pack"

Requires Python 3.10+ unless noted. Linux, Personal Windows and Corporate Windows are supported; see the repo's platform READMEs for differences.

Core commands

Command What it does
devin-qa-pack audit --session <id> audit one session, print PASS/PARTIAL/UNVERIFIED
devin-qa-pack audit --latest audit the most recent session
`devin-qa-pack audit --transcript --format aider claude-code`
devin-qa-pack report render the audit as HTML/JSON report
devin-qa-pack session-end hook entry point: audit at session end

What it touches

Reads: sessions.db, tool_call_state, message_nodes, git history, transcript files. Writes: report output only. Read-only on stores. Offline by default; URL checks need explicit --online --allow-domain.

Limitations

  • UNVERIFIED means evidence is missing, not that work is bad.
  • Transcript adapters (Aider, Claude Code) are experimental; the Devin-native path is the stable one.
  • It verifies claims, not code quality.

Unofficial community project. Not affiliated with Cognition AI. Part of the devin-* ecosystem.

Clone this wiki locally