Skip to content

docs(debugging): define dap failure benchmark - #264

Closed
627150795 wants to merge 1 commit into
openpi-dev:mainfrom
627150795:docs/issue-168-dap-benchmark
Closed

docs(debugging): define dap failure benchmark#264
627150795 wants to merge 1 commit into
openpi-dev:mainfrom
627150795:docs/issue-168-dap-benchmark

Conversation

@627150795

Copy link
Copy Markdown
Contributor

Problem

Issue #168 needs evidence about whether on-demand DAP improves diagnosis over logs and tests, but a prompt-only approximation or a debugger self-report would not be a valid comparison.

Value

Define a failure-class benchmark that measures diagnosis quality, time, tokens, false leads, debugger failures, cleanup, and side effects.

Approach

  • Add docs/design/DAP_FAILURE_CLASS_BENCHMARK_PROTOCOL.md.
  • Define logs and tests, on-demand DAP, and combined arms with a fixed failure corpus.
  • Freeze runtime, model, tool, permission, verifier, and timeout variables.
  • Record diagnosis rubric, DAP safety boundaries, overhead, failures, and a decision gate.
  • Keep the first step protocol-only; mark DAP not-run when unavailable.

Validation

  • git diff --check
  • bun run lint
  • bun run typecheck

Impact

Documentation only. This is a focused research slice for #168 and contains no DAP runtime or experimental result.

Addresses #168

@tt-a1i

tt-a1i commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Thank you for writing down this candidate. We are closing it under the future-capability disposition in #277. The PR remains the preserved research source, but OpenPI will not promote a protocol-only or candidate design into a repository contract before there is an active implementation or experiment with a fixed owner, revision, verifier, and acceptance evidence. This is not a content rejection, and no rebase is requested. Restart the work from the linked source Issue when that evidence frontier is ready.

@tt-a1i tt-a1i closed this Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants