How many red PR builds would in-loop verification prevent? A 14-hour look at Paperclip's CI

[adityankale190895]

Hi all. I build DiffCI, an open-source (MIT) tool that lets a coding agent run the tests and checks its change affects before it pushes. Disclosure up front: it's my project, and I did this analysis with Claude Code. I'd have asked in #dev, but I can't use Discord, so I'm posting here instead.

Paperclip has a lot of agent-written PRs, so I wanted to know how many red builds come from failures the author's agent could have caught locally. I used read-only GitHub API calls on your public CI history, and nothing ran on your infra.

Window: 300 runs of the PR workflow, 10 Nov 20:37 to 11 Nov 10:27 UTC (about 14 hours).

	Red runs	Runner-minutes
Red	74 of 300 (25%)	~25,600
Every failure was the PR's own, in a job you could run before pushing (Vitest, tsc, build)	12	~4,050
— of those, agent-written (co-author line or agent branch)	11	~3,700
Own failure plus a shared/E2E one	14
Not the PR's own (failures shared across PRs, timeouts)	36
E2E shards only	11

"Own" means no other PR in the window failed the same test with the same message. Shared failures, such as manual-private-cell-report-assets on 10 PRs and heartbeat-plugin-environment on 9, are counted separately because the author couldn't have fixed them. A red run took a median of 32 minutes.

Examples of own failures from agent-written PRs:

• a vi.mock of ../services/task-retry.js that was missing an export;
• a tsc error in packages/paperclip-runner;
• timer.unref is not a function in event-loop-stall-monitor.ts.

Limits:

• 14 hours of one repo.
• Agent markers come from head commits only, so 11 is a lower bound.
• I haven't replayed these commits to check that DiffCI's plan would have included each failing test.

What DiffCI selects on Paperclip today (CLI 0.3.40, local observe --no-send, no tests run):

Change	Tests selected
3 UI files (f77b8a288)	1,754 of 4,864
2 server files (bc453342e)	~1,835 of 4,875

Most of the UI breadth comes from your plugin loaders' computed import()s, which DiffCI can't resolve statically, so it widens to be safe. With a one-line opt-in in a diffci.json at the root:

{ "computedImports": "follow-path-literals" }

the UI change drops to 35 tests. The server change still selects 1,646 tests (34%), because those are real import chains.

I know you already select Vitest files in CI. This isn't a replacement for that. The aim is to stop agent-written failures before they push.

If any of this is useful:

• No action needed. The numbers are yours to use.
• Optionally, an agent working on Paperclip can try it with npx @diffci.com/diffci observe --no-send. It reports what it would select and why, and runs nothing.
• There's also an observe-only GitHub Action (DiffCI/core) that only writes a report. I'm happy to open a PR adding it if a maintainer wants it, with the Model Used section filled in.

Happy to share the scripts and raw data. And if this isn't welcome here, just say so and I'll leave it.