feat(eval): 北极星双臂基准 v1——首个数字：管线臂 6/12，输给裸读

[2233admin]

干什么

北极星第一问（"让 AI 便宜地理解代码"）的第一个数字。确定性双臂基准，零 LLM、零网络、stdlib-only、逐字节可复现：

• Arm A（artifact-guided）：一次性跑自家管线产工件（setup 成本单记摊销），逐题只允许从工件找指针、读指针 ±10 行窗口。
• Arm B（naive）：关键词检索 + 整文件读，5 文件封顶。
• 12 题覆盖 CLI 路由 / sentrux gate / capability inventory / digest pins / retirement packets / CI / skill 包，黄金出处逐题人工核验。

结果（HEAD 8a79814，如实记录）

	Arm A（管线）	Arm B（裸读）
覆盖	6/12	10/12
覆盖题字节	6,456,569	1,081,508
胜负	赢 2	赢 10

管线臂输了，且贵。 分类拆解最有信息量：

• how 类 A 5/5 全覆盖——工件对"怎么做"够用；
• why 类 A 0/4 全灭——工件里没有任何 rationale 内容（lifecycle: 规范阅后即焚——一次性 spec 自动归档 + 活契约绑测试 + stale-spec 门禁 #104 的实证）；
• A 失败原因：no_pointer_found×4（rationale 活在常量名/字符串字面量里，工件不存原文）、窗口截断×2；
• A 的两胜不是效率胜，是 B 的朴素排序被泛词稀释——不值得庆祝。
• 顺手抓到真扫描器缺口：run_raw / 多行签名 execute 函数在 symbol 索引里缺失（题目故意打在缺口上，不粉饰）。

这组数字是 #58（查询面全是 read/audit 形状）、#105（自动调用路线）的最硬证据：当前工件不是为回答问题设计的。eval 就是干这个的——输了认，然后当路标。

工程细节

• 结果 JSON machine-first（eval/baseline-8a79814.json 为真源），BASELINE.md 由 --render 派生，禁止手改。
• 自测 23/23；两次运行结果逐字节一致（meta 时间戳除外）。
• 第二 commit 拆分 harness 过 sentrux 棘轮（functions>25 巨石规则），权威 self-scan exit 0，Quality 3554 → 3633。

Refs #93

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[repowise-bot]

🔒 Repowise is not analyzing this repository

The PR bot is free on public repositories. This one is private, which needs a Pro plan.

See plans · Manage this repository

[coderabbitai]

Warning

Review limit reached

@2233admin, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 16 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 48cc468c-f6b6-42e1-94d6-4d182de95123

📥 Commits

Reviewing files that changed from the base of the PR and between 8a79814 and 9fc1b89.

📒 Files selected for processing (16)

• eval/BASELINE.md
• eval/arms.py
• eval/baseline-8a79814.json
• eval/fixtures/questions.json
• eval/fixtures/repo/notes.md
• eval/fixtures/repo/other.py
• eval/fixtures/repo/sample.py
• eval/fixtures/run/objects/sha256/chunks.json
• eval/fixtures/run/objects/sha256/imports.json
• eval/fixtures/run/objects/sha256/manifest.json
• eval/fixtures/run/objects/sha256/symbols.json
• eval/fixtures/run/run-complete.json
• eval/harness.py
• eval/questions.json
• eval/reporting.py
• eval/test_harness.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}