test(drivers): a conformance run that discovers zero drivers is a failure, not an OK (#4646) - #4648
Merged
Merged
Conversation
…lure, not an OK (#4646) check-driver-conformance discovers driver packages from disk under a hardcoded DRIVERS_DIR. listDir swallows ENOENT and returns [], and all three invariants iterate the discovered set — CONSUMED over `drivers`, RECONCILED over LEDGER (empty since #4405, the intended steady state), CLASSIFIED not over drivers at all. A stale DRIVERS_DIR therefore printed `OK — 0 covered cell(s)` and exit 0. CI never had this exposure: lint.yml runs `pnpm check:driver-conformance`, which is `--self-test && audit`, and the self-test carried a discovery assertion. The false green was on the bare `node scripts/check-driver-conformance.mjs` the script's own header documents as a usage. Leaving the guard there was wrong twice over. It read `drivers.length >= 3 && drivers.includes('driver-sql')` — a hardcoded name and count inside the one script whose stated rule is that drivers come from disk and are never listed, so both wanted hand-editing on the next driver added or package moved, which is when the guard earns its keep. And its failure text named neither DRIVERS_DIR nor the stale path. DISCOVERED is now a fourth invariant in audit(), and its message names the directory searched. The self-test drives the invariant in both directions instead of standing in for it, and asserts nothing about which drivers exist. The case-set axis cannot rot this way and is left alone: CASE_SETS is a declared expectation, so a vanished spec/src/data fails CLASSIFIED's reverse direction with one error per case-set. The driver axis is disk-discovery with nothing declared to reconcile against — that asymmetry is why zero was reachable on one axis and not the other, and it is what DISCOVERED supplies. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VTj56JLhVk585TA8Ez3yGK
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
This was referenced Aug 2, 2026
os-zhuang
marked this pull request as ready for review
August 2, 2026 14:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4646.
问题
scripts/check-driver-conformance.mjs从磁盘发现 driver 包,根目录写死在DRIVERS_DIR。listDir吞掉 ENOENT 返回[],而三条不变量全部遍历「发现到的集合」:drivers→ 零次迭代LEDGER→LEDGER自 filter-logic conformance 少了两个 backend:MongoDB 的 translateFilter 从来没被 #3774 的标准收编 #4405 起为空(注释写明「the intended steady state」)于是一个失效的
DRIVERS_DIR会打印OK — 0 covered cell(s)并 exit 0。先把范围说准:CI 从来没有这个暴露。
lint.yml:304跑的是pnpm check:driver-conformance,即--self-test && audit,self-test 里有一条 driver 发现断言会先失败。假绿只出现在脚本 header 第 21 行文档化的node scripts/check-driver-conformance.mjs裸调用上。为什么不是「self-test 已经管了,就这样吧」
那条守卫本身有两个问题:
drivers.length >= 3 && drivers.includes('driver-sql')—— 一个硬编码的 driver 名和数量,出现在唯一一份声明「driver 从磁盘发现、绝不列表」(:145)的脚本里。下次加 driver 或挪包时两处都要手改,而那正是守卫该发挥作用的时刻。discovers driver packages from disk既不提DRIVERS_DIR也不说路径已失效,踩中的人得自己去找。改动
audit()增加第四条不变量 DISCOVERED,信息里带上实际搜索的目录:检查体抽成
discoveredErrors(drivers),好让 self-test 驱动这条不变量本身而不是它的替身;self-test 里对 driver 名和数量的断言随之删除。为什么不给 case-set 轴加同样的检查
它不会这样烂掉,实测过。
CASE_SETS是脚本里声明的预期,所以spec/src/data消失时 CLASSIFIED 的反向对账会逐条报错:driver 轴是纯磁盘发现、没有任何声明预期可对账 —— 这个不对称正是「零」只在一条轴上可达的原因,也正是 DISCOVERED 补上的那一块。给 case-set 轴再加一层是冗余。
验证
audit--self-testDRIVERS_DIR失效 →auditDRIVERS_DIR失效 →--self-testCASE_SETS_DIR失效 →auditpnpm check:driver-conformance(self-test + audit)全绿,eslint干净。失效场景通过复制脚本改常量的方式复现,未改动仓库内容。来由
来自 turso driver 归属评估(#4645):那次重组要把
packages/plugins/driver-*挪到packages/drivers/*,正好会动DRIVERS_DIR和 self-test 里那两个硬编码值。但这个缺陷独立于重组成立,所以单独出 PR,作为 #4645 的前置项。分支名不走
claude/issue-<n>-<slug>约定,因为本会话被指派了固定分支claude/turso-driver-evaluation-beoewn。Generated by Claude Code