ci: shard command_test so compat-offline-core can finish - #449
Merged
Conversation
…ishing `compat-offline-core` has not run to completion in weeks. `cargo test --all` is fail-fast, and the lib suite has been failing since 2026-07-28, so cargo aborted there every time and never reached a single integration binary. Runs looked like 53-minute failures; the suite behind them was never measured. Once the lib failures were fixed on a branch, the job ran past the 360-minute platform cap. Three separate problems, each only visible because the run got further than any before it: 1. `tests/command_test.rs` compiles 150 modules (~2950 tests) into one binary — deliberately, to avoid one binary per command. Roughly a third of them take the process-global cwd lock every `ChangeDirGuard` holds, so it runs at about one core no matter the machine: 479 of its tests logged "running for over 60 seconds" while the host sat near 0.3 load. It is lock-bound, not CPU-bound, and the only thing that shortens it is more than one PROCESS. Split into `compat-offline-command`, a four-way matrix. Measured: 739 tests per shard, ~2h50m each, comfortably inside the cap. 2. `command::switch_test::test_detach_head_basic` ABORTED the whole binary with a stack overflow. Not recursion — an unoptimized async state machine inlines every awaited future into one frame, and libtest's 2 MiB test threads cannot hold `switch::execute`'s chain. Reproduced in isolation at 2 MiB, passes at 4 MiB. `RUST_MIN_STACK` is 16 MiB here for headroom. The shipped CLI runs those futures on the 8 MiB main thread, so this is a harness limit, not a product one. 3. compat-offline-core now names its targets instead of using `--all`, so the two halves can be split. Targets whose `required-features` are off must be dropped rather than named: `cargo test --all` skips them silently, but `--test <name>` on one is a hard error. Verified: 208 test targets, 200 after the filter, and the eight dropped are exactly the ones this job's later steps already run explicitly with the feature each needs. Coverage is unchanged. Shards partition by enumerated test NAME, so the split is exhaustive and disjoint by construction — a rename cannot drop a test the way a hand-maintained filter list would. Both jobs refuse to run a suspiciously small set rather than pass on partial coverage, and both carry `timeout-minutes: 350` so an overrun fails with its log intact instead of being reaped at the cap. Validated on #447: shard 0 finished 737/739 and shard 1 finished 738/739, where every previous attempt had timed out or aborted with no result at all. The handful of remaining failures are pre-existing and unrelated to this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
说明：本 PR 无法自我验证
on:
pull_request:
paths-ignore:
- '.github/**'
- 'docs/**'所以只改 workflow 的 PR 永远不会触发 因此本 PR 的验证证据来自 #447，那里同样的
对照：在此之前每一次 剩余的少量失败是既有问题，与本 PR 无关，正由 #450 单独处理。 |
Contributor
Author
完整验证结果（四分片全部跑完）#447 上这一轮已全部结束，四个分片无一超时、无一 abort：
每片 739 条，合计 2956 条，与本地验证的划分完全一致；最长 3h05m，距 350 分钟上限有充分余量。 对照：在此改动之前， 合计 7 条失败，全部是既有问题、全部确定性（隔离单跑同样失败），与本 PR 无关。其中测试侧的部分由 #450 处理；另有几条属于产品与测试的契约不一致（错误码、HTTP 状态码、警告文案），需要维护者定夺，已在 #450 中列出。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
compat-offline-core已经数周没有完整跑完过。cargo test --all是 fail-fast 的，而 lib 套件自 2026-07-28 起一直有失败，cargo 每次都在 lib 阶段中止，一个集成测试二进制都没跑到过。表面上看是「53 分钟后失败」，实际上它背后的整个集成套件从未被测量。把 lib 的失败在一个分支上修好之后，这个 job 直接顶破了 GitHub 的 360 分钟平台上限。随后暴露出三个彼此独立的问题——每一个都是「跑得比以往任何一次都远」才看得见的：
三个问题与修法
1.
command_test是锁瓶颈，不是算力瓶颈tests/command_test.rs把 150 个模块（约 2950 条测试）编进一个二进制——这是有意为之，为了省下「每个命令一个二进制」的编译开销。但其中约三分之一会去抢ChangeDirGuard持有的那把进程级 cwd 锁，于是无论机器多少核，它实际都只跑在一个核上：日志里 479 条测试单条超过 60 秒，而同期宿主机 load 只有 0.3。加核没用，只有加进程有用。所以拆成
compat-offline-command四路矩阵。实测：每片 739 条，约 2h50m，安全落在上限内。2.
switch_test栈溢出导致整个二进制 abortcommand::switch_test::test_detach_head_basic触发fatal runtime error: stack overflow，整个进程 abort，连test result:都没有。不是递归：debug 构建下 async 状态机会把每一层 await 的 future 内联进同一个栈帧，
switch::execute的调用链装不进 libtest 给测试线程的 2 MiB 栈。隔离单跑可稳定复现：2 MiB 必溢出，4 MiB 稳定通过。这里设RUST_MIN_STACK为 16 MiB 留足余量。发布的 CLI 跑在 8 MiB 主线程上，这是测试框架的限制，不是用户能碰到的产品缺陷。
3. 目标枚举必须过滤
required-features拆分要求
compat-offline-core点名目标而非用--all。但required-features未启用的目标必须丢弃而不能点名：cargo test --all会静默跳过它们，用--test <name>点名则是硬错误（target agent_live_gate_test requires the features: test-live-agent）。已验证：共 208 个 test target，过滤后 200 个；被丢掉的 8 个正是本 job 后续步骤已经带着各自 feature 显式运行的那些。
覆盖面不变
lib、bins、doctests 和其余 200 个集成目标留在
compat-offline-core，command_test在分片里跑。分片按枚举出的测试名做取模划分，穷尽且互斥是构造保证的——重命名或新增测试不可能从所有分片里漏掉，而手工维护的过滤名单迟早会漏。本地验证：2956 条，四片各 739，并集去重 2956，交集为空。
两个 job 都会在枚举数异常偏小时拒绝运行而不是带着残缺覆盖面通过，并都设了
timeout-minutes: 350，让超时以完整日志失败，而不是被平台在 360 分钟处直接砍掉。验证
在 #447 上实测：分片 0 完成 737/739，分片 1 完成 738/739 —— 而在此之前每一次尝试都是超时或 abort，拿不到任何结果。剩下的少量失败是既有问题，与本改动无关。
🤖 Generated with Claude Code
Note
Low Risk
Workflow-only changes; risk is CI coverage drift if enumeration logic is wrong, mitigated by minimum-count checks and unchanged feature-gated steps.
Overview
Splits the monolithic
command_testrun into a newcompat-offline-commandjob with a 4-shard matrix (fail-fast: false). Each shard buildscommand_testonce, lists every test name, assigns tests by index moduloSHARD_COUNT, and runs them with--exactso parallel processes avoid the cwd-lock bottleneck that kept a single job on one core past GitHub’s time limit.compat-offline-coreno longer runscargo test --all. It runs--lib,--bins, doctests, and every integration test target exceptcommand_test, discovered viacargo metadatawhile dropping targets whoserequired-featuresare not enabled (matching--all’s silent skip). Feature-gated suites stay on the existing explicit steps. Both jobs settimeout-minutes: 350andRUST_MIN_STACK=16MiBso libtest threads don’t abort on large unoptimized async stacks (e.g.switch_test).Guardrails: each job fails if enumeration looks truncated (<100 integration targets or <2000 command tests).
Reviewed by Cursor Bugbot for commit 18db9e7. Bugbot is set up for automated code reviews on this repo. Configure here.