Replies: 3 comments
EnglishRelevance selection answers which of these tests should run for this change, taking the suite's composition as given. I want to put the prior question on the table: why is each of these tests in Playwright at all? Most of them are not there because they need Electron. What is checkableFrom ls *.spec.ts | wc -l # 37
grep -lE "getComputedStyle|boundingBox|scrollTop|scrollHeight|clientHeight" *.spec.ts | wc -l # 19
grep -lE "window\.maka\." *.spec.ts | wc -l # 14
Correcting an earlier version of this comment, which listed Two questions decide the layerflowchart TD
A[one assertion] --> B{needs a real window?<br/>native input, process boundary}
B -->|yes| C[Electron E2E]
B -->|no| D{needs a real layout engine?<br/>geometry, scrolling, computed style}
D -->|yes| E[Storybook + Chromium]
D -->|no| F[component test]
The criterion is what must genuinely exist for the assertion to hold — not whether it is visual. Scroll anchoring and focus rings are behaviour and still land in the middle tier: Chromium is the same rendering engine, runs in parallel, and does not contend for OS focus. Applying itClassified by test title and assertion style. This part is judgement, not measurement:
The tracker shows the same overlap. Five flaky-E2E issues were filed over 2026-09-03 and 09-04: #4648 ( What it implies for the questionsQ1/Q3: if the Electron-requiring tier is ~9 tests plus a launch smoke, the suite finishes in under a minute and selection has nothing left to optimise — a capability vocabulary plus an enforcement axis is a large permanent surface for the 7-of-136 narrowing the opening post reports. Q6: enforcement tiers accommodate unreliable signal, but reliability here is a property of the layer, not the test; moved down, they stop needing observational status. Q7: yes — but the seam worth strengthening first is the renderer's testability, not CI ownership metadata. Tier three: checkedI went and looked, since this decides whether tier three is a move or a project. It is a move.
The seam exists. What is missing is a convention: all 13 DOM tests assert immediately after mount — Two things fall out. CounterweightOnly two tests prove anything cross-process, and neither covers launch or restart recovery, so that remains a genuine hole which shrinking must not widen. Some geometry does require Electron — frameless metrics, Suggested order
This does not contradict the model above — it changes what the model applies to. 中文相关性选择回答的是「这次改动该跑哪些测试」，它默认套件的构成是既定的。我想先提前置问题：这些测试为什么会在 Playwright 里？ 其中大多数并不是因为需要 Electron。 可复核的部分在 ls *.spec.ts | wc -l # 37
grep -lE "getComputedStyle|boundingBox|scrollTop|scrollHeight|clientHeight" *.spec.ts | wc -l # 19
grep -lE "window\.maka\." *.spec.ts | wc -l # 14
更正本评论的早前版本：我曾把 两个问题决定层次flowchart TD
A[一条断言] --> B{需要真实窗口吗？<br/>原生输入、进程边界}
B -->|是| C[Electron E2E]
B -->|否| D{需要真实布局引擎吗？<br/>几何、滚动、计算样式}
D -->|是| E[Storybook + Chromium]
D -->|否| F[组件测试]
判据是「这条断言成立最少需要什么真的存在」，而不是「它是不是视觉的」。滚动锚定和焦点环都是行为，仍落在中间层：Chromium 是同一个渲染引擎，可并行，也不争抢系统焦点。 套用结果按测试标题和断言方式归类。这部分是判读，不是测量：
tracker 上能看到同样的重合。2026-09-03 到 09-04 两天里开了五个 flaky E2E issue：#4648（ 这对那些问题意味着什么Q1/Q3：如果真正需要 Electron 的只有约 9 条加一个启动 smoke，整套跑完不到一分钟，选择机制也就没有可优化的对象——为开头帖给出的「136 个提交里缩小 7 个」维护一套能力词表加一条 enforcement 轴，代价偏重。Q6：enforcement 分级是为容纳不可靠信号，但这里的可靠性是层的属性而非测试的属性；下沉后它们不再需要观察态。Q7：应该——但更该先补的接缝是渲染层的可测性，不是 CI 的归属元数据。 第三层：已核实这一步决定第三层是搬迁还是项目，所以我去查了。是搬迁。
接缝是有的，缺的是惯例：13 个 DOM 测试全部在挂载后立即断言—— 由此带出两点。 反面目前只有两条测试真正证明了跨进程行为，且都不覆盖启动与重启恢复，所以那仍是真实的空洞，缩小时不能让它变大。有些几何确实需要 Electron——无边框度量、 建议顺序
这不与上面的模型冲突，只是改变了它作用在什么之上。 |
|
I did a first pass over the latest 300 completed CI runs to test the Among runs that had an executable first attempt fail and later passed on Looking at seven recurring scenarios, five appear to need a real layout I still find the relevance/enforcement split in the opening post useful, Would a phased order make sense here?
|
|
English Updated: the original version of this comment reported four PRs as a data point for the layering hypothesis. Having since traced why those tests started failing, the disclosure stands but the conclusion has changed — the failures are a composition problem, not a placement one, and #4741's diagnosis (mine) was wrong about the cause. Disclosure first. Four PRs since this thread opened:
111 tests in 37 files → 86 in 35. 6.1 → 4.0 minutes. The flakes have a start date
Six reports in the 43 hours after #4523. Zero for these files in the three weeks before — the two earlier ones are a different file. What changed, in the deleted comment's own wordsThe
The 19 of 37 spec files asserting computed style, bounding boxes or scroll offsets are exactly the contracts that sentence names. CI now runs four Xvfb displays. The reason and the setting went together. The second half is subtler: process-per-test isolation was replaced by a reset protocol — await worker.page.waitForSelector('[data-turn-id]', { timeout: 20_000 });That is #4707 exactly. Its trace shows the wait completing on That one is not load sensitivity. It is a premise the tests were written against — a fresh renderer per test — that stopped holding. On a per-test enforcement axis it reads as an unreliable test and gets demoted, which hides a harness defect instead of surfacing it. Why this is a composition problem, not a selection problemRelevance asks should this test run for this change; enforcement asks should its result block merging. Both are relations between source and test. Neither can see what actually moved here. The deeper point is that #4523 was not gratuitous. The suite was slow because it holds a large majority of tests that never needed Electron, and the pressure that produced four workers and warm reuse came from that composition. Selecting fewer of those tests does not change it: the ones that still run, run in a runner whose premises no longer hold. How far the composition has drifted, in one number: after #4741, the warm-reuse machinery — 87 lines of fixture code, a So I would put @Phoenix500526's step 1 ahead of the selector rather than beside it. A first pass over the 35 remaining files by what each assertion actually needs:
If tier one lands near that size, What the moves cost#4752 was cheap only because the target module and its test file already existed. Where that is not true, tier three is an extraction first — the part of @Phoenix500526's phased order I expect to dominate. Two things worth keeping visible from that move. It left one assertion uncovered in any tier: None of this argues against #4523. It bought a real 40% and it added the release bridge rather than ignore isolation. It under-estimated one premise, and the cost landed on the tests that depended on it most. I have rewritten #4761 around this rather than around the layout tier alone. 简体中文更新：这条评论原来是把四个 PR 作为分层假设的数据点。之后追清了那些测试为什么开始挂，交代部分不变，但结论变了——这是成分问题不是放置问题，而 #4741 的诊断（我写的）把原因说错了。 先交代。这个串开起来之后有四个 PR：#4741 整个删掉 抖动有明确起点。 #4523 于 09-02 07:08Z 合入，之后 43 小时内出现六份 flake 报告（#4573、#4648、#4664、#4675、#4688、#4707）；在它之前三周这几个文件一份都没有——更早的 #2948、#3289 是另一个文件。 #4523 删掉的那段配置注释把不变量写得很清楚：单 worker 的理由是「并发的隐藏窗口会节流动画帧、共享 OS 焦点，从而使几何与焦点契约失效」。37 个 spec 里断言计算样式、bounding box 或滚动偏移的那 19 个，正是这句点名的契约。原因和设置一起没了。 更隐蔽的一半：每测试一进程的隔离被换成测试间的重置协议，就绪判据是 为什么这是成分问题，不是选择问题。 relevance 与 enforcement 都是源码与测试之间的关系，都看不见这次真正移动的东西。更关键的是 #4523 并非多余：套件慢，是因为里面绝大多数测试从不需要 Electron，四个 worker 和 warm 复用正是这个成分逼出来的。少选几个不改变它——真正跑起来的那些，仍然跑在一个守不住前提的 runner 里。 成分漂移到什么程度，有个最干脆的量度：#4741 之后，整套 warm 复用机制（87 行夹具、产品侧观察注册表上的 所以我倾向把 @Phoenix500526 的第一步放在选择器之前而不是并列。按每条断言实际需要什么，对剩下 35 个文件做的第一遍：约 27 个既无几何也不跨进程；约 42 个需要真实布局引擎但不需要 Electron；真正需要 Electron 的 CI 测试约 4 个。 第一层收到这个规模后， 搬移的成本。 #4752 之所以便宜，只是因为目标模块和它的测试文件本来就在；不具备这个条件的地方要先做一次提取——我预计那才是主要成本。两件值得摆在明面上的事：#4752 留下一条断言在任何一层都没覆盖（ 这些都不是反对 #4523。它换来的 40% 是真的，也确实加了 release bridge 而不是无视隔离。它低估了一个前提，代价落在最依赖那个前提的测试上。 我已经按这个思路重写了 #4761，而不只是围绕布局那一层。 |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
English
We have been looking at the role of Desktop E2E in CI. Today the planner treats the Playwright suite as one indivisible surface: once a change is considered Desktop-relevant, all 37 spec files / 110 tests run, and any failure blocks the required
testcheck. A recent full run takes roughly five to six minutes.That broad gate has value: real regressions have surfaced through it. But it also means a timing- or load-sensitive test can block a change that never touched the behavior it exercises. The question is therefore not simply how to make the suite faster. It is how CI should model relevance and enforcement without weakening the evidence required to merge.
This discussion is intentionally not an implementation proposal. We would like to agree on the problem and the constraints before choosing a mechanism.
What we currently believe
An idealized direction worth discussing
At the level of mechanism—not file format or implementation—we currently find one model especially coherent:
Source changes would resolve to the product capabilities they may affect, while each E2E scenario would state which capabilities it provides evidence for. CI would join the two through that shared vocabulary. A scope would name something stable such as conversation, session navigation, workbar, settings, or shell behavior—not a particular spec file and not an implementation directory merely because it exists today.
The selection semantics would be deliberately conservative:
Separately, every selected test would have an enforcement status. Stable evidence could remain merge-blocking, while useful but load-sensitive evidence could stay observational until its signal justifies promotion. Relevance would answer should this scenario run for this change?; enforcement would answer should this result decide whether the change may merge?
The appealing property is monotonic safety: incomplete metadata costs time by widening the run, rather than creating a silent coverage hole. As architectural ownership improves, selection can become narrower without changing the underlying model.
This is only a candidate abstraction. It does not decide whether scopes should be expressed through Playwright tags, a manifest, directory ownership, generated metadata, or some combination. Those choices should follow after we agree that the relationship and its failure behavior are the right ones.
Evidence and constraints
#4478 measured Desktop E2E as the largest single part of the required job and found that 57% of sampled commits selected the entire suite.
#4583 explored per-spec selection through static import reachability. Review found that this relation was not sound for Electron: most main/preload/renderer product files were invisible from the specs, and 11 of 39 E2E-triggering commits in a replay selected zero tests—including changes to surfaces exercised by the suite. Adding the runtime application entry points would restore reachability but would also make nearly every spec reachable again.
We replayed the current planner over the latest 300 first-parent commits. 174 selected Desktop E2E; 38 were already full-suite plans. Of the remaining 136, a deliberately conservative selector based only on current feature ownership could safely narrow just seven (four spec-only changes and three feature-scoped changes). The current ownership structure is therefore a useful foundation, but not yet enough on its own to produce a meaningful reduction. Main, legacy renderer, preload, shared contracts, and the mostly flat UI package remain the difficult parts.
There is also a workflow boundary to preserve: the current
e2eplan output controls the Playwright suite, the Browser WebContentsView smoke, and the alignment audit together. Those checks do not necessarily have the same relevance relation.Runtime coverage could provide evidence about which code a scenario exercised, but it would still describe observed execution rather than every path a test is capable of protecting. We do not think it should be treated as proof of irrelevance by itself.
Questions for the project
main, a schedule, release qualification, or some combination?The goal is not to run fewer tests as an end in itself. It is to make each merge decision depend on the tests that provide relevant, trustworthy evidence, while retaining a clear fallback for uncertainty.
中文
最近几次 CI 把 Desktop E2E 的问题暴露得比较明显：现在只要 planner 认为改动碰到了桌面端，就会把 37 个 spec、110 个测试整套跑完，其中任何一个失败都会卡住 required
test。一轮通常要五六分钟。全跑当然不是完全没价值，过去确实有产品问题是被 E2E 翻出来的。但另一面也很现实：某个依赖负载、动画帧或者滚动稳定时机的测试抖一下，就可能把一个根本没改这块功能的 PR 拦住。我们真正需要讨论的不是“怎么少跑几个测试”，而是 CI 凭什么判断哪些测试和这次改动有关，以及哪些失败值得阻止合并。
这里先不定实现方案，只想把我们目前的判断和还没想清楚的问题摆出来。
我们目前的看法
一个我们觉得足够干净的方向
先不讨论最终写成 tag、JSON 还是目录规则，只看机制本身，我们目前比较认同下面这个抽象：
源码这一侧说明“这次可能影响哪些能力”，测试这一侧说明“我能为哪些能力提供证据”，planner 只负责把两边接起来。这里的范围应该是相对稳定的产品概念，例如会话阅读、Session 导航、Workbar、设置或窗口外壳，而不是某个 spec 文件名，也不应该因为今天代码恰好放在某个目录，就把目录结构原样搬进 CI。
选择规则宁愿保守一些：
另一个轴再决定测试结果有没有合并否决权。稳定、可信的证据继续卡合并；方向有价值但容易受负载影响的场景先作为观察项。这样“该不该为这次改动运行”和“失败后该不该拦住合并”不会混成一件事。
这个模型最吸引我们的地方，是它在信息不完整时只会多花时间，不会悄悄留下覆盖空洞。随着 ownership 逐渐清楚，选择自然可以越来越细，底层规则不需要换一套。
这仍然只是拿出来讨论的抽象，不代表已经选定 Playwright tag、中心清单、目录继承或自动生成中的任何一种。具体载体可以等大家先认可关系模型和兜底方式以后再谈。
已经确认的事实
#4478 的统计里，Desktop E2E 是 required job 最大的一块；抽样提交中有 57% 会把整套测试拉起来。
#4583 尝试过按静态 import 可达性选 spec，但 Electron 的运行时边界让结果失真。回放中 39 个本应触发 E2E 的提交有 11 个选成了零测试，其中包括设置页、共享 Session 控件等真实界面改动。要是把应用入口补进依赖图，绝大多数 spec 又会重新变成“全部相关”，等于没有选择。
我们又用当前 planner 回放了主线最近 300 个 first-parent commit：174 个会触发 Desktop E2E，其中 38 个本来就是 full plan。剩下 136 个里，如果只认可现在已经落到 feature ownership 下的边界，能够放心缩小的只有 7 个——4 个只改 spec，3 个只碰明确的 feature slice。说明现有架构方向能承载这件事，但 main、旧 renderer、preload、shared contract 和扁平的 UI 包还没有足够清楚的归属，暂时不能指望自动选择马上省下很多时间。
还有一处容易误伤：当前 planner 的
e2e输出同时控制 Playwright、Browser WebContentsView smoke 和 alignment audit。这三类检查验证的东西不同，不能默认共用同一套“相关性”。运行时 coverage 可以拿来帮助理解某个场景实际走过哪些代码，但它只能说明这次执行碰到了什么，不能证明没碰到的路径永远与测试无关。所以它可以当线索，不能单独拿来做跳过测试的依据。
想听听大家的意见
main后、定时任务、发布验证，还是几者结合？目标不是为了数字好看而少跑测试，而是让每次合并依赖真正相关、值得信任的证据；遇到不确定的地方，也始终有清楚的兜底。
All reactions