[pull] next from umijs:next

benchmark/README.md

@@ -1,73 +1,116 @@
 # qiankun benchmark
 
-This workspace compares the cold-load path of the locally built qiankun package with a pinned Wujie release. It is intentionally separate from `e2e/`: Playwright Library only drives the browser boundaries, while all timing and core-element detection run inside the page.
+This workspace measures cold micro-app loading from the instant an adapter invokes its mount path until the app's core element is paintable. It compares the locally built qiankun package with qiankun 2.10.16, native iframe, Wujie 2.1.0, MicroApp 1.0.0-rc.32, and Garfish 1.19.7. Playwright Library owns browser lifecycle only; timing and paint detection run inside the page.
 
 ## Commands
 
 Run from the repository root:
 
 ```bash
-# Five samples per product cell. Validates plumbing only; not performance data.
+# Fast plumbing checks: one browser trial and five samples per cell.
 pnpm benchmark:check
+pnpm benchmark:ecosystem:check
+pnpm benchmark:ssr-streaming:check
 
-# Five warmups, 100 valid attempts per product cell, and 50 samples per A/A arm.
+# Formal runs: three independent browser trials, each with five warmups,
+# 100 samples per product cell, and 100 samples per A/A arm.
 pnpm benchmark:smoke
+pnpm benchmark:ecosystem
+pnpm benchmark:site-isolation
+pnpm benchmark:ssr-streaming
 ```
 
-To compare a performance change against an interleaved baseline, snapshot the fully bundled host before editing package code, then run the comparison after the change:
+The check commands disable the A/A gate and are not performance evidence. Formal run results are written to `benchmark/results/<timestamp>-<commit>/`; raw warmups and measurements are retained.
 
-```bash
-# On the clean baseline revision. Fails instead of overwriting an existing snapshot.
-pnpm benchmark:baseline
-
-# After making and building the candidate change. Runs 100 paired samples per revision.
-pnpm benchmark:compare
+To compare two repeated runs by absolute median, use the guarded comparison command:
 
-# Five paired samples for plumbing only; the improvement gate is disabled.
-pnpm benchmark:compare:check
+```bash
+pnpm --filter @qiankunjs/benchmark run compare:results -- \
+  benchmark/results/<reference> benchmark/results/<candidate>
 ```
 
-The baseline snapshot contains the complete Vite host bundle, so its qiankun, loader, sandbox, and shared dependency graph cannot mix with candidate packages. Revision samples use different host origins, fresh BrowserContexts, a balanced alternating order, and the same streamed fixture. A formal comparison passes only when every sample is valid and the paired bootstrap 95% confidence interval is entirely below 0%.
+This command rejects results whose harness fingerprints differ. The fingerprint covers the environment, Chromium and Playwright versions, launch/context options, suite definition, sampling configuration, selected adapter configuration and versions, selected Vite entry graph, fixture behavior, and timing/statistics source. Cross-run medians remain diagnostic; the paired comparisons inside one run are the primary result.
 
 Install the pinned Chromium revision once if Playwright reports that it is missing:
 
 ```bash
 pnpm --filter @qiankunjs/benchmark exec playwright install chromium
 ```
 
-Local snapshots are written to `benchmark/artifacts/`; run results are written to `benchmark/results/<timestamp>-<commit>/`. Both are gitignored.
+## Suites
 
-## Matrix
+Suites are explicit and independent; adding an ecosystem framework does not alter the frozen core schedule.
 
-The matrix is explicit rather than a cartesian product:
+| Suite | Cells | Purpose |
+| --- | --: | --- |
+| `core` | 8 | qiankun isolation cost plus buffered and streamed comparisons with native iframe and Wujie |
+| `site-isolation` | 6 | same-site anchors and cross-site entries for qiankun, native iframe, and Wujie |
+| `ecosystem-html` | 6 | one canonical same-site, buffered, isolated cell for each framework/version |
+| `ssr-streaming` | 6 | literal SSR progressive reveal compared with an identical delayed buffer, native iframe, qiankun v2, Wujie, and Garfish |
 
-1. qiankun without sandbox or style isolation
-2. qiankun with sandbox only
-3. qiankun with sandbox and style isolation
-4. Wujie with its intrinsic iframe and Shadow DOM isolation
-5. qiankun with full isolation and a three-chunk HTML response
-6. Wujie with the same three-chunk HTML response
+The core suite contains qiankun with no isolation, sandbox only, and sandbox plus style isolation; same-site buffered native iframe and Wujie; and streamed native iframe, qiankun, and Wujie cells.
 
-Wujie does not expose switches that disable its JavaScript or CSS isolation. Its off-cases are therefore marked N/A instead of being compared against a semantically different configuration.
+The ecosystem suite intentionally adds only these canonical cells:
+
+- native iframe;
+- qiankun 2.10.16 with its Proxy sandbox and experimental scoped CSS isolation;
+- the locally built qiankun with sandbox and style isolation;
+- Wujie with its intrinsic iframe and Shadow DOM isolation;
+- MicroApp with its default sandbox and scoped CSS;
+- Garfish with Browser VM sandbox, effect collection enabled (`cache: false`), and strict Shadow DOM isolation.
+
+The ecosystem suite stays intentionally small and does not duplicate its canonical cells under streamed delivery. Native HTML parsing and the locally built qiankun can render progressively; qiankun 2.10.16, Wujie, MicroApp, and Garfish accept chunked responses but currently consume the complete HTML before parsing. The core suite retains Wujie's streamed-response diagnostic because Wujie is qiankun's phase-one comparison target. Framework isolation semantics are also not identical: qiankun v2 uses selector rewriting, the local qiankun uses CSS `@scope`, and Garfish/Wujie use Shadow DOM. The suite compares observed user-visible latency under these isolated configurations, not security equivalence.
+
+The dedicated SSR suite is the canonical streaming comparison. Unlike the core suite's client-rendered chunk diagnostic, its response contains literal server-rendered markup: the first chunk includes the stylesheet, critical element, 12 visible rows, and native paint reporter; the second adds 44 deferred rows; and the final chunk adds another 44 rows, the stream-tail marker, serialized SSR data, the only external lifecycle entry script, and the closing tags. The default schedule is 0/50/100 ms. Keeping the entry script last lets progressive consumers reveal the SSR core while framework settlement and cleanup still wait for the complete response and lifecycle execution. An identical-byte qiankun v3 control withholds the complete response until 100 ms, isolating progressive reveal from fixture content and response-tail duration. Native iframe and qiankun v3 can expose the critical markup before EOF; the pinned qiankun v2, Wujie, and Garfish adapters currently buffer the complete HTML response first.
+
+This is a deterministic SSR-shaped fixture, not a React/Vue server-rendering or hydration throughput benchmark. Its entry script supplies framework lifecycles and marks the already rendered core as mounted. The streamed-versus-delayed result therefore represents earlier CSS discovery, progressive DOM insertion, and hiding response-tail latency; it must not be interpreted as pure parser, sandbox, or loader CPU throughput.
+
+The cross-site suite uses identical response bytes for all products. Native iframe must create an OOPIF under Chromium's `--site-per-process`; a separate preflight browser verifies that condition. Wujie 2.1.0 fetches its cross-site entry while retaining a host-origin `srcdoc` JavaScript realm, so that cell is described as a cross-site entry rather than a cross-site iframe.
 
 ## Measurement contract
 
-- `t0`: immediately before `loadMicroApp` or `startApp` is called in the host page.
-- `t1`: the core marker exists, has non-zero geometry, is visible, has the expected CSS sentinel, and survives two `requestAnimationFrame` callbacks.
-- The framework must still finish mounting successfully; settlement is awaited outside the measured duration.
-- Buffered and streamed entry responses contain identical bytes. Streamed HTML is emitted at 0/50/100 ms by default.
-- The browser process is reused, but every attempt receives a fresh BrowserContext and page.
-- No trace, video, screenshot, HAR, route interception, retry, or parallel execution is enabled.
-- Raw warmup and measured samples are retained; none are silently replaced or filtered as outliers.
+- `t0`: immediately before the selected adapter invokes `loadMicroApp`, `startApp`, `renderApp`, native iframe navigation, or `Garfish.loadApp`.
+- `t1`: the core marker has non-zero geometry, contains `[data-benchmark-critical]`, is visible, has the expected CSS sentinel, and survives two `requestAnimationFrame` callbacks. It does not require `data-mounted=true`, so visible SSR content can finish this metric before client lifecycle settlement.
+- Framework settlement and cleanup must succeed, but both remain outside the measured duration.
+- After settlement, the harness separately requires `data-mounted=true`; a framework cannot pass by exposing SSR markup without completing its lifecycle.
+- Every attempt gets a fresh BrowserContext and page. Every formal trial gets a newly launched Chromium process.
+- Adapter module initialization happens before `t0` and may not request fixture resources; the runner fails the sample if it observes such a request.
+- Native iframe reports its child-document paint timestamp through an origin-, source-, version-, and token-validated `postMessage`.
+- Buffered and streamed responses contain identical bytes. Streamed HTML is emitted at 0/50/100 ms by default.
+- The SSR delayed-buffer control contains the same bytes as its streamed counterpart and sends them together at the time the stream would have ended.
+- HTML responses are `no-store`; deterministic static assets are immutable so qiankun's walk-ahead preload can satisfy its subsequent sandbox fetch. Fresh BrowserContexts keep every measured sample cold.
+- No trace, video, screenshot, HAR, route interception, retry, outlier replacement, or parallel execution is enabled.
+- Host bundle download and top-level framework initialization are outside this benchmark's scope.
+
+The primary metric remains `t1 - t0`. Reports also retain framework-settlement time (`settled - t0`) and the entry HTML resource's response-end time (`PerformanceResourceTiming.responseEnd - t0`) as diagnostics; response end does not represent the entire static-resource waterfall. `response end - core paint` is reported as paint lead: a positive value means this harness observed the user-visible core before the complete HTML response arrived. Native iframe navigation timing is not visible as a parent-page resource entry, so its response-end and paint-lead diagnostics are `n/a`; its child-reported primary paint duration remains valid. These per-variant medians explain the result but do not replace the primary metric or participate in paired comparisons, confidence intervals, or pass/fail gates.
 
-The primary statistic is the paired median relative delta with a 10,000-resample bootstrap 95% confidence interval. Median, mean, p75, p95, MAD, standard deviation, and CV are also retained. Positive comparison deltas mean the candidate is slower than the reference.
+Rounds are paired within each independent browser trial. The primary estimate is the median of per-trial median log ratios. Its 95% confidence interval comes from a hierarchical 10,000-resample bootstrap that resamples trials first and rounds second, so a `3 × 100` run is not treated as 300 independent observations. Median, mean, p75, p95, MAD, standard deviation, CV, and each trial's summary are also retained. Positive deltas mean the candidate is slower than the reference.
 
 ## A/A calibration
 
-Before the product matrix, two aliases of the exact same fully isolated qiankun variant are interleaved. The formal run requires:
+Each trial first interleaves two aliases of the exact same selected qiankun cell. The SSR suite uses its streamed qiankun v3 cell; the other suites use the canonical fully isolated cell. Both every trial and the aggregate must satisfy:
 
 - absolute paired median delta no greater than 3%;
 - the bootstrap 95% interval includes 0%;
 - interval width no greater than 10 percentage points.
 
-The benchmark is manual in phase one and is not part of the regular PR CI gate.
+## Revision comparison
+
+To validate a qiankun optimization, snapshot the fully bundled host from a clean baseline revision, then run an interleaved candidate comparison:
+
+```bash
+# On the clean baseline revision. Existing snapshots are never overwritten.
+pnpm benchmark:baseline
+
+# After making and building the candidate change.
+pnpm benchmark:compare
+
+# Five paired samples for plumbing only; the improvement gate is disabled.
+pnpm benchmark:compare:check
+```
+
+The snapshot contains the complete Vite host bundle, preventing baseline and candidate package graphs from mixing. It also records a revision-harness compatibility hash that excludes qiankun package sources but covers the adapter, fixture, timing contract, runner, statistics, browser options, and toolchain inputs. The runner rejects stale snapshots when any of those inputs change; recreate old schema snapshots before running a revision comparison.
+
+Revision mode remains a single browser trial with balanced baseline/candidate rounds and fresh BrowserContexts; it passes only when every sample is valid and the paired-bootstrap 95% confidence interval is entirely below 0%. Use `--scenario=sandbox` with a named snapshot to isolate the buffered sandbox-only path.
+
+Local snapshots are written to `benchmark/artifacts/`. Snapshots and results are gitignored, and the benchmark remains a manual gate rather than part of regular PR CI.

benchmark/compare-results.mjs

@@ -0,0 +1,29 @@
+import { readFile, stat } from 'node:fs/promises';
+import { join, resolve } from 'node:path';
+
+import { compareBenchmarkResults } from './src/result-comparison.mjs';
+
+async function readResult(argument) {
+  const path = resolve(argument);
+  const resultPath = (await stat(path)).isDirectory() ? join(path, 'result.json') : path;
+  return JSON.parse(await readFile(resultPath, 'utf8'));
+}
+
+function formatPercent(value) {
+  return `${value >= 0 ? '+' : ''}${value.toFixed(2)}%`;
+}
+
+const [referencePath, candidatePath, ...extra] = process.argv.slice(2);
+if (!referencePath || !candidatePath || extra.length > 0) {
+  throw new Error('usage: node compare-results.mjs <reference-result> <candidate-result>');
+}
+
+const rows = compareBenchmarkResults(await readResult(referencePath), await readResult(candidatePath));
+console.log('| Variant | Reference median (ms) | Candidate median (ms) | Absolute-run delta |');
+console.log('| --- | ---: | ---: | ---: |');
+for (const row of rows) {
+  console.log(
+    `| ${row.label} | ${row.referenceMedian.toFixed(2)} | ${row.candidateMedian.toFixed(2)} | ${formatPercent(row.relativeDeltaPercent)} |`,
+  );
+}
+console.log('\nCross-run absolute medians are diagnostic only; within-run paired comparisons remain primary.');

benchmark/fixtures/host/garfish.html

@@ -0,0 +1,13 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>Garfish benchmark host</title>
+    <link rel="stylesheet" href="/src/host.css" />
+  </head>
+  <body>
+    <div id="micro-app-container"></div>
+    <script type="module" src="/src/garfish.ts"></script>
+  </body>
+</html>

benchmark/fixtures/host/micro-app.html

@@ -0,0 +1,13 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>MicroApp benchmark host</title>
+    <link rel="stylesheet" href="/src/host.css" />
+  </head>
+  <body>
+    <div id="micro-app-container"></div>
+    <script type="module" src="/src/micro-app.ts"></script>
+  </body>
+</html>

benchmark/fixtures/host/native.html

@@ -0,0 +1,13 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>Native iframe benchmark host</title>
+    <link rel="stylesheet" href="/src/host.css" />
+  </head>
+  <body>
+    <div id="micro-app-container"></div>
+    <script type="module" src="/src/native.ts"></script>
+  </body>
+</html>

benchmark/fixtures/host/qiankun-v2.html

@@ -0,0 +1,13 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>qiankun v2 benchmark host</title>
+    <link rel="stylesheet" href="/src/host.css" />
+  </head>
+  <body>
+    <div id="micro-app-container"></div>
+    <script type="module" src="/src/qiankun-v2.ts"></script>
+  </body>
+</html>