Skip to content

ci(playwright): raise chromium shard budget 19 → 20 min - #30616

Closed
chirag-madlani wants to merge 2 commits into
mainfrom
ci/bump-chromium-shard-budget
Closed

ci(playwright): raise chromium shard budget 19 → 20 min#30616
chirag-madlani wants to merge 2 commits into
mainfrom
ci/bump-chromium-shard-budget

Conversation

@chirag-madlani

@chirag-madlani chirag-madlani commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

The chromium lane hit the shard planner's hard ceiling on the merge queue for PR #30563 (run 30394014577):

```
Lane chromium needs more than 24 shards to stay within the 19-minute plan budget;
the heaviest shard is predicted at 19.1m
##[error]Process completed with exit code 1.
```

The planner refused to build a valid plan → `plan-playwright` failed → all downstream jobs skipped → `playwright-summary` failed → merge queue blocked.

Root cause

Aggregate growth. The chromium suite currently holds ~1053 min of test content per the committed timing baseline — 90.6 % of the 1163 min planner ceiling (24 shards × 3 workers × 19 min × 0.85 efficiency). LPT bin-packing is working; there's just no headroom left, and the current heaviest-shard prediction is 0.1 min over the hard budget.

Change

One constant: raise the chromium shard budget from 19 → 20 min (aligns it with `TARGET_MS` used by the other lanes).

```python

  • COMMON_SHARD_BUDGET_MS = 19 * 60 * 1000
  • COMMON_SHARD_BUDGET_MS = 20 * 60 * 1000
    ```

Wall clock is unaffected — the change only widens the acceptance threshold. The predicted heaviest shard runtime is unchanged; the planner will now accept a plan it previously refused.

Follow-up (not in this PR)

This is a stop-gap. Investigation of the timing baseline turned up the actual load hot spots:

Duration Test
605 s (10.1 min!) `TestCaseImportExportE2eFlow.spec.ts › EditAll User: Complete export-import-validate flow`
603 s (10.0 min!) `TestCaseImportExportE2eFlow.spec.ts › Admin: Complete export-import-validate flow`
362 s (6.0 min) `ContextCenterArticles.spec.ts › Article edits and navigation-flushed titles persist`

Two ~10-minute tests in a single file are effectively immovable in LPT scheduling — they consume 20 % of a shard budget between them. Splitting or trimming those three tests would give ~15 min of aggregate breathing room without another budget bump. Filing separately.

Test plan

  • Retriggering PR fix(ui): update styling imports and fix CSS specificity #30563 in the merge queue: `plan-playwright` completes past "Build duration-aware shard plans" and emits a valid `matrix.json`.
  • Any subsequent PR going through the queue is unblocked.
  • Full-suite dispatch / nightly runs succeed with no wall-clock regression.

🤖 Generated with Claude Code

Greptile Summary

Updates the Playwright shard planner and its tests:

  • Raises the Chromium shard budget from 19 to 20 minutes.
  • Updates Chromium budget and boundary assertions to match the new threshold.

Confidence Score: 5/5

The PR appears safe to merge.

The previously reported stale Chromium budget assertions now match the production 20-minute threshold, and no blocking failure remains.

Important Files Changed

Filename Overview
.github/scripts/build_playwright_shards.py Raises the Chromium shard acceptance budget to 20 minutes while leaving shard runtime prediction unchanged.
.github/scripts/tests/test_playwright_ci_planning.py Correctly updates the previously stale budget and rejection-boundary assertions from 19 to 20 minutes.

Reviews (2): Last reviewed commit: "address greptile P1: update planning tes..." | Re-trigger Greptile

The chromium lane hit the planner's ceiling on the merge queue for
PR #30563 (run 30394014577):

  Lane chromium needs more than 24 shards to stay within the 19-minute
  plan budget; the heaviest shard is predicted at 19.1m
  ##[error]Process completed with exit code 1.

Root cause is aggregate growth: the chromium suite currently holds
~1053 min of test content per the committed timing baseline — 90.6%
of the planner's 24 × 3 × 19min × 0.85 capacity ceiling (1163 min).
LPT bin-packing is doing its job; there's just no headroom left, and
the current heaviest-shard prediction is 0.1 min over the hard budget.

Bump COMMON_SHARD_BUDGET_MS from 19 min to 20 min, aligning it with
the non-chromium TARGET_MS. The change only widens the acceptance
threshold — the predicted heaviest shard runtime is unchanged, so
wall-clock is unaffected. What changes is the planner will now accept
a plan it previously refused.

This is a stop-gap. The heaviest individual tests are worth attacking
at the root:

  605s  TestCaseImportExportE2eFlow.spec.ts › EditAll User: Complete export-import-validate flow
  603s  TestCaseImportExportE2eFlow.spec.ts › Admin: Complete export-import-validate flow
  362s  ContextCenterArticles.spec.ts › Article edits and navigation-flushed titles persist

Two ~10-minute tests in a single file are effectively immovable in LPT
scheduling — they consume 20% of a shard budget between them. Splitting
or trimming those three tests would give ~15 min of aggregate breathing
room without another budget bump. Leaving that as a follow-up so the
merge queue can move again immediately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 29, 2026 05:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

Copy link
Copy Markdown
Contributor

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

  • No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

@github-actions github-actions Bot added safe to test Add this label to run secure Github workflows on PRs UI UI specific issues labels Jul 29, 2026
Comment thread .github/scripts/build_playwright_shards.py
@greptile-apps caught two test-file assertions still hardcoded to the
former 19-minute chromium budget after the constant bumped to 20:

  test_common_lane_keeps_one_minute_of_allocation_reserve
    asserts shard_budget_ms_for_lane("chromium") == 19 * 60 * 1000

  test_common_shards_enforce_the_nineteen_minute_budget
    matches SystemExit "above the 19-minute plan budget"
    uses 19-min weights for within/above units

Updates:

  - Rename first test to test_common_lane_matches_target_execution_budget
    and assert 20 min (chromium == TARGET_MS after the alignment).
    Added a comment explaining that the previous 1-minute reserve was
    removed deliberately, not accidentally, so a future reader doesn't
    revert.
  - Rename second test to test_common_shards_enforce_the_twenty_minute_budget
    and update the weight_ms constants + the SystemExit match string.

pytest verifies: 55 passed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 29, 2026 05:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chirag-madlani

Copy link
Copy Markdown
Collaborator Author

@greptile-apps good catch — addressed in a624cd3287.

Two test-file assertions were still hardcoded to the former 19-minute chromium budget:

  1. test_common_lane_keeps_one_minute_of_allocation_reserve asserted shard_budget_ms_for_lane("chromium") == 19 * 60 * 1000

    • Renamed to test_common_lane_matches_target_execution_budget and updated to assert 20 * 60 * 1000 (chromium is now aligned with TARGET_MS)
    • Added a comment explaining the 1-min reserve was removed deliberately so a future reader doesn't unwind this change
  2. test_common_shards_enforce_the_nineteen_minute_budget matched SystemExit("above the 19-minute plan budget") and used 19 * 60 * 1000 weights

    • Renamed to test_common_shards_enforce_the_twenty_minute_budget and updated match string + within/above weight_ms constants

Verified: pytest .github/scripts/tests/test_playwright_ci_planning.py55 passed locally.

@greptile-apps

greptile-apps Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Thanks for the thorough fix. Both stale assertions are updated, the test names now accurately describe what they enforce, and the explanatory comment is a nice touch to prevent future confusion. 55/55 passing confirms nothing regressed. LGTM.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Playwright Results — workflow succeeded

Validated commit a624cd32875ed59104c2fc6d0b9aad287e26dc65 in Playwright run 30425770283, attempt 1.

✅ 537 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 5 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 50m 5s

⏱️ Max setup 2m 54s · max shard execution 17m 48s · max shard-job elapsed before upload 20m 56s · reporting 4s

🌐 203.00 requests/attempt · 2.86 app boots/UI scenario · 10.67% common-shard skew

Optimization targets still in progress:

  • Browser traffic was 203 requests per attempt (convergence target: fewer than 200).
  • Application boot ratio was 2.86 per UI scenario (1597 boots / 558 scenarios; convergence target: at most 1).
Shard Passed Failed Flaky Skipped Lifecycle failed Lifecycle flaky
✅ Shard chromium-01 131 0 0 0 0 0
✅ Shard chromium-02 139 0 0 3 0 0
✅ Shard chromium-03 129 0 0 0 0 0
✅ Shard data-asset-rules-01 61 0 0 0 0 0
✅ Shard domain-isolation-01 14 0 0 0 0 0
✅ Shard global-state-01 23 0 0 0 0 0
✅ Shard ingestion-01 1 0 0 0 0 0
✅ Shard reindex-01 2 0 0 0 0 0
✅ Shard search-01 10 0 0 0 0 0
✅ Shard search-rbac-01 27 0 0 2 0 0

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

@chirag-madlani

Copy link
Copy Markdown
Collaborator Author

Will split the spec instead increase the timing: #30617

@gitar-bot

gitar-bot Bot commented Jul 29, 2026

Copy link
Copy Markdown
Code Review ✅ Approved

Raises the Chromium Playwright shard-planning budget from 19 to 20 minutes and updates the corresponding planning tests. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar | Powered by Gitar — free for open source

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test Add this label to run secure Github workflows on PRs UI UI specific issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants