Skip to content

Changed main E2E tests to run across ten shards#28079

Merged
9larsons merged 1 commit into
mainfrom
codex/e2e-main-10-shards
May 23, 2026
Merged

Changed main E2E tests to run across ten shards#28079
9larsons merged 1 commit into
mainfrom
codex/e2e-main-10-shards

Conversation

@9larsons
Copy link
Copy Markdown
Contributor

Summary

  • changes the main browser E2E matrix from 8 shards to 10 shards
  • leaves analytics E2E at 2 shards
  • keeps TEST_WORKERS_COUNT=1 unchanged

Expected impact

Historic successful CI runs show the current main E2E split is consistently bottlenecked by one or two slow shards:

Current shard Avg total Avg setup Avg test
Main 2/8 11m08s 1m56s 9m12s
Main 6/8 9m52s 2m09s 7m43s
Main 8/8 9m24s 1m58s 7m26s

Average main-shard setup is about 2m, so adding two shards adds roughly 4m of total runner setup, but wall-clock can improve if the slowest Playwright shard drops by more than the fixed setup cost on that lane.

Expected wall-clock saving: likely 1-2 minutes if Playwright's native file/test grouping spreads the current long shard better; best case is larger, but unlikely because the suite is not fullyParallel and large files remain chunky.

Success criteria

Compare this PR's E2E matrix with recent baseline runs and check:

  • max main shard duration drops from ~11-12m to ~9-10m
  • total E2E runner minutes do not grow disproportionately
  • no increase in E2E flakes or retries

If this only moves the bottleneck, the next better step is splitting or manually binning the large files instead of adding more shards.

Testing

  • not run locally; workflow matrix experiment

The current eight-shard main E2E split has a consistent long shard around eleven minutes. Trying ten main shards is a small, reversible experiment to see whether Playwright's native sharding can lower the slowest shard before we invest in manual file rebalancing.
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 23, 2026

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3fd86a01-71b7-4d74-a1f4-388424c23bc3

📥 Commits

Reviewing files that changed from the base of the PR and between 88d5d08 and 88aab93.

📒 Files selected for processing (1)
  • .github/workflows/ci.yml

Walkthrough

The CI workflow in .github/workflows/ci.yml was updated to expand E2E test parallelization for the Main Playwright project. The job_e2e_tests matrix shard configuration was modified to increase shardTotal from 8 to 10, with corresponding matrix entries now covering shards 2 through 10. This adds two new shard indices (9 and 10) to the test matrix. The Analytics project sharding configuration remains unchanged at shardTotal: 2.

Possibly related PRs

  • TryGhost/Ghost#27654: Modifies .github/workflows/ci.yml job_e2e_tests sharding matrix for Playwright Main lane, directly related to the shard enumeration changes in this PR.
  • TryGhost/Ghost#28001: Modifies the GitHub Actions E2E workflow in .github/workflows/ci.yml to gate job execution, directly impacting the same E2E test job behavior.

Suggested reviewers

  • rob-ghost
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title directly and accurately summarizes the main change: updating the main E2E test matrix from 8 shards to 10 shards.
Description check ✅ Passed The description is clearly related to the changeset, providing detailed context about the shard configuration change, expected impact, success criteria, and rationale.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/e2e-main-10-shards

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 23, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.81%. Comparing base (88d5d08) to head (88aab93).

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #28079   +/-   ##
=======================================
  Coverage   73.80%   73.81%           
=======================================
  Files        1528     1528           
  Lines      129417   129417           
  Branches    15504    15506    +2     
=======================================
+ Hits        95521    95528    +7     
+ Misses      32917    32910    -7     
  Partials      979      979           
Flag Coverage Δ
admin-tests 54.04% <ø> (+<0.01%) ⬆️
e2e-tests 73.81% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@9larsons 9larsons merged commit ac1e702 into main May 23, 2026
50 checks passed
@9larsons 9larsons deleted the codex/e2e-main-10-shards branch May 23, 2026 02:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant