Skip to content

Expanded ghost/core unit test timeouts and retries#28067

Merged
9larsons merged 2 commits into
mainfrom
fix/ghost-core-unit-test-flakiness
May 22, 2026
Merged

Expanded ghost/core unit test timeouts and retries#28067
9larsons merged 2 commits into
mainfrom
fix/ghost-core-unit-test-flakiness

Conversation

@9larsons
Copy link
Copy Markdown
Contributor

@9larsons 9larsons commented May 22, 2026

ghost/core's unit tests fail intermittently on main CI. Two distinct things were found:

  • The previously-skipped cache-invalidation test was a genuine timeout. testTimeout was 2000ms — a mocha holdover; under vitest's per-file isolation the cold-import cost makes that too tight on a loaded runner.
  • The remaining failures are not clean timeouts — vitest's github-actions reporter emits no test-level annotation, so the ghost/core vitest process is exiting abnormally (a worker crash), not failing a reported test. It could not be reproduced locally (12/12 clean); it is specific to loaded CI runners.

This is interim stabilization, not the root fix:

  • testTimeout 2000ms → 5000ms — clears the timeout-flake class with no loss of coverage.
  • retry: 2 in vitest config — retries a failed test up to twice (3 attempts), absorbing transient test-level failures.
  • CI step retry (up to 3 attempts) on the "Run unit tests" step — masks the intermittent worker crash; nx caching means a retry only re-runs the crashed project.
  • CI console reporter dotdefaultdot emits only dots, so a crash leaves no clue which file was running; default names each file. github-actions annotations are kept for clean per-failure reporting.

The worker-crash root cause is expected to be addressed by the upcoming isolate: false infrastructure rework, which reworks the same vitest worker/teardown code.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 22, 2026

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

This PR raises Vitest's test.testTimeout from 2000ms to 5000ms, updates inline comments about CI cold-import/isolate behavior, and changes CI reporter selection to ['default','github-actions'] (local runs remain 'dot'). Separately, the GitHub Actions job_unit-tests step is replaced with a Bash retry loop that runs pnpm nx run-many -t test:unit up to three times, warning on each failure and exiting on first success.

Possibly related PRs

  • TryGhost/Ghost#27898: Introduces the Vitest configuration file that this PR modifies (timeouts and reporter selection).
  • TryGhost/Ghost#28030: Also changes Vitest unit-test configuration (test discovery/include behavior), affecting the same config surface.
  • TryGhost/Ghost#28049: Addresses an intermittent test that times out under prior timeout settings; related to the timeout changes in this PR.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately reflects the two main changes: increased unit test timeouts and added retry logic in ghost/core configuration.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The pull request description accurately and comprehensively explains the changes made to address intermittent unit test failures, covering both the specific file modifications and the underlying rationale.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/ghost-core-unit-test-flakiness

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@9larsons 9larsons changed the title Bumped ghost/core vitest testTimeout to 5000ms Expanded ghost/core unit test timeouts and retries May 22, 2026
@9larsons 9larsons force-pushed the fix/ghost-core-unit-test-flakiness branch from 57b5c6f to c305943 Compare May 22, 2026 17:26
@9larsons 9larsons force-pushed the fix/ghost-core-unit-test-flakiness branch from c305943 to 560a019 Compare May 22, 2026 17:29
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci.yml:
- Around line 394-399: The retry loop prints "Unit tests attempt X failed —
retrying" even on the final (third) iteration; update the in-loop echo so it
distinguishes the last failure: inside the for loop that runs pnpm nx run-many
-t test:unit ..., check the attempt number (e.g., if attempt is less than 3) and
print "Unit tests attempt ${attempt} failed — retrying", otherwise print "Unit
tests attempt ${attempt} failed — no more retries" (or similar final-failure
wording) so logs clearly show when no further retries will occur.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 808b8af6-7997-4a4d-84e8-e08434ac53c7

📥 Commits

Reviewing files that changed from the base of the PR and between ebb9c3a and c305943.

📒 Files selected for processing (2)
  • .github/workflows/ci.yml
  • ghost/core/vitest.config.ts

Comment thread .github/workflows/ci.yml
- the CI retry loop logged "attempt N failed — retrying" on every
  iteration, including the third, where no retry follows
- the final attempt now logs a distinct "no more retries" error so the
  log unambiguously shows when the unit tests have given up
@codecov
Copy link
Copy Markdown

codecov Bot commented May 22, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.82%. Comparing base (fcad5b8) to head (bd6f8e1).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #28067   +/-   ##
=======================================
  Coverage   73.81%   73.82%           
=======================================
  Files        1528     1528           
  Lines      129417   129417           
  Branches    15506    15509    +3     
=======================================
+ Hits        95527    95537   +10     
+ Misses      32910    32900   -10     
  Partials      980      980           
Flag Coverage Δ
admin-tests 54.03% <ø> (ø)
e2e-tests 73.82% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@9larsons 9larsons enabled auto-merge (squash) May 22, 2026 17:56
@9larsons 9larsons merged commit 532e0ab into main May 22, 2026
47 checks passed
@9larsons 9larsons deleted the fix/ghost-core-unit-test-flakiness branch May 22, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant