Skip to content

Disabled v8-compile-cache to fix intermittent V8 deserializer crashes#26652

Merged
cmraible merged 1 commit intoTryGhost:mainfrom
cmraible-bot:fix/disable-v8-compile-cache-in-ci
Mar 12, 2026
Merged

Disabled v8-compile-cache to fix intermittent V8 deserializer crashes#26652
cmraible merged 1 commit intoTryGhost:mainfrom
cmraible-bot:fix/disable-v8-compile-cache-in-ci

Conversation

@cmraible-bot
Copy link
Copy Markdown

@cmraible-bot cmraible-bot commented Mar 2, 2026

Problem

CI intermittently fails with a fatal V8 crash during Build TS packages:

# Fatal error in , line 0
# unreachable code
v8::internal::Deserializer<v8::internal::Isolate>::ReadSingleBytecodeData
...
v8::internal::CodeSerializer::Deserialize
v8::internal::Compiler::GetSharedFunctionInfoForScriptWithCachedData

Example failing run: https://github.com/TryGhost/Ghost/actions/runs/22564490904/job/65358043431

Root cause

The dependency chain ts-nodev8-compile-cache-lib is responsible. When ts-node loads the TypeScript compiler it calls v8-compile-cache-lib.install(), which caches compiled V8 bytecode to disk in /tmp.

With Nx running 4 parallel worker processes (the default), multiple workers call this simultaneously and race to read/write the same blob cache files. This corrupts the bytecode, causing the crash on deserialization.

This is a known Node.js issue: nodejs/node#51555

The Node.js core team confirmed: "Incorrect use of the v8 code cache can lead to corruption and crashes like this... This might be especially true if you have concurrent processes or threads which need careful synchronization."

Fix

Set DISABLE_V8_COMPILE_CACHE=1 globally in the CI workflow environment. This is the confirmed workaround from the Node.js issue thread.

Performance impact

None. The bytecode cache is written to /tmp and discarded at the end of each run — it never persists between CI jobs. Every run was already starting cold.

Test plan

  • Verify CI passes without the intermittent V8 crash on a few runs

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 2, 2026

Walkthrough

The pull request sets DISABLE_V8_COMPILE_CACHE=1 in the env section of the CI workflow (.github/workflows/ci.yml) and adds comments explaining this disables the v8-compile-cache to avoid intermittent V8 deserializer crashes when parallel Nx workers race on shared bytecode cache files (cache location: /tmp, references Node.js issue 51555). No workflow logic or control flow was changed. Lines changed: +6 / -0.

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title directly and clearly describes the main change: disabling v8-compile-cache to fix intermittent V8 deserializer crashes, which matches the core purpose of the changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description check ✅ Passed The pull request description is directly related to the changeset, explaining the problem, root cause, fix, and performance impact of disabling v8-compile-cache in CI.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@cmraible cmraible self-requested a review March 2, 2026 20:04
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 2, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.20%. Comparing base (e7505d9) to head (7798580).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #26652   +/-   ##
=======================================
  Coverage   73.20%   73.20%           
=======================================
  Files        1534     1534           
  Lines      121034   121034           
  Branches    14635    14635           
=======================================
+ Hits        88598    88599    +1     
  Misses      31405    31405           
+ Partials     1031     1030    -1     
Flag Coverage Δ
admin-tests 54.32% <ø> (+<0.01%) ⬆️
e2e-tests 73.20% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cmraible cmraible changed the title ci: Disabled v8-compile-cache to fix intermittent V8 deserializer crashes Disabled v8-compile-cache to fix intermittent V8 deserializer crashes Mar 2, 2026
Parallel Nx workers each invoke ts-node to load the TypeScript compiler,
which calls v8-compile-cache-lib.install() and races to read/write shared
bytecode blob files in /tmp. This causes intermittent fatal crashes:

  # Fatal error in , line 0
  # unreachable code
  v8::internal::Deserializer::ReadSingleBytecodeData

Setting DISABLE_V8_COMPILE_CACHE=1 eliminates the race. There is no
performance cost since the cache lives in /tmp and is discarded at the
end of each CI run anyway — every run already starts cold.

See: nodejs/node#51555

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@cmraible cmraible force-pushed the fix/disable-v8-compile-cache-in-ci branch from 1a4c084 to 7798580 Compare March 11, 2026 16:54
@cmraible cmraible requested a review from troyciesco March 11, 2026 19:33
Copy link
Copy Markdown
Contributor

@troyciesco troyciesco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, link to the node issue was extremely helpful.

question/suggestion: does this happen on just browser tests, like the one linked in the description, or any job? (I don't think I've ever run into this before). if it is just on browser tests it might be worth running that once here.

@cmraible
Copy link
Copy Markdown
Collaborator

does this happen on just browser tests, like the one linked in the description, or any job?

It could happen on any job that builds all the typescript packages - browser tests, acceptance tests, legacy tests, etc, so it's run successfully at least a couple times on this PR 😄

@cmraible cmraible merged commit 569df15 into TryGhost:main Mar 12, 2026
62 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants