Headless chrome indexer error audit results fixes by habdelra · Pull Request #3445 · cardstack/boxel

habdelra · 2025-10-20T22:52:40Z

This PR fixes indexing issues discovered during the headless chrome indexer error audit:

Indexing jobs need more lenient timeouts. When realms have no indexing errors or recoverable indexing errors the indexing is very fast, faster than our current indexer. However, when there are render errors that are not recoverable, the indexing is much much slower. in general the catalog realm needs much more time to index from a blank slate. Generally i'm seeing it take about 6 minutes to index from a blank slate. The experiments realm (2.5min), base realm (20s), and skills realm (5s) all index faster than before tho.
The last known good html that appears in the error docs is getting confused for actual DOM when we try to render the serialized form of these errors. we strip this out from our errors before rendering them in the DOM so the browser doesn't get confused (this HTML is only meant for human consumption on errors--the prerenderer doesn't need it)
the NotLoadedValue sentinel that we put in our deserialized state bucket for instances was leaking out when we try to render a card while the links are still inflight. I was able to clean this up for all teh field types except for LinksToMany. there is a deeper component stability issue here that we cant really address until the .value refactor. details have been left in the comments. however, this means that for LinksToMany, the NotLoaded sentinel will occasionally be exposed to card developers. I cleaned up a bunch of cards in our catalog realm where that happens. Codex was very very adamant that we don't try to go deeper with a fix here. it brought up lots of very good points about a super painful refactor that leaves us with lots of duplicated state management barring a .value refactor.
Our glimmer scoped styles is actually causing issues for us due to the way it manipulates the DOM out from underneath glimmer. when we reuse a page and the render route is trying to tear down the DOM, glimmer gets mad that one of the DOM nodes that its trying to remove is no longer there (the <style> tag embedded in the card). I tried a bunch of approaches to this, but none were stable. so instead we detect this error and when we see it we clear the store cache and try again--this solves the issue.

Most of the rest of the issues are actually related to downloading modules from external sites and CORS violations due to our own custom headers. I broke that out into a separate ticket that i'll followup with shortly https://linear.app/cardstack/issue/CS-9562/remove-the-need-to-use-x-boxel-building-index-header.

Beyond that there are a handful of other errors that we don't see in the current indexer that I'm still tracking down. One nice thing tho is that there are also a bunch of errors in our current indexer around rendering cycles that are actually solved in the headless chrome indexer.

- waiting for linked fields to load only _after_ /render/html templates have been rendered - more efficient render options - improve error msg settling - all node indexing test for headless chrome passing

…or-isused-field-directive-to-include-linked

…oveChild' on 'Node': The node to be removed is not a child of this node." - scoped-css needs to respect the DOM tree that glimmer is managing when it moves the style tag around. otherwise this will blow up during re-renders in the /render route.

…s when links are being loaded still.

…en we try to render the serialized error

…lms with lots of render errors.

…ll-indexing-errors-that-we-see-when-starting-up

github-actions · 2025-10-20T23:14:34Z

Host Test Results

1 files ± 0 1 suites ±0 3h 2m 23s ⏱️ + 1h 53m 5s
1 523 tests + 98 1 509 ✅ + 102 13 💤 + 1 0 ❌ - 2 1 🔥 - 3
4 397 runs +2 960 4 350 ✅ +2 935 39 💤 +27 4 ❌ - 2 4 🔥 ±0

For more details on these errors, see this check.

Results for commit f4ee460. ± Comparison against base commit 7486399.

This pull request removes 2 and adds 100 tests. Note that renamed tests count towards both.

Chrome ‑ Global error: Uncaught TypeError: Failed to fetch at http://localhost:7357/assets/chunk.8807cb9de65ae32da336.js, line 142924  After execution of test: Acceptance | Code patches tests: action bar disappears when "Cancel" button is clicked 
Chrome ‑ Global error: Uncaught TypeError: Failed to fetch at http://localhost:7357/assets/chunk.8807cb9de65ae32da336.js, line 142924  While executing test: Acceptance | Code patches tests: action bar disappears when "Cancel" button is clicked

Chrome ‑ Acceptance | Code patches tests: LLM mode event controls auto-apply of code patches with timestamp checking
Chrome ‑ Acceptance | Code patches tests: action bar disappears when "Cancel" button is clicked
Chrome ‑ Acceptance | Code patches tests: automatic Accept All spinner appears in Act mode for multiple patches
Chrome ‑ Acceptance | Code patches tests: can create new files using the search/replace block
Chrome ‑ Acceptance | Code patches tests: can restore content of a patched file to its original state
Chrome ‑ Acceptance | Code patches tests: does not display the action bar when the streaming is cancelled
Chrome ‑ Acceptance | Code patches tests: loader reset happens when restoring patched executable files
Chrome ‑ Acceptance | Code patches tests: previously applied code patches show the correct applied state
Chrome ‑ Acceptance | Code patches tests: schema editor gets refreshed when code patches are executed on executable files
Chrome ‑ Acceptance | Code patches tests: when code patch is historic (user moved on to the next message), or it was applied, it will render the code (replace portion of the search/replace block) in a standard (non-diff) editor
…

♻️ This comment has been updated with latest results.

…mer scoped styles manipulating the DOM out from under glimmer)

habdelra · 2025-10-21T14:18:32Z

@codex review

chatgpt-codex-connector · 2025-10-21T14:25:13Z

Codex Review: Didn't find any major issues. Bravo.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…eChild issue

backspace · 2025-10-21T14:56:13Z

+  // that a node it was tracking is no longer there. performing a new prerender
+  // capture with a cleared store/loader cache will workaround this issue.
+  [`Failed to execute 'removeChild' on 'Node'`, 'NotFoundError'],
+];


I’m surprised the style element is even present, I thought the AST transform would remove it from the component altogether

agreed, I wonder if it’s the AST removing it that is what glimmer is mad about. i did confirm tho if I change the AST to leave the style tag in the card DOM it error goes away.

Maybe there’s a way to have the transform happen earlier in the chain, before Glimmer has a hold on any elements at all. The way it runs on boxel-ui, for instance, is static. But I’m definitely out of my depth here.

…ll-indexing-errors-that-we-see-when-starting-up

habdelra added 11 commits October 16, 2025 22:03

Refactor /render route model loading

8b18530

- waiting for linked fields to load only _after_ /render/html templates have been rendered - more efficient render options - improve error msg settling - all node indexing test for headless chrome passing

fixed typo in data attribute access

911b8ba

cleanup

24bac12

Support for isUsed in /render route

d9b0e5f

Merge remote-tracking branch 'origin/main' into cs-9539-add-support-f…

ed33ddb

…or-isused-field-directive-to-include-linked

better handling of NotLoadedValue sentinel during renders of instance…

c74f38d

…s when links are being loaded still.

Strip last known good html from errors as this can confuse the DOM wh…

01cff93

…en we try to render the serialized error

relax timeout for indexing. prerendering can take much longer for rea…

73c6cb2

…lms with lots of render errors.

louder timeout log messages

74751c4

Merge remote-tracking branch 'origin/main' into cs-9500-investigate-a…

b825384

…ll-indexing-errors-that-we-see-when-starting-up

habdelra marked this pull request as draft October 20, 2025 23:15

habdelra added 5 commits October 20, 2025 20:49

Added temp patch for glimmer-scoped-css until PR lands

c48c8ac

stripping out style stubs and more forgiving test waiting

cf57b0a

remove debug code

e5819c4

back out scoped-glimmer-approach

7441a84

retry render when we encounter removeChild error (symptomatic of glim…

bda6215

…mer scoped styles manipulating the DOM out from under glimmer)

added comment to explain why we attempt a 2nd prerender for the remov…

447bd01

…eChild issue

habdelra marked this pull request as ready for review October 21, 2025 14:30

habdelra requested a review from a team October 21, 2025 14:31

backspace approved these changes Oct 21, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into cs-9500-investigate-a…

f4ee460

…ll-indexing-errors-that-we-see-when-starting-up

habdelra merged commit 9d4e0d4 into main Oct 22, 2025
110 of 116 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Headless chrome indexer error audit results fixes#3445

Headless chrome indexer error audit results fixes#3445
habdelra merged 18 commits intomainfrom
cs-9500-investigate-all-indexing-errors-that-we-see-when-starting-up

habdelra commented Oct 20, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 20, 2025 •

edited

Loading

Uh oh!

habdelra commented Oct 21, 2025

Uh oh!

chatgpt-codex-connector bot commented Oct 21, 2025

Uh oh!

backspace Oct 21, 2025

Uh oh!

habdelra Oct 21, 2025 •

edited

Loading

Uh oh!

backspace Oct 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

habdelra commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Host Test Results

Uh oh!

habdelra commented Oct 21, 2025

Uh oh!

chatgpt-codex-connector bot commented Oct 21, 2025

Uh oh!

backspace Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

habdelra Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

backspace Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

habdelra commented Oct 20, 2025 •

edited

Loading

github-actions bot commented Oct 20, 2025 •

edited

Loading

habdelra Oct 21, 2025 •

edited

Loading