CS-11123: Unwind cacheOnlyDefinitions workaround

[habdelra]

Summary

Cleanup PR. With pre-warm (#4799) populating the modules table before every render fires, lookupDefinition from inside populateQueryFields always hits a populated DB row instead of spawning a nested prerender. The cacheOnlyDefinitions:true short-circuit and the __boxelDuringPrerender header plumbing are no longer load-bearing in the indexing flow — remove them.

This is not a performance PR — the goal is to drop the workaround scaffolding now that the root-cause fix (pre-warm) is in. No measurable wall-clock change expected.

Removed pieces:

• DURING_PRERENDER_HEADER constant + isDuringPrerenderRequest helper in runtime-common/realm.ts; matching re-export in realm-server/prerender/prerender-constants.ts; import in realm-server/handlers/handle-search.ts.
• Host SPA's duringPrerenderHeaders() helper and both _federated-search fetch call sites in host/app/services/store.ts.
• Prerender page-pool's #markPageAsInPrerender (which injected globalThis.__boxelDuringPrerender = true via evaluateOnNewDocument) and its two call sites.
• handle-search.ts's cacheOnlyDefinitions = ctxt.get(...).length > 0 branch; the search handler now just calls searchRealms(realms, query).
• Realm.search's opts?: { cacheOnlyDefinitions?: boolean } parameter and the conditional spread on searchCards opts; searchResponse no longer threads cacheOnlyDefinitions through.
• searchRealms / SearchableRealm.search lose the opts parameter.
• RealmIndexQueryEngine.Options.cacheOnlyDefinitions and its three consumers (fieldExpectsFileMeta cache-only branch, lookupDefinitionForOpts cache-only branch, populateQueryFieldsFromMeta short-circuit at the loadLinks layer).
• lookupDefinitionForOpts collapses to a direct lookupDefinition call — both call sites are inlined.
• populateQueryFieldsFromMeta is removed entirely; the loadLinks walker now always calls the live populateQueryFields.

Intentionally left alone: QueryFieldMeta and the indexer's queryFieldDefs emission in host/app/utils/file-def-attributes-extractor.ts. They're still written into resource.meta and consumed elsewhere as metadata; only the cache-only-read short-circuit they served is gone.

Net diff: -169 lines across host, realm-server, runtime-common.

Built directly on top of #4799. Re-target this PR's base to main before merging.

Test plan

• Verify no remaining references to DURING_PRERENDER_HEADER, duringPrerenderHeaders, __boxelDuringPrerender, markPageAsInPrerender, cacheOnlyDefinitions, populateQueryFieldsFromMeta, or lookupDefinitionForOpts in the source tree (excluding the index-runner's Why: comment about the historical deadlock).
• Realm-server search integration tests pass; verify no test relied on the cacheOnlyDefinitions flag implicitly.
• Spot-check a reindex run does not regress on the realms covered by #4799's bench — if the safety net was load-bearing somewhere pre-warm doesn't cover, the cleanup needs to be deferred or scoped down.

🤖 Generated with Claude Code

[github-actions]

Preview deployments

• Host staging preview

• Host production preview

Host Test Results

    1 files  ±0      1 suites  ±0   2h 2m 15s ⏱️ + 16m 57s
2 658 tests ±0  2 643 ✅ ±0  15 💤 ±0  0 ❌ ±0 
2 677 runs  ±0  2 662 ✅ ±0  15 💤 ±0  0 ❌ ±0 

Results for commit 550d864. ± Comparison against earlier commit a05a3ee.

Realm Server Test Results

    1 files  ±0      1 suites  ±0   12m 8s ⏱️ +4s
1 345 tests ±0  1 345 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 424 runs  ±0  1 424 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 177a0f2. ± Comparison against earlier commit 550d864.

[Copilot]

Pull request overview

This PR removes the now-obsolete “cache-only definitions during prerender” workaround and associated prerender-detection plumbing, based on the assumption that the modules table is pre-warmed ahead of renders so lookupDefinition no longer needs to trigger nested prerenders during indexing/search.

Changes:

• Removes cacheOnlyDefinitions support end-to-end (API surface, options threading, and the meta-based short-circuit path) so searches always use the normal definition lookup path.
• Deletes the prerender marker/header mechanism (DURING_PRERENDER_HEADER, host-side header injection, and PagePool’s evaluateOnNewDocument global injection).
• Simplifies realm-server federated search handling to always call searchRealms(realms, query) with no prerender-specific behavior.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
packages/runtime-common/search-utils.ts	Removes opts parameter from SearchableRealm.search and searchRealms().
packages/runtime-common/realm.ts	Drops DURING_PRERENDER_HEADER + prerender detection; simplifies Realm.search() signature and usage.
packages/runtime-common/realm-index-query-engine.ts	Removes cacheOnlyDefinitions option and all cache-only/meta short-circuit branches; always uses live lookupDefinition.
packages/runtime-common/index-runner.ts	Updates explanatory comment describing why pre-warm exists (removes cache-only reference).
packages/realm-server/prerender/prerender-constants.ts	Removes re-export of the prerender marker header constant.
packages/realm-server/prerender/page-pool.ts	Removes prerender tab global injection (#markPageAsInPrerender) and its call sites.
packages/realm-server/handlers/handle-search.ts	Removes prerender-header handling; always federates search without cache-only mode.
packages/host/app/services/store.ts	Removes prerender header injection on _federated-search requests and the related helper/import.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a05a3eed61

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 177a0f27ef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Keep cache-only path for un-prewarmed search results

When a prerendered card’s query field returns resources outside the current invalidation batch (for example a search over existing authors/tags), loadLinks now always calls live populateQueryFields for those returned resources. Fresh evidence beyond the earlier proxy concern is that this layer walks arbitrary search result resources, while IndexRunner.preWarmModulesTable only warms modules derived from the invalidation URLs/deps; if one of these returned resources has an uncached definition, this call can still enter lookupDefinition and spawn a nested prerender while the original prerender tab is held. Please preserve a cache-only/meta path for resources that prewarm did not cover.

Useful? React with 👍 / 👎.

[habdelra]

[Claude Code] The reachability claim is consistent with empirical bench data I collected before the PR was reframed as cleanup:

Realm	SHA	Wall (s)	filesIndexed	instancesIndexed	instanceErrors
user/prudent-octopus	pre-warm (#4799 c85633d237)	467	118	91	0
user/prudent-octopus	+ unwind (this PR's prior SHA 550d86490d)	550	118	90	1

Same realm, same protocol, same rig — the unwind reintroduced a cohort-instance error that the pre-warm PR's adoptsFrom fix had resolved on octopus. loadLinks's walk over query-field-returned resources is the obvious candidate path: pre-warm only covers the invalidation batch's deps closure, so a returned existing card with an uncached definition can still hit the live populateQueryFields → lookupDefinition → prerenderModule path while the original tab is held. The 256 fed-search hits on the unwind run (vs 176 with the safety net) is consistent — every render now walks the live path instead of degrading via populateQueryFieldsFromMeta.

Hassan's reply on the earlier proxy-renders concern still applies (a single-URL render has no batch deps closure to walk, so single-URL paths can't easily piggyback on pre-warm). The same shape applies here.

Tagging Hassan (@habdelra) to decide whether to keep just the parts of the unwind that don't surface this regression (drop the populateQueryFieldsFromMeta removal at line ~1104 + the Options.cacheOnlyDefinitions consumers, leave the host/page-pool/header plumbing removal in place), defer the unwind entirely, or accept the regression on these realms.

[habdelra]

Closing this PR. The audit of which parts of the unwind are actually safe-to-remove vs. load-bearing came back as: the safety net is fully connected end-to-end (page-pool injection → host header → realm-server inbound branch → searchCards short-circuit → populateQueryFieldsFromMeta). Removing any link breaks the chain, and bench data confirms removing it surfaces a real correctness regression on prudent-octopus (90 instances / 1 error vs pre-warm-only's clean 91 / 0).

The truly-cosmetic parts that could survive (DURING_PRERENDER_HEADER re-export in prerender-constants.ts, comment in index-runner.ts referencing the workaround) add up to ~20 lines — not worth a separate cleanup PR.

The honest conclusion: until pre-warm's closure is extended to cover what loadLinks walks through query-field-returned cards (not just the invalidation batch's deps), cacheOnlyDefinitions stays load-bearing. Filing a follow-up to revisit once that closure exists, or once cohort-render-specific work makes this exposure unreachable.

Ship only #4799 (pre-warm) from this stack.