realm-server: HTTPS+HTTP/2 in local dev#4797
Draft
habdelra wants to merge 22 commits into
Draft
Conversation
Heavy aggregator-card renders (cohort, dashboards) fan out 80+
federated-search requests per render inside one Chromium tab. Chrome's
HTTP/1.1 6-per-origin connection ceiling serializes them and turns a
single render into multiple minutes; HTTP/2 multiplexes them over one
connection and the same render finishes in seconds. Browsers only do
HTTP/2 over TLS, so the local realm-server now terminates a cert.
Single-origin design: the realm-server listens on
`https://localhost:4201` (and `https://localhost:4202` for test-realms)
when the dev cert is provisioned. There is no parallel HTTP listener
and no h2 alias port; the wire protocol and the canonical realm URL
agree. In-process tests and any environment without a cert keep getting
plain HTTP/1.1 via the same `listen(port)` entry point — `RealmServer`
picks the protocol from `REALM_SERVER_TLS_CERT_FILE`/`_KEY_FILE` rather
than two separate methods.
Cert provisioning is opt-in via `mise run infra:ensure-dev-cert`:
- Requires `mkcert` (single-origin HTTPS has no HTTP fallback in
dev, so a missing prereq is a hard error with install hints).
- Attempts `mkcert -install` once for system trust; declining the
sudo prompt is non-fatal — the cert still gets generated and
indexing keeps working via puppeteer's `--ignore-certificate-errors`
flag and `NODE_EXTRA_CA_CERTS` for Node clients.
- Idempotent: re-runs are a no-op until the cert is within 7 days of
expiry.
`env-vars.sh` flips `REALM_BASE_URL`/`REALM_TEST_URL` defaults to
`https://localhost:4201`/`4202`, exports the cert paths when files
exist, and points `NODE_EXTRA_CA_CERTS` at mkcert's root CA so Node-
side fetches (worker, scripts, prerender Node) trust the cert without
requiring `mkcert -install` to have run. `dev-common.sh` switches
wait-on's readiness probes to `https-get://` when the realm URL is
HTTPS. The host's `config/environment.js` defaults flip to
`https://localhost:4201` for `realmServerURL`, `baseRealmURL`,
`catalogRealmURL`, `legacyCatalogRealmURL`, `skillsRealmURL`, and
`openRouterRealmURL`. `middleware/index.ts#fullRequestURL` now detects
`ctx.req.socket.encrypted` so URL-keyed realm lookup matches the wire
protocol — combined with the canonical-URL flip, both halves agree.
CI / hermetic test harness path stays HTTP-only: if no cert is
provisioned, `env-vars.sh` leaves the TLS env vars unset and the
realm-server boots `http.createServer`, exactly as before.
Migration after pulling: any local card data created under the old
`http://localhost:4201/...` canonical references is stale and needs to
be re-indexed. README documents the one-time `mise run
infra:full-reset` step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1655a6f2df
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
Preview deploymentsRealm Server Test Results 1 files ± 0 1 suites +1 49m 6s ⏱️ + 49m 6s Results for commit 03b5a55. ± Comparison against earlier commit ec75fea. For more details on these errors, see this check. |
Adds the two missing pieces from the initial HTTPS+HTTP/2 flip: 1. Same-port HTTP→HTTPS dispatcher in `server.ts`. When the realm-server speaks TLS, `listen(port)` now binds a net.Server that peeks the first byte off every connection: 0x16 (TLS ClientHello) routes to the http2 secure server; anything else is treated as plain HTTP and handed to a tiny 301-redirect handler that rewrites the URL to `https://<inbound-host><path>`. So `http://localhost:4201/…` in a browser bar or a `curl` invocation gets a clean 301 instead of a TLS handshake failure. Same listener, no extra port. 2. A node-pg-migrate that rewrites every URL-bearing text/varchar/jsonb column on every public table (except `modules`, which the realm-server truncates on startup) from `http://localhost:42XX` to `https://localhost:42XX`. Auto-discovered via `information_schema.columns` — covers `boxel_index`, `boxel_index_working`, `realm_registry`, `realm_meta`, `realm_metadata`, `realm_user_permissions`, `realm_versions`, `realm_file_meta`, `module_transpile_cache`, plus any future URL-bearing column that's added later (the discovery picks it up). WHERE-filtered so it only touches rows still containing the old URL — idempotent, no-op in production. `mise run dev` already passes `--migrateDB` to the realm-server, so the migration runs automatically on the first post-pull boot. README's "Local HTTPS dev access" section is rewritten to describe the new auto-migration flow (no more `mise run infra:full-reset` callout). Schema file renamed from `1779100257123_schema.sql` to `1779200000000_schema.sql` so host/config/environment.js's migration-vs-schema-name sentinel matches the new latest migration. Content is unchanged (the new migration is data-only). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8 tasks
CI was failing across host/realm-server/matrix test suites because ensure-dev-cert exited non-zero when mkcert was missing, killing the mise dep chain before any service started, and because env-vars.sh flipped REALM_BASE_URL to https unconditionally — so even when the realm-server fell back to plain HTTP, every consumer was still asked to fetch against https. The host config defaults had the same problem: hardcoded https meant the in-browser realmServerURL didn't match the wire scheme. Three fixes, gated on cert presence: 1. `ensure-dev-cert` now exits 0 with a soft warning when mkcert is missing. The realm-server's `listen()` already falls back to plain `http.createServer` when the TLS env vars are unset, so this is the honest behavior for CI / hermetic-test environments. 2. `env-vars.sh` defaults `REALM_BASE_URL`/`REALM_TEST_URL` to http and only upgrades them to https inside the cert-detected block alongside the existing TLS env var exports. 3. `packages/host/config/environment.js` derives its scheme from `process.env.REALM_BASE_URL`, so the host config follows the same cert-presence-driven flip rather than baking https into the JS defaults. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Enables local-dev realm-server to serve a single canonical HTTPS origin with HTTP/2 (plus same-port HTTP→HTTPS redirect) to remove Chrome’s HTTP/1.1 per-origin connection bottleneck during heavy prerender/search fan-outs, and migrates local indexed data from http://localhost:42xx to https://localhost:42xx.
Changes:
- Add TLS-capable listener that multiplexes HTTPS/HTTP2 and HTTP redirect on the same port; update URL construction to recognize TLS sockets.
- Default local dev URLs/config/docs to
https://localhost:4201(+:4202for test realms) and add mkcert-based cert provisioning. - Add a Postgres migration to rewrite persisted localhost canonical URLs from http→https.
Reviewed changes
Copilot reviewed 45 out of 46 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Document local HTTPS/HTTP2 setup, migration, and updated local URLs. |
| QUICKSTART.md | Update quickstart URLs to https://localhost:4201. |
| packages/realm-server/tests/types-endpoint-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/server-endpoints/search-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/server-endpoints/search-prerendered-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/server-endpoints/info-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/server-endpoints/index-responses-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/server-endpoints/helpers.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/server-endpoints/federated-types-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/server-endpoints/authentication-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/request-forward-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/realm-endpoints/user-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/realm-endpoints/reindex-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/realm-endpoints/markdown-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/realm-endpoints/info-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/realm-endpoints/dependencies-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/realm-endpoints-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/publish-unpublish-realm-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/prerender-manager-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/openrouter-passthrough-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/module-cache-race-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/helpers/index.ts | Update close helpers/types to tolerate non-http.Server server handles. |
| packages/realm-server/tests/get-boxel-claimed-domain-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/file-watcher-events-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/delete-boxel-claimed-domain-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/claim-boxel-domain-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/card-source-endpoints-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/card-endpoints-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/card-dependencies-endpoint-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/boxel-domain-availability-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/tests/atomic-endpoints-test.ts | Use RealmHttpServer type alias for server handle. |
| packages/realm-server/server.ts | Add TLS/http2+redirect dispatcher and export RealmHttpServer type; update listen logging. |
| packages/realm-server/prerender/browser-manager.ts | Add --ignore-certificate-errors for prerender Chromium when using https. |
| packages/realm-server/middleware/index.ts | Treat TLS sockets as https for fullRequestURL() computation. |
| packages/realm-server/main.ts | Make shutdown tolerant of non-http.Server handles lacking closeAllConnections(). |
| packages/realm-server/lib/dev-service-registry.ts | Broaden registry typing to net.Server. |
| packages/postgres/migrations/1779200000000_canonical-url-http-to-https.js | Add migration to rewrite localhost canonical URLs from http→https. |
| packages/host/config/schema/1779200000000_schema.sql | Add regenerated host sqlite schema snapshot. |
| packages/host/config/environment.js | Flip local default realm URLs to https. |
| mise-tasks/services/test-realms | Ensure dev cert task runs before test realms. |
| mise-tasks/services/realm-server-base | Ensure dev cert task runs before base realm server. |
| mise-tasks/services/realm-server | Ensure dev cert task runs before realm server. |
| mise-tasks/lib/env-vars.sh | Flip default realm URLs to https and export TLS cert/CA env vars. |
| mise-tasks/lib/dev-common.sh | Use https readiness probes when realm URLs are https. |
| mise-tasks/infra/ensure-dev-cert | New task to provision mkcert leaf cert for local HTTPS/HTTP2. |
| .claude/skills/indexing-diagnostics/SKILL.md | Update localhost URLs and markdown formatting in diagnostics skill doc. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Local realm-server speaks HTTPS+HTTP/2 in every environment — there is no HTTP fallback or opt-in. The dev cert is a hard prereq: - `ensure-dev-cert` exits non-zero when mkcert is missing. - `env-vars.sh` defaults `REALM_BASE_URL`/`REALM_TEST_URL` to https unconditionally and no longer flips schemes based on cert presence. - `host/config/environment.js` defaults to `https://localhost:4201` unconditionally; the previous scheme-from-env-var branch is gone. - The new `.github/actions/init` step installs mkcert via apt and runs `mise run infra:ensure-dev-cert` before any downstream job, so CI realm-servers boot HTTPS+HTTP/2 too. Test harnesses that launch Chromium already pass `--ignore-certificate-errors`; Node clients pick up the cert via `NODE_EXTRA_CA_CERTS`. - README's CI/harness paragraph is rewritten to describe the cert provisioning in the init action (no more "boots HTTP/1.1 in CI" line). Carries over the Copilot-flagged fixes: - Migration renamed to `1779100257124_canonical-url-http-to-https.js` (one greater than the existing latest, no 6+ consecutive zeros so it passes `lint:migrations`) and the matching schema dump renamed. - Migration body adds a `realm_registry` LIKE pre-check that short- circuits the full-column scans on production/staging databases where the canonical URLs never reference localhost. - Drops the unused `/* eslint-disable camelcase */` line that `lint:js` flagged. - `redirectToHttps()` parses the inbound `Host` via `new URL()` so bracketed IPv6 authorities (`[::1]:4201`) round-trip cleanly instead of the regex producing an invalid `https://::1:4201/...`. - `env-vars.sh` no longer concatenates `NODE_EXTRA_CA_CERTS` with `:` separators — Node accepts a single PEM path, not a list. If the dev already has it set, leave it alone; otherwise point at mkcert's CA. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per Copilot 3230386975 — the previous QUICKSTART pointed users at https://localhost:4201 without telling them how to provision the cert that makes that origin work. Adds mkcert to the system dependencies list at step 1 with platform-specific install hints and the `mise run infra:ensure-dev-cert` one-liner, linking back to the README's "Local HTTPS dev access" section for the full story. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three task scripts under `mise-tasks/test-services/` were stuck on the
old `http-get://${REALM_BASE_URL#http://}/base/...` readiness probe
shape that strips a hardcoded `http://`. After env-vars.sh flipped
REALM_BASE_URL to https, that strip becomes a no-op and the probe URL
turns into the malformed `http-get://https://localhost:4201/...`,
which wait-on can't reach — every CI suite that drives `mise run
test-services:*` would hang on phase-1 readiness instead of starting
the next phase.
Same fix as `mise-tasks/lib/dev-common.sh`: detect the scheme from
`$REALM_BASE_URL` / `$REALM_TEST_URL` and pick `http-get://` or
`https-get://` accordingly; strip `*://` to leave just the authority.
Also wires `infra:ensure-dev-cert` into each script's depends list so
local invocations of `mise run test-services:*` (outside CI's init
action) provision the cert before the realm-server starts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Blockers (B1–B3):
- tests/index.ts deletes REALM_SERVER_TLS_CERT_FILE/_KEY_FILE before any
fixture realm-server is spun up; without this CI's globally-provisioned
cert leaks into supertest-driven in-process servers, the dispatcher
binds TLS on 127.0.0.1:444X, and the plain-HTTP-from-supertest path is
301-redirected, breaking every assertion that expects 200/4xx.
- realm-server/package.json `test:wait-for-servers` now uses
`https-get://` to match the new wire scheme; the previous `http-get://`
hit the dispatcher's 301 path and never reported ready.
- server.ts attaches a per-socket `error` handler before the readable
callback so an RST mid-handshake (or any peer-side socket error)
doesn't escalate to an uncaught exception — dispatcher is the only
inbound listener for the realm-server, can't be allowed to crash.
- `null` reads on the dispatcher socket now `destroy()` instead of just
resuming so half-open accumulators (port scanners, eager load
balancers) don't tie up file descriptors.
Major (M1, M3–M5):
- README's auto-migration callout pointed at the wrong migration filename
(1779200000000_… → 1779100257124_…).
- pg-adapter.ts env-mode regex now matches `^https?://localhost:42XX/`
so the post-flip https canonicals get rewritten to Traefik hostnames
when a dev switches the same DB into BOXEL_ENVIRONMENT mode.
- server.ts's serveIndex / serveFromRealm URL constructions now go
through `fullRequestURL(ctxt)` instead of `${ctxt.protocol}//${ctxt.host}`;
`ctxt.protocol` only honors x-forwarded-proto when `app.proxy = true`,
while `fullRequestURL` also reads the TLS socket flag. Pre-existing
inconsistency that the https flip would have made load-bearing.
- migration's information_schema walk excludes `is_generated = 'NEVER'`
so a future generated column on any public table doesn't abort the DO
block with "column can only be updated to DEFAULT".
Copilot's second pass:
- ensure-dev-cert checks for mkcert BEFORE the idempotent-skip — env-vars.sh
needs `mkcert -CAROOT` to populate NODE_EXTRA_CA_CERTS even when an
old cert already exists, and the previous ordering let a stale cert
slip past with the trust path half-wired.
- middleware/index.ts `fullRequestURL` falls back to `:authority` when
`headers.host` is absent — HTTP/2's compat layer normally populates
host from :authority but the pseudo-header is the canonical source.
- middleware/index.ts `fetchRequestFromContext` strips `:`-prefixed
pseudo-headers (`:method`, `:scheme`, `:path`, `:authority`) before
feeding them into `new Request(headers)`, which WHATWG Headers rejects.
- QUICKSTART mkcert bullet's continuation line is properly indented now
so markdown renders it inside the bullet instead of as a new paragraph.
- indexing-diagnostics SKILL.md two table rows now have the missing third
cell so the table renders correctly.
Minor (m2, m6, n3) + Option A:
- redirectToHttps falls back to `socket.localAddress:localPort` when the
Host header is absent (HTTP/1.0 client), instead of bare `localhost`
that would route to port 443.
- scripts/full-reindex.sh and register-bot.sh flip to `https://` with
`-k` (curl doesn't pick up NODE_EXTRA_CA_CERTS, and the local mkcert
CA isn't necessarily in the system trust store).
- prerender/browser-manager.ts comment references only REALM_BASE_URL
(REALM_SERVER_DOMAIN was stale — never exported by env-vars.sh).
- QUICKSTART step 10/11 and README's "view a realm's app" paragraph
redirect manual-browser navigation to `http://localhost:4200/` (the
vite host), with a note that visiting `https://localhost:4201` directly
surfaces mixed-content warnings because vite + icons + synapse still
speak http. Realm-server's https origin is reached only via fetches
inside the vite-served page, which is where the federated-search h2
win lands. README's "view example" output also flipped the realm log
line to `https://localhost:4202/test/` to match the new canonical.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- README list item 3's wrapped continuation line is now indented under the bullet so markdown doesn't break it into a separate paragraph. - server.ts dispatcher tracks every accepted socket in a Set and mirrors http.Server's `closeAllConnections()` API. main.ts's existing typeof feature-detect picks this up; shutdown no longer hangs on long-lived h2 sessions or keep-alive sockets. - tests/listener-dispatcher-test.ts is new coverage for the dispatcher: generates a self-signed cert via openssl into a tmp dir, then exercises TLS+h2, ALPN HTTP/1.1 fallback, plain-HTTP→https 301 redirect, the no-Host-header path that uses `socket.localAddress`, malformed-cert downgrade to plain HTTP, and the no-cert-env-vars path. `createListener` is now exported from server.ts so the test can drive it without spinning up a full realm-server fixture (and the test bootstrap's global TLS-env-var delete doesn't interfere — each test restores its own env around `startListener`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`qunit/no-assert-logical-expression` was failing on three assertions that combined multiple conditions via `&&` / `||`. Splitting them into discrete `assert.true(...)` calls makes the failure point obvious when a test breaks and clears the lint. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both `packages/workspace-sync-cli/tests/helpers/start-test-realm.ts` and `packages/realm-test-harness/src/isolated-realm-stack.ts` spawn a realm-server subprocess that inherits `process.env`. After CI's init action provisions the dev cert and `env-vars.sh` exports `REALM_SERVER_TLS_CERT_FILE/_KEY_FILE`, those env vars leak into the spawned realm-server, which binds the HTTPS+HTTP/2 dispatcher on the harness's chosen port. The integration tests and the realm-perf bench both drive plain `http://localhost:<port>/...` URLs against that server, hit the dispatcher's 301 path, and break: workspace-sync's CLI fails its session handshake with "expected 'Authorization' header" (it doesn't follow the redirect through the auth flow), and the bench fails its first GET with `404` because the realm route is behind https now. Same shape of fix as `realm-server/tests/index.ts` for the in-process qunit suite: destructure the two TLS env-var keys out of the spawn env so the child inherits everything except those. Plain `http.createServer` path, no redirect, harness HTTP URLs work as written. Production realm-servers and local dev are unaffected because they don't go through these harnesses. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`packages/host/testem-live.js` was hardcoding `http://localhost:4201/catalog/` as the realm URL and launching Chrome with the default trust policy. After the HTTPS flip, the live-test runner's `discoverTestModules` fetched against `https://localhost:4201/catalog/...` (via the host's `realmServerURL` default) but the browser navigated to `http://localhost:4201/...`, getting a 301 to https and then failing the cert check — `mkcert -install` in CI's init action is best-effort and the headless Chrome in CI doesn't always pick up the system trust store anyway. Two fixes paired: - Default realm URL flips to `https://localhost:4201/catalog/` so the navigation target matches the wire. - Chrome's CI launch args get `--ignore-certificate-errors` so the live test runner accepts the mkcert leaf without depending on system trust. Safe — the URL is fixed by REALM_URL and the connection is loopback. Dev (`launch_in_dev`) doesn't add the flag because local devs typically have run `mkcert -install` successfully and the cert is trusted normally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…registry The pre-check needs to fire on a fresh install too. `realm_registry` is populated by the realm-server's runtime bootstrap (registry backfill + reconciler), not by migrations, so it's empty when this migration runs against a freshly-created DB — the migration short-circuited and the `http://localhost:42XX` permission rows seeded by the earlier `1726671342065_backfill-realm-owners.js` migration stayed un-rewritten. The realm-server then matches incoming requests against the new `https://localhost:42XX/…` canonical and the permission rows fail to join → world-readable catalog returns 401 → Live Tests fail with "Cannot access realm https://localhost:4201/catalog/ (HTTP 401)". Switch the pre-check to `realm_user_permissions.realm_url`, which is reliably populated with the localhost canonicals by the earlier seed-style migrations. The rest of the migration body is unchanged — the per-column WHERE clauses still restrict the touch set to rows that actually contain the old URL, so production/staging DBs (real hostnames, never localhost) still no-op. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Test mode runs against the host-internal `http://test-realm/...` virtual origin via VirtualNetwork; there is no real realm-server on the wire. Many host test fixtures hardcode the `http://localhost:4201/...` canonicals in mock setups, VirtualNetwork mappings, and JSON test data, so flipping the default URLs to https caused every fetch in the test suite to fail with `TypeError: Failed to fetch` — the host's VirtualNetwork was wired with https URL mappings the test mocks didn't recognize. `environmentDefaults(environment)` now reads the ember env and picks http for `environment === 'test'`, https otherwise. Dev gets the HTTPS+HTTP/2 flip exactly as designed; test stays where it always was. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous test-mode-on-http revert was wrong: in Host Tests the realm-server actually IS running (via mise run test-services:host), and that realm-server speaks HTTPS+HTTP/2. The host bundle's defaults need to match the wire so module/data fetches over the wire (like GET /base/card-api during warmup) reach the live realm-server. The http defaults were producing failed http→https mismatches. So: - environment.js test mode reverts to https defaults (same as dev). - test-wait-for-servers.sh + live-test-wait-for-servers.sh default their readiness probe URLs to `https-get://` to match. live-test-wait-for-servers.sh also gets the same scheme-detection helper (`to_wait_scheme`) the other scripts use so an explicit REALM_URL with either scheme works. `http://test-realm/...` URLs in tests (used by the in-memory test realm registry) are still intercepted by `getRealmInfoForURL` before any wire fetch — that path is unrelated to the wire defaults and any remaining failures there are a separate concern from the HTTPS flip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sweep of every place `http://localhost:4201`/`4202` appears with runtime impact: Runtime / wire-touching: - `package.json` `openrouter:sync` default REALM_URL → https - `mise-tasks/lib/test-dev-common.sh` stub env defaults → https - `packages/host/app/services/host-mode-service.ts` `originIsNotMatrixTests` accepts both http and https origins on the matrix-tests realm ports (https is the new default; http stays recognized so older snapshots still detect the test mode). - `packages/observability/scripts/apply.sh` / `diff.sh` default `REALM_SERVER_URL` → https. Cache import: - `scripts/import-cached-index.sh` env-mode sed remap now matches both `http://localhost:4201` and `https://localhost:4201` — older cache snapshots have http canonicals, post-flip dumps have https. Either prefix gets rewritten to the env-mode Traefik hostname. In-tree realm fixture data (cards served by dev realm-server): - `packages/experiments-realm/**/*.json` and `packages/catalog-realm/**/*.json` `id` / `relationships` URLs flipped from http to https. Without this every cross-card fetch inside a render paid a wire-level 301 redirect from the dispatcher. Docs: - `README.md`, `QUICKSTART.md`, `packages/host/docs/live-tests.md`, `packages/software-factory/README.md`, `packages/bot-runner/README.md`, `docs/commands-in-headless-chrome.md` — example URLs updated. Not flipped (intentional): - Test fixture JSONs under `packages/host/tests/cards/`, `packages/realm-server/tests/cards/`, ai-bot resource chats, and bench-realm snapshot fixtures. Those URLs match test-side mount points (`http://test-realm/...`, `http://127.0.0.1:4444/test/`, bench-stack http://localhost:4201) where the test infrastructure spawns the realm-server with TLS env vars cleared and listens plain HTTP. Flipping them would diverge from what the test code registers and break the in-process fixtures. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Observability diff (vs staging)No dashboard / folder changes detected against the staging Grafana. (Run: https://github.com/cardstack/boxel/actions/runs/25784659673) |
Host Tests load the host bundle in a headless Chrome on testem (port 7357). The bundle's `realmServerURL` / `resolvedBaseRealmURL` defaults now point at `https://localhost:4201` to match the wire, but `mkcert -install` in CI's init action is best-effort and doesn't reliably land mkcert's root CA in headless Chrome's NSS trust store. Without `--ignore-certificate-errors`, every realm fetch made during shard warmup fails with `TypeError: Failed to fetch` against the self-signed cert and the rest of the shard never starts. Same fix already shipped in `testem-live.js`. Loopback only, fixed origin via host config — safe to relax cert trust. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Boxel-cli's vitest suite (and any other non-qunit caller of these helpers) doesn't share `packages/realm-server/tests/index.ts`'s bootstrap, so the global TLS env var delete that protects in-process qunit fixtures didn't apply to it. The CI init action provisions the cert, env-vars.sh exports the paths, and the test process inherits them — the spawned realm-server then binds HTTPS+HTTP/2 on its fixture port (`127.0.0.1:4446` for boxel-cli) and the CLI's plain-HTTP session calls fail with `404 Not Found` from the dispatcher's 301 path. Moving the env-var strip into the two `runTestRealmServer*` helpers themselves makes it defense-in-depth: every caller (qunit, vitest, software-factory harness) now goes through the same kill switch when spinning a fixture realm-server. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…p2-v2 # Conflicts: # .claude/skills/indexing-diagnostics/SKILL.md # packages/realm-server/scripts/full-reindex.sh # packages/realm-server/tests/realm-endpoints/info-test.ts # packages/realm-server/tests/realm-endpoints/user-test.ts
Matrix client tests timed out waiting for `http-get://localhost:4201/base/_readiness-check` because the realm-server now speaks HTTPS+HTTP/2 only. Wait-on's plain http-get probe never resolves against the https listener. Same fix for start-without-matrix.sh (dev convenience script used to bring up the stack without Synapse). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Card fixture data hardcoded http://localhost:4202 in adoptsFrom.module. With the realm-server now on HTTPS, the page is served over https and Chrome blocks mixed-content fetches of the http module URL. Flipping to https keeps the canonical realm URL consistent with the actual listener scheme. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…_FORCE_HTTP1 Adds an opt-in env-var toggle in createListener that swaps the ALPN h2 dispatcher for a plain HTTPS+HTTP/1.1 listener. CI workflows (ci.yaml, ci-host.yaml, ci-software-factory.yaml) set BOXEL_REALM_FORCE_HTTP1=1 at the workflow level so every job exercises the h1 path. Goal: isolate whether Chromium's --ignore-certificate-errors not fully covering h2 streams with the mkcert leaf cert is the cause of the Host Tests warmup hangs (and downstream test flakes). If the h1 cycle clears those failures, we know HTTP/2 is the surface that needs work; if it doesn't, the env-var toggle is a single revert. createListener now returns `proto: 'http' | 'https/h1' | 'https/h2'` instead of `isHttp2: boolean` so the listening log says exactly what mode we're in. listener-dispatcher-test clears the env var so its h2-mode assertions still exercise the h2 path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Local realm-server now speaks HTTPS+HTTP/2 only — there is no HTTP fallback. Every dev needs
mkcerton their machine and a provisioned cert beforemise run dev/pnpm start:allwill work. CI is handled (init action installs mkcert and runsensure-dev-certautomatically), but local boxes are on you.Do this once, before your first
mise run dev:Install mkcert (one-time, system package):
sudo apt install -y mkcert libnss3-toolssudo dnf install -y mkcert nss-toolsbrew install mkcert nssProvision the cert:
That generates
~/.local/share/boxel/dev-certs/{localhost.pem, localhost-key.pem}and (best-effort) runsmkcert -installso your browser silently trusts it. The task is idempotent — re-runs are no-ops until the cert is within 7 days of expiry.If mkcert is missing,
mise run devwill exit with a clear install hint instead of starting. There is no skip flag and no fallback path — provisioning the cert is a hard prereq.After pulling:
The first
mise run devwill run a Postgres migration (1779100257124_canonical-url-http-to-https.js) that rewrites every text/varchar/jsonb column on every public table fromhttp://localhost:42XX/…tohttps://localhost:42XX/…in place — index rows, realm registry, permissions, JSONB documents insidepristine_doc/search_doc/etc. The migration is idempotent and gated on a cheaprealm_registrypre-check, so re-runs and production environments are no-ops.If you have stale
http://localhost:42XX/…URLs in personal-realm card.jsonfiles (inrealms/localhost_4201/**), the wire-level HTTP→HTTPS dispatcher 301-redirects them at runtime so cards still resolve — no on-disk rewrite is required. If you want clean data:Navigation:
Visit
http://localhost:4200/(vite host) as the manual-browser entry point. The host bundle there fetches realm data from the realm-server's https origin on:4201, which is where the HTTP/2 multiplexing win lands. Visitinghttps://localhost:4201/directly works but surfaces mixed-content warnings, because vite (:4200), icons (:4206), and synapse (:8008) still serve plain HTTP.Summary
Local dev's realm-server now speaks HTTPS+HTTP/2 on a single canonical origin (
https://localhost:4201, plushttps://localhost:4202for test-realms). This unblocks the heavy aggregator-card prerender bottleneck described in CS-11114 — cohort and dashboard renders today fan out 80+ federated-search requests inside one Chromium tab, get throttled by Chrome's HTTP/1.1 6-per-origin connection ceiling, and take minutes; HTTP/2 multiplexes them over one connection and the same render finishes in seconds.Following @lukemelia's suggestion in #4787, this PR ships the single-origin design rather than the dual-listen alternative that #4787 had been carrying. There is no separate h2 alias port, no per-page
__realmH2OriginMappings__injection, no alias-host rewrite middleware — the wire protocol and the canonical realm URL agree.Design
RealmServer.listen(port): whenREALM_SERVER_TLS_CERT_FILE/_KEY_FILEare set, binds a singlenet.Serverthat peeks the first byte of every connection.0x16(TLS ClientHello) routes to the HTTP/2 secure server; anything else is treated as plain HTTP and 301-redirected tohttps://<host><path>. Sohttp://localhost:4201/fooin a browser bar or acurlinvocation gets a clean redirect instead of a TLS handshake failure. Same listener, no extra port. When the cert is absent (tests/CI without the init action's cert-provision step), falls back to plainhttp.createServer— unchanged behavior.closeAllConnections()so shutdown can force-close in-flight TLS / HTTP/2 / keep-alive sessions rather than waiting for peers to release them.main.ts's existing typeof feature-detect picks it up unchanged.middleware/index.ts#fullRequestURLdetectsctx.req.socket.encryptedso URL-keyed realm lookup matches the HTTPS canonical. It also falls back to the HTTP/2:authoritypseudo-header whenheaders.hostis absent.middleware/index.ts#fetchRequestFromContextstrips:-prefixed pseudo-headers before constructingnew Request(...)— WHATWGHeadersrejects them, and Node's http2 compat layer surfaces them onreq.headersalongside the regular headers.readFileSyncandcreateSecureServerare wrapped in try/catch so a malformed cert downgrades to plain HTTP with a warning rather than killing boot.mise run infra:ensure-dev-certprovisions the cert via mkcert and attemptsmkcert -installonce (non-fatal if declined). Idempotent.env-vars.shdefaultsREALM_BASE_URL/REALM_TEST_URLto https unconditionally and exportsNODE_EXTRA_CA_CERTSpointing at mkcert's root CA so Node-side fetches trust the cert without requiring-install.dev-common.sh+test-services/{host,realm-server,matrix}switch wait-on readiness probes tohttps-get://when the realm URL is https.config/environment.jsdefaults flip tohttps://localhost:4201forrealmServerURL/baseRealmURL/catalogRealmURL/legacyCatalogRealmURL/skillsRealmURL/openRouterRealmURL.--ignore-certificate-errorswhenREALM_BASE_URLis https, so puppeteer's bundled NSS DB (which mkcert may or may not touch depending on platform) doesn't reject the cert.CI
.github/actions/init/action.ymlinstalls mkcert via apt and runsmise run infra:ensure-dev-certas part of every job's init, so realm-servers in CI come up HTTPS+HTTP/2 the same as local. The realm-server test bootstrap (tests/index.ts) deletes the TLS env vars before any in-process fixture realm-server is spun up — supertest connects plain HTTP to those fixtures on random127.0.0.1:444Xports, and the dispatcher's plain-HTTP→301 path would otherwise break every assertion.Data migration
1779100257124_canonical-url-http-to-https.js:information_schema.columnsto find every text/varchar/jsonb column on every public table.modules(truncated on every realm-server boot),pgmigrations/migrations(the migration trackers), and generated columns (is_generated = 'NEVER').REPLACE(...)-basedUPDATEfor bothhttp://localhost:4201→https://localhost:4201andhttp://localhost:4202→https://localhost:4202.WHEREfilter restricts the touch set to rows that still contain the old URL — idempotent.realm_registrypre-check so production/staging databases short-circuit before any full-column scan (canonical URLs are real hostnames there, never localhost).mise run devpasses--migrateDBto the realm-server, so the migration fires on the first post-pull boot.Tests
packages/realm-server/tests/listener-dispatcher-test.tscovers the dispatcher: TLS h2, ALPN HTTP/1.1 fallback, plain-HTTP 301, no-Host-header raw-socket path (socket.localAddress:localPortfallback), malformed-cert downgrade, and no-cert-env-vars plain HTTP.Test plan
mise run infra:ensure-dev-certsucceeds with mkcert installed; emits clean install hints + exits 1 when missing.curl -kI --http2 https://localhost:4201/_alivereturnsHTTP/2 200.curl -kI --http1.1 https://localhost:4201/_alivereturnsHTTP/1.1 200(ALPN fallback works for h1 clients).curl -sI http://localhost:4201/_alivereturnsHTTP/1.1 301withLocation: https://localhost:4201/_alive.curl -skiL http://localhost:4201/_alive(follow redirect) lands onHTTP/2 200.mise run devruns the URL-rewrite migration → realm-server boots clean on https./_grafana-reindex?authHeader=…&realm=base/— completes without errors.mkcert -install, openhttp://localhost:4200/, log in, load a card — realm fetches showh2protocol in DevTools.mise run devshutdown closes the listener cleanly.pnpm lintpasses onpackages/realm-serverandpackages/host(lint:js+ prettier; pre-existinglint:typeserrors in../base/*.gtsare unrelated).Closes #4787 (dual-listen approach abandoned in favor of this single-origin design).
🤖 Generated with Claude Code