Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 111 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,10 @@ For a quickstart, see [here](./QUICKSTART.md)

Two catalog realms run side by side. The **Cardstack Catalog** (served from [cardstack/boxel-catalog](https://github.com/cardstack/boxel-catalog)) is the source of truth for new development and the destination for community submissions. The **Legacy Catalog** (shipped from this monorepo) remains available during the deprecation window for content that hasn't been migrated upstream.

| Realm | Source | URL path | Purpose |
| ---------------------- | --------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------- |
| **Cardstack Catalog** | `packages/catalog` (clones [boxel-catalog](https://github.com/cardstack/boxel-catalog)) | `/catalog/` | Official catalog. New listings and community submissions land here via `pr-listing-create` PRs to `cardstack/boxel-catalog`. |
| **Legacy Catalog** | `packages/catalog-realm` | `/legacy-catalog/` | Historical catalog shipped from this repo. Kept visible in the workspace chooser while existing content migrates upstream. |
| Realm | Source | URL path | Purpose |
| --------------------- | --------------------------------------------------------------------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| **Cardstack Catalog** | `packages/catalog` (clones [boxel-catalog](https://github.com/cardstack/boxel-catalog)) | `/catalog/` | Official catalog. New listings and community submissions land here via `pr-listing-create` PRs to `cardstack/boxel-catalog`. |
| **Legacy Catalog** | `packages/catalog-realm` | `/legacy-catalog/` | Historical catalog shipped from this repo. Kept visible in the workspace chooser while existing content migrates upstream. |

Both catalogs appear in the workspace chooser by default; the Cardstack Catalog is sorted first, Legacy Catalog last. Both are controlled by the same `SKIP_CATALOG` flag — setting `SKIP_CATALOG=true` skips setup and startup for both.

Expand Down Expand Up @@ -195,6 +195,8 @@ Here's what is spun up with `mise run dev`:
| :4201 | `/skills` skills realm | ✅ | 🚫 |
| :4201 | `/experiments` experiments realm | ✅ | 🚫 |
| :4202 | `/test` host test realm, `/node-test` node test realm | ✅ | 🚫 |
| :4203 | HTTPS/HTTP-2 alias for the :4201 realm-server (opt-in — see "HTTP/2 dev access" below) | 🟡 | 🟡 |
| :4204 | HTTPS/HTTP-2 alias for the :4202 test realm-server (opt-in — see "HTTP/2 dev access" below) | 🟡 | 🚫 |
| :4205 | `/test` realm for matrix client tests (playwright controlled) | 🚫 | 🚫 |
| :4206 | Boxel icons server | ✅ | 🚫 |
| :4210 | Development Worker Manager (spins up 1 worker by default) | ✅ | 🚫 |
Expand All @@ -207,6 +209,105 @@ Here's what is spun up with `mise run dev`:
| :5435 | Postgres DB | ✅ | 🚫 |
| :8008 | Matrix synapse server | ✅ | 🚫 |

#### HTTP/2 dev access

##### Why this exists

Heavy aggregator cards (Cohort, dashboards) fan out 80+ federated-search
requests per render. Chrome — including the headless Chromium driving
the prerender — caps any single origin at 6 concurrent HTTP/1.1 connections,
so the 80+ requests serialize. A single cohort render takes ~4–5 minutes
under that ceiling. HTTP/2 multiplexes them over one connection and the
same render finishes in seconds.

Browsers only do HTTP/2 over TLS, so the realm-server has to terminate a
cert. We don't want a cert-management story to land in everyone's PATH,
so HTTP/2 in local dev is **opt-in via one mise task**.

##### Opting in

```
mise run infra:ensure-dev-cert
```

That task:

1. Generates a leaf cert at `~/.local/share/boxel/dev-certs/localhost.pem`
using [`mkcert`](https://github.com/FiloSottile/mkcert) (install with
`sudo apt install -y mkcert libnss3-tools` on Debian/Ubuntu or
`brew install mkcert nss` on macOS — the task prints these
instructions and exits cleanly if mkcert is missing).
Comment thread
habdelra marked this conversation as resolved.
2. Is idempotent: re-running is a no-op until the cert is within 7 days
of expiry.
3. Does **not** require sudo. The cert is signed by a CA that mkcert
keeps in your user data dir; it is not added to the system trust
store.

After provisioning, your next `mise run dev` (or `mise run dev-all`)
will bring up two extra listeners alongside the existing HTTP/1.1
servers:

- `https://localhost:4203` — h2 alias for `http://localhost:4201`
- `https://localhost:4204` — h2 alias for `http://localhost:4202`

The h2 endpoints serve the exact same content as their HTTP/1.1 peers —
they are aliases, not separate realms. A request hitting `:4203` is
rewritten internally so URL-keyed realm lookup matches the canonical
`:4201` mounts.

##### What automatically uses HTTP/2

- The prerender service's Chromium pages. They get
`--ignore-certificate-errors` plus a `window.__realmH2OriginMappings__`
override that tells the host's `VirtualNetwork` to re-route realm
fetches from `http://localhost:4201/…` → `https://localhost:4203/…`
(and `:4202` → `:4204`). Canonical card URLs are unchanged in card
data; only the wire fetch is rewritten.
- The test-realms server's indexer (via the same shared prerender).

##### What stays on HTTP/1.1

- Your terminal `curl http://localhost:4201/_alive` and other scripts —
unchanged.
- Your manual browser when you visit `http://localhost:4200/…` — that
page (and its fetches to `:4201`) continue using HTTP/1.1, so you do
_not_ need to trust the cert to keep your daily workflow working.

##### Optional: trust the cert for your own browser

If you want HTTP/2 to also speed up cards you load manually (visiting
`https://localhost:4203/…` directly), run **once**:

```
mkcert -install # one-time, requires sudo
```

That adds mkcert's root CA to your system trust store so Chrome stops
warning about the self-signed cert. Your bookmarks stay
`http://localhost:4201`/`:4200` — the install just unlocks the option
of visiting the `:4203` alias without click-throughs.

##### Verifying HTTP/2 is live

```
curl -kI --http2 https://localhost:4203/_alive
```

Look for `HTTP/2 200`. The dev `mise run dev` log line `Realm server
listening on port 4203 (https/h2 alias)` also confirms the listener
came up.

##### Opting out

Delete the cert dir or unset the env vars:

```
rm -rf ~/.local/share/boxel/dev-certs
```

The next `mise run dev` reverts to HTTP/1.1-only and indexing falls
back to the slow serial path for heavy aggregator cards.

#### Using `mise run services:realm-server`

You can also use `mise run services:realm-server` if you want the functionality of `mise run dev`, but without running the test realms. This will enable you to open http://localhost:4201 and allow to select between the cards in the /base and /experiments realm. You must also make sure to run `mise run services:worker` in order to start the workers which are normally started in `mise run dev`.
Expand Down Expand Up @@ -610,12 +711,12 @@ BOXEL_ENVIRONMENT=parallel mise run services:ai-bot

In environment mode, services are available at:

| Service | Hostname |
| ------------- | ----------------------------------------- |
| Host app | `http://host.<slug>.localhost` |
| Realm server | `http://realm-server.<slug>.localhost` |
| Matrix | `http://matrix.<slug>.localhost` |
| Icons | `http://icons.<slug>.localhost` |
| Service | Hostname |
| ------------ | -------------------------------------- |
| Host app | `http://host.<slug>.localhost` |
| Realm server | `http://realm-server.<slug>.localhost` |
| Matrix | `http://matrix.<slug>.localhost` |
| Icons | `http://icons.<slug>.localhost` |

Where `<slug>` is the lowercased, sanitized form of `BOXEL_ENVIRONMENT` (e.g., `feature/my-branch` becomes `feature-my-branch`).

Expand Down
83 changes: 83 additions & 0 deletions mise-tasks/infra/ensure-dev-cert
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
#!/usr/bin/env bash
#MISE description="Provision a local dev cert so realm-server can serve HTTPS/HTTP-2"
#
# Generates ~/.local/share/boxel/dev-certs/{localhost.pem, localhost-key.pem}
# using mkcert. Idempotent — re-runs are a no-op if the cert already exists
# and isn't near expiry.
#
# Why: indexing fans out 80+ federated-search requests per heavy aggregator
# card render. Chrome's HTTP/1.1 per-origin connection limit (6) serializes
# them and turns a single cohort render into ~4–5 minutes. HTTP/2 multiplexes
# them over one connection, dropping the same render to seconds. Browsers
# only do HTTP/2 over TLS, so realm-server needs a cert.
#
# Note: this script does NOT run `mkcert -install`. That step adds mkcert's
# root CA to the system trust store and requires sudo. It's only needed if
# you want YOUR manual browser sessions (visiting https://localhost:4203
# directly) to trust the cert without a warning. The prerenderer and the
# test-harness indexer don't need it — they launch Chromium with
# `--ignore-certificate-errors`. The hint at the end of this script shows
# how to opt in.

set -euo pipefail

CERT_DIR="${BOXEL_DEV_CERT_DIR:-$HOME/.local/share/boxel/dev-certs}"
CERT_FILE="$CERT_DIR/localhost.pem"
KEY_FILE="$CERT_DIR/localhost-key.pem"

# Idempotent skip when the cert already exists and isn't within 7 days of
# expiry. openssl's `-checkend` returns 0 if the cert is valid for at least
# the given number of seconds.
if [ -f "$CERT_FILE" ] && [ -f "$KEY_FILE" ]; then
if openssl x509 -in "$CERT_FILE" -checkend $((7 * 24 * 60 * 60)) -noout >/dev/null 2>&1; then
exit 0
fi
echo "[ensure-dev-cert] Existing cert at $CERT_FILE is near expiry; regenerating."
fi

if ! command -v mkcert >/dev/null 2>&1; then
cat >&2 <<'EOF'
[ensure-dev-cert] mkcert not installed — skipping HTTP/2 cert provisioning.
The realm-server will boot HTTP/1.1-only; indexing falls back to the slow
serial path for heavy aggregator cards. This is a soft warning, not an
error — `mise run dev` continues normally.

To enable HTTP/2, install mkcert and re-run this task:
Linux (Debian/Ubuntu): sudo apt install -y mkcert libnss3-tools
Linux (Fedora/RHEL): sudo dnf install -y mkcert nss-tools
macOS (Homebrew): brew install mkcert nss

Then: `mise run infra:ensure-dev-cert`

See the repo-root README ("HTTP/2 dev access") for the why.
EOF
exit 0
fi

mkdir -p "$CERT_DIR"

echo "[ensure-dev-cert] Generating cert at $CERT_FILE"
mkcert \
-cert-file "$CERT_FILE" \
-key-file "$KEY_FILE" \
localhost 127.0.0.1 ::1

if ! mkcert -CAROOT >/dev/null 2>&1; then
# mkcert prints CAROOT to stdout; the check above only confirms the binary
# works. If a CA root doesn't exist yet on this machine, the cert above
# will still have been generated against a freshly created root in mkcert's
# own data dir — it just isn't installed into the system trust store.
:
fi

cat <<EOF
[ensure-dev-cert] Cert provisioned. After your next `mise run dev`, the
realm-server will accept HTTP/2 on https://localhost:4203 in addition to
the existing HTTP/1.1 on http://localhost:4201.

The prerenderer and test-harness indexer always use the HTTP/2 alias
(automatic). If you also want YOUR browser to trust the cert without a
warning when visiting https://localhost:4203 directly, run:

mkcert -install # one-time, requires sudo
EOF
24 changes: 24 additions & 0 deletions mise-tasks/lib/env-vars.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,30 @@ unset _ENV_VARS_DIR _REPO_ROOT

export PGPORT="${PGPORT:-5435}"

# HTTP/2 dev access: when the dev cert provisioned by
# `mise run infra:ensure-dev-cert` is present, expose its paths to the
# realm-servers so each can bring up an HTTPS/h2 alias listener alongside
# the existing HTTP/1.1 listener. Two realm-servers run side-by-side in
# local dev: the development server on 4201 (h2 alias 4203) and the
# test-realms server on 4202 (h2 alias 4204). REALM_H2_ORIGIN_MAPPINGS
# tells the prerender's Chromium (via the host's VirtualNetwork) which
# canonical http origin to rewrite to which https origin so per-page
# fetches multiplex over h2. Absent → realm-servers stay HTTP-only.
# See the repo-root README ("HTTP/2 dev access") for the why.
_BOXEL_DEV_CERT_DIR="${BOXEL_DEV_CERT_DIR:-$HOME/.local/share/boxel/dev-certs}"
_BOXEL_DEV_CERT_FILE="$_BOXEL_DEV_CERT_DIR/localhost.pem"
_BOXEL_DEV_KEY_FILE="$_BOXEL_DEV_CERT_DIR/localhost-key.pem"
# Env-mode (BOXEL_ENVIRONMENT) uses Traefik subdomains and has its own
# routing layer; HTTP/2 wiring is standard-mode only for now.
if [ -z "${BOXEL_ENVIRONMENT:-}" ] && [ -f "$_BOXEL_DEV_CERT_FILE" ] && [ -f "$_BOXEL_DEV_KEY_FILE" ]; then
export REALM_SERVER_TLS_CERT_FILE="$_BOXEL_DEV_CERT_FILE"
export REALM_SERVER_TLS_KEY_FILE="$_BOXEL_DEV_KEY_FILE"
export REALM_SERVER_TLS_PORT="${REALM_SERVER_TLS_PORT:-4203}"
export REALM_TEST_TLS_PORT="${REALM_TEST_TLS_PORT:-4204}"
export REALM_H2_ORIGIN_MAPPINGS="${REALM_H2_ORIGIN_MAPPINGS:-[{\"from\":\"http://localhost:4201\",\"to\":\"https://localhost:${REALM_SERVER_TLS_PORT}\"},{\"from\":\"http://localhost:4202\",\"to\":\"https://localhost:${REALM_TEST_TLS_PORT}\"}]}"
fi
unset _BOXEL_DEV_CERT_DIR _BOXEL_DEV_CERT_FILE _BOXEL_DEV_KEY_FILE

# Turbo mode: boost parallelism for local development.
# All turbo defaults can be overridden individually.
if [ "${BOXEL_TURBO:-}" = "true" ]; then
Expand Down
2 changes: 1 addition & 1 deletion mise-tasks/services/realm-server
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
#MISE description="Start realm development server"
#MISE depends=["infra:ensure-traefik", "infra:ensure-pg", "infra:ensure-db", "infra:wait-for-prerender"]
#MISE depends=["infra:ensure-dev-cert", "infra:ensure-traefik", "infra:ensure-pg", "infra:ensure-db", "infra:wait-for-prerender"]
Comment thread
habdelra marked this conversation as resolved.
#MISE dir="packages/realm-server"
Comment thread
habdelra marked this conversation as resolved.

# Propagate realm-server's exit status through the trailing `| dev-log-tee.sh`
Expand Down
2 changes: 1 addition & 1 deletion mise-tasks/services/realm-server-base
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/sh
#MISE description="Start base realm server only"
#MISE depends=["infra:ensure-pg", "infra:wait-for-prerender"]
#MISE depends=["infra:ensure-dev-cert", "infra:ensure-pg", "infra:wait-for-prerender"]
#MISE dir="packages/realm-server"

if [ -z "$MATRIX_REGISTRATION_SHARED_SECRET" ]; then
Expand Down
3 changes: 2 additions & 1 deletion mise-tasks/services/test-realms
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/sh
#MISE description="Start test realm servers"
#MISE depends=["infra:ensure-traefik", "infra:ensure-pg", "infra:wait-for-prerender"]
#MISE depends=["infra:ensure-dev-cert", "infra:ensure-traefik", "infra:ensure-pg", "infra:wait-for-prerender"]
#MISE dir="packages/realm-server"

SCRIPTS_DIR="./scripts"
Expand Down Expand Up @@ -45,6 +45,7 @@ NODE_ENV=test \
GRAFANA_SECRET="shhh! it's a secret" \
MATRIX_URL="${MATRIX_URL_VAL}" \
REALM_SERVER_MATRIX_USERNAME=realm_server \
REALM_SERVER_TLS_PORT="${REALM_TEST_TLS_PORT:-}" \
ts-node \
--transpileOnly main \
--port="${TEST_PORT}" \
Expand Down
67 changes: 67 additions & 0 deletions packages/host/app/services/network.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ export default class NetworkService extends Service {

private makeVirtualNetwork() {
let virtualNetwork = new VirtualNetwork(globalThis.fetch);
this.installH2OriginMappings(virtualNetwork);
let resolvedBaseRealmURL = new URL(
withTrailingSlash(config.resolvedBaseRealmURL),
);
Expand Down Expand Up @@ -85,6 +86,22 @@ export default class NetworkService extends Service {
return virtualNetwork;
}

// HTTP/2 alias re-route: when the prerender harness injects a JSON
// list of {from, to} origin pairs as window.__realmH2OriginMappings__,
// route realm-server fetches through the HTTPS/h2 listener(s) so
// they multiplex over one connection per origin instead of
// serializing through Chrome's HTTP/1.1 6-per-origin ceiling.
// Canonical realm URLs (in card data) stay on the http origin — only
// the wire fetch is rewritten. See the repo-root README's "HTTP/2 dev
// access" section.
private installH2OriginMappings(virtualNetwork: VirtualNetwork) {
let raw = (globalThis as unknown as { __realmH2OriginMappings__?: string })
.__realmH2OriginMappings__;
for (let { from, to } of parseH2OriginMappings(raw)) {
virtualNetwork.addURLMapping(from, to);
}
Comment on lines +97 to +102
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Claude Code] Fixed in e8411db — extracted parseH2OriginMappings and require each mapping's destination to be https: and a loopback hostname (localhost, *.localhost, 127.0.0.1, ::1). A polluted global can no longer redirect realm fetches off-loopback; off-loopback / non-https entries are silently dropped.

}

resetState = () => {
this.virtualNetwork = this.makeVirtualNetwork();
};
Expand All @@ -99,3 +116,53 @@ declare module '@ember/service' {
function withTrailingSlash(url: string): string {
return url.endsWith('/') ? url : `${url}/`;
}

// Parses the JSON-encoded h2 origin mappings injected by the prerender
// harness. Each entry must have a parseable `from` URL and a `to` URL
// whose scheme is `https:` and whose hostname is a loopback alias —
// this contains the `--ignore-certificate-errors` trust relaxation to
// local-dev only. A malformed input, an empty global, or a `to` that
// would redirect realm fetches off-loopback returns nothing.
export function parseH2OriginMappings(
raw: string | undefined,
): Array<{ from: URL; to: URL }> {
if (!raw) return [];
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
return [];
}
if (!Array.isArray(parsed)) return [];
let mappings: Array<{ from: URL; to: URL }> = [];
for (let entry of parsed) {
if (!entry || typeof entry !== 'object') continue;
let from = (entry as { from?: unknown }).from;
let to = (entry as { to?: unknown }).to;
if (typeof from !== 'string' || typeof to !== 'string') continue;
let fromUrl: URL;
let toUrl: URL;
try {
fromUrl = new URL(from);
toUrl = new URL(to);
} catch {
continue;
}
if (toUrl.protocol !== 'https:') continue;
if (!isLoopbackHostname(toUrl.hostname)) continue;
if (fromUrl.origin === toUrl.origin) continue;
mappings.push({ from: fromUrl, to: toUrl });
}
return mappings;
}

function isLoopbackHostname(hostname: string): boolean {
let h = hostname.toLowerCase();
return (
h === 'localhost' ||
h.endsWith('.localhost') ||
h === '127.0.0.1' ||
h === '::1' ||
h === '[::1]'
);
}
Loading
Loading