Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c22d07243
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Preview deployments |
Host Test Results 1 files ± 0 1 suites ±0 2h 8m 56s ⏱️ + 4m 57s Results for commit b24d62d. ± Comparison against base commit 86cb8e6. This pull request removes 1 and adds 206 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
tintinthong
force-pushed
the
is-live-test
branch
from
518f28f to
891ea2c
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces a “live test” runner page intended to execute realm-hosted (card) tests in a host-like Ember/QUnit environment, and refactors an experiments realm sample card test to use a consolidated host test setup helper.
Changes:
- Add a dedicated
live-test.html+live-test.jsrunner to load realm modules and execute theirrunTests()exports under QUnit. - Update the standard host
tests/test-helper.jsboot to skip normal initialization when running on the live-test page. - Add a
setupCardTest(hooks)helper and updatesample-command-card.gtstests (including a new search assertion).
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/host/tests/test-helper.js | Skips normal test initialization on live-test.html; imports live-test runner module. |
| packages/host/tests/live-test.js | New runtime for loading realm test modules and starting QUnit manually. |
| packages/host/tests/live-test.html | New test runner HTML page with QUnit UI and a “Start QUnit” button. |
| packages/host/tests/helpers/index.gts | Adds setupCardTest() convenience helper used by realm-hosted tests. |
| packages/host/ember-cli-build.js | Formatting-only change. |
| packages/host/app/lib/externals.ts | Adds shims for QUnit/test helpers/test modules to support realm-loaded tests. |
| packages/experiments-realm/sample-command-card.gts | Refactors test setup to setupCardTest() and adds a search-based assertion. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
This PR adds a “live” test runner mode that can load and execute test modules served from a running realm (instead of only running the host’s compiled test bundle), and wires it into CI.
Changes:
- Add a live-test bootstrap (
tests/live-test.js+ optionaltests/live-test.html) and gate the normaltests/test-helper.jsstartup when?liveTest=true. - Add a Testem config + wait-for-services script + CI job to run these realm-backed tests headlessly.
- Add a convenience helper (
setupCardTest) and a sample realm card test that validates indexing/search against a live realm.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/host/tests/test-helper.js | Skips standard Ember Exam/QUnit startup when running in live-test mode; imports live-test bootstrap. |
| packages/host/tests/live-test.js | New live-test runner: boot app, discover realm modules, shim helpers into loader, run runTests() exports. |
| packages/host/tests/live-test.html | Optional standalone HTML runner with UI controls/status for live tests. |
| packages/host/tests/helpers/index.gts | Adds setupCardTest() helper for realm card tests. |
| packages/host/testem-live.js | New Testem config to run tests/index.html?liveTest=true... and emit junit in CI. |
| packages/host/scripts/live-test-wait-for-servers.sh | Waits for dependent services/readiness endpoints then runs live tests. |
| packages/host/package.json | Adds test:live script. |
| packages/host/ember-cli-build.js | No functional change (whitespace). |
| packages/host/app/lib/externals.ts | Adds QUnit shim + test-module shims intended to let cards with colocated tests load. |
| packages/host/.eslintrc.js | Allows linting testem-live.js. |
| packages/catalog-realm/sample-command-card.gts | Updates sample card’s colocated tests; adds an indexing/search assertion. |
| mise-tasks/test-services-host | Keeps sample-command-card.gts when preparing the catalog realm for tests. |
| .github/workflows/ci-host.yaml | Adds a new “Live Tests (realm)” job to CI. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const testOnlyStub = (moduleId: string) => | ||
| new Proxy( | ||
| {}, | ||
| { | ||
| get: () => { | ||
| throw new Error( | ||
| `${moduleId} is only available in a test environment.`, | ||
| ); | ||
| }, | ||
| }, | ||
| ); |
There was a problem hiding this comment.
testOnlyStub throws on any property access via Proxy.get. That will break module evaluation for realm cards that have static imports like import { click } from '@ember/test-helpers' or import { setupCardTest } from '@cardstack/host/tests/helpers', because the export is accessed during module instantiation (before any test code runs). To allow cards to load outside tests, the stub should return callable functions (or undefined) and only throw when those functions are invoked, not when the export is imported.
| const testOnlyStub = (moduleId: string) => | |
| new Proxy( | |
| {}, | |
| { | |
| get: () => { | |
| throw new Error( | |
| `${moduleId} is only available in a test environment.`, | |
| ); | |
| }, | |
| }, | |
| ); | |
| const testOnlyStub = (moduleId: string) => { | |
| const throwingFunction = function () { | |
| throw new Error(`${moduleId} is only available in a test environment.`); | |
| }; | |
| return new Proxy( | |
| {}, | |
| { | |
| get() { | |
| return throwingFunction; | |
| }, | |
| }, | |
| ); | |
| }; |
packages/host/tests/live-test.js
Outdated
| } catch { | ||
| testModuleNames = []; |
There was a problem hiding this comment.
The fetch(realmURL) error is swallowed and replaced with testModuleNames = [], which can lead to a confusing "0 tests" outcome without indicating why test discovery failed. It would be more diagnosable to log the exception (or surface a failing test) when realm discovery fails, especially in CI.
| } catch { | |
| testModuleNames = []; | |
| } catch (error) { | |
| console.error('Failed to discover tests from realm', realmURL, error); | |
| throw error; |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c9bff23ae0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
packages/host/tests/live-test.js
Outdated
| testModuleNames = Object.entries(data.relationships ?? {}) | ||
| .filter( | ||
| ([name, entry]) => | ||
| name.endsWith('.gts') && entry.meta?.kind === 'file', | ||
| ) |
There was a problem hiding this comment.
loadRealmTests() only reads one directory payload and selects root-level entries whose names end with .gts. Realm directory responses include nested folders as meta.kind: 'directory' entries (see the directory API expectations in packages/realm-server/tests/realm-endpoints-test.ts), so any co-located tests under folders like catalog-realm/<feature>/*.gts are never imported or executed. This leaves the new live-test job with incomplete coverage and can silently miss regressions in most realm cards.
Useful? React with 👍 / 👎.
|
So i think your PR description is a little ambiguous, when we say tests are run against a live realm server, we mean a realm that is running in the browser correct like the current host tests? it's not actually a server in the classic sense, right? |
confirmed. Everything is made similar to host tests. The only difference is that we need to dynamically load up test specified in realm modules. |
Live Tests: Co-locate QUnit tests inside realm cards
I decided to call this live-test. But I'm not so commited to the name. Open to suggestions. I call it live-test bcos these are test loaded from the realm
Exercised in external catalog
Look at this PR cardstack/boxel-catalog#244
What this does
This adds a new test mode called live tests that lets you write QUnit tests directly inside a realm (e.g. alongside a card definition in
catalog-realm/), and run them against a live running realm server — without having to add them to the host test bundle.Previously, all host tests were compiled into the Ember test bundle at build time. Live tests are different: the realm serves the test files over HTTP, and the host loads and executes them at runtime.
What changed
test-helper.js— the existing test entry point now checks for a?liveTestquery param. If present it runs the live-test path; otherwise it runs the normal ember-exam path as before. A single check prevents double QUnit initialisation (two competing app instances).live-test.js— containsloadRealmTests(application), which is the async function that:.gtstest filesrunTests()on each to register QUnit moduleshelpers/setup-qunit.js— shared QUnit bootstrap (maxDepth, test waiters, qunit-dom setup, autostart flag) used by both live and normal test paths.testem-live.js— Testem config that drives Chrome againsttests/index.html?liveTest=true&realmURL=http://localhost:4201/catalog/.scripts/live-test-wait-for-servers.sh— waits for all required services then runsember test --config-file testem-live.js --path ./dist.package.json— addstest:livescript.ci-host.yaml— adds alive-testCI job (non-sharded, runs with catalog realm, noSKIP_CATALOG).catalog-realm/sample-command-card.gts— example card with a co-locatedrunTests()export demonstrating the pattern.How tests are discovered in a co-located module
A
.gtsfile in the realm can export arunTests()function alongside its card definition:At runtime,
loadRealmTestsfetches the realm index, finds all.gtsfiles, imports each one through the realm loader, and callsrunTests()if it exists. Files withoutrunTests()are skipped silently.The loader boundary
There are two separate module loaders at play:
application.buildInstance(...)) that fetches modules over HTTP from the realm server. Live test files are imported through this loader.The realm loader needs the test helpers (
@cardstack/host/tests/helpers, mock-matrix, etc.) to be shimmed into it explicitly — otherwise realm test files can't import them.loadRealmTestsdoes this before importing any test modules.Things to know
/catalog/) which has'*': read=truepermissions.live-testjob starts the catalog realm (noSKIP_CATALOG) and runspnpm test:liveagainst it.sample-command-card.gtsincatalog-realm/is the first test covered.Future work
<iframe>— giving it a fully isolated JS heap, its own Ember app instance, its own SQLite WASM, and no shared state with the running host app. Testem is not involved in the Code Mode case; QUnit renders pass/fail directly inside the iframe's own UI.How to run locally