diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 61f72112b..abcf12831 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -364,6 +364,8 @@ cases for these three BSON types — they fell to `default: return nil`. | The SessionStart hook `mxcli init` writes cannot survive an idle reap: it is guarded on `test -x ./mxcli`, and `.gitignore` excludes that binary (~85 MB) on purpose. The container is reclaimed, the repo re-cloned without it, the guard fails and the hook no-ops through `\|\| true` — the next session has no mxcli, no MxBuild cache, no database, and no message saying so | The hook inlined the whole bring-up in one shell line, so "binary missing" could only be expressed as "skip". A hook line cannot reasonably do OS/arch detection and a download; something committed has to | `cmd/mxcli/init_hook.go` (`bootstrapScriptTemplate`, `writeBootstrapScript`, `sessionStartHookCommand`, `sessionStartHookMarkers`) | Emit a committed `.claude/bootstrap-mxcli.sh` that resolves OS/arch, fetches the binary when absent (`MXCLI_TAG` to pin), then runs the setup; the hook becomes `sh .claude/bootstrap-mxcli.sh \|\| true`. **Changing the hook command breaks dedupe**, which matched on the old command string — so `addSessionStartHook` now recognises *any* known marker and **rewrites the entry in place**, migrating an old project instead of leaving it with two hooks that both run. **Generalisable**: a guard whose condition is something you deliberately do not commit is a silent no-op waiting for a fresh clone — make the guard able to satisfy itself. Verified by reproducing the reap: moved `./mxcli` out of the project, ran the hook command verbatim, watched it re-download (88 MB, new mtime) and finish with "Setup complete … database ready". Tests `TestAddSessionStartHook_MigratesLegacyCommand`, `TestEnsureSessionStartHook_WritesFile`. mxcli-todo #2 | | `mxcli exec -p app.mpr - <<'EOF' … EOF` fails with `Error reading file: open -: no such file or directory` — `-` is taken literally as a filename, so MDL cannot be piped or written as a heredoc and every ad-hoc script needs a temp file first | `exec` (and `check`) called `os.ReadFile(path)` directly, with no case for the conventional stdin spelling | `cmd/mxcli/mdlsource.go` (new `readMDLSource`, `mdlSourceLabel`), `cmd/mxcli/cmd_exec.go`, `cmd/mxcli/cmd_check.go` | One helper both commands share, so `check` gained the same spelling rather than only the reported one; `check` reports the source as `` instead of a bare `-`. Verified live: a heredoc through `exec` and a pipe through `check` both run. Tests `cmd/mxcli/mdlsource_test.go`. mxcli-todo #5 | | `mxcli syntax` documents spellings the parser rejects, so an agent following the reference writes MDL that fails — `TEXTBOX … (Binds: Attr)` ("'Binds:' is no longer supported, use 'Attribute:' instead") and `DataSource: MICROFLOW Module.MF()` (a zero-arg microflow datasource takes NO parens, unlike RETRIEVE/CALL) | Nothing checks the `syntax` corpus against the parser. `make check-skill-mdl` validates MDL blocks in the skills and the docs site, but the `Syntax`/`Example` strings in `cmd/mxcli/syntax/*.go` are not covered, so a retired spelling can sit there indefinitely | `cmd/mxcli/syntax/features_page.go` (10 × `Binds:` → `Attribute:`, the datasource parens), `cmd/mxcli/syntax/retired_spellings_test.go` (new guard) | Fix the text **and** pin it: a table-driven test fails if a retired spelling reappears in any topic's Syntax or Example. It is a spelling guard rather than a parse — the snippets are fragments (a DATAVIEW body, a property line) that do not stand alone as statements, so they cannot just be fed to the parser. Proven by reintroducing `Binds:` and watching the test name the topic and field. **A third claim in the same report did not reproduce**: `CONTAINER (OnClick: SHOW_PAGE M.P(Param: $currentObject))` parses fine on current main, so only the two verified ones were changed. mxcli-todo #8 | +| `mxcli test … --local` (or any other `StartLocalApp` caller) fails with MxBuild's `the project file path should be an absolute path`, followed by a page of Windows sample requests, whenever `-p` is given a **relative** path | `ServeServer.Build` forwarded `ProjectFilePath` verbatim. `mxcli run` had learned to absolutize at the CLI layer (findings #17), but that fix lived in `cmd_run.go`, not in the code that talks to MxBuild — so the next caller re-hit it | `cmd/mxcli/docker/mxserve.go` (`ServeServer.Build`) | Absolutize `req.ProjectFilePath` in `Build` itself, the single place that talks to MxBuild, so no future caller can miss it; also resolve `LocalAppOptions.ProjectPath` in `applyDefaults` so `DeployDir` and the runtime log path are not derived from a relative value. Test by pointing a `ServeServer` at an `httptest` fake and asserting on the request body — the CLI-layer fix cannot be tested that way, which is part of why it did not generalise | +| `mxcli test --attach` fails with `reload_model failed: Authentication failed.` — after the test microflows have already been injected into the project | The M2EE admin API and the test endpoint are **different secrets**. `attach` built its `RuntimeController` with `M2EEOptions{Token: hs.Token}` — the endpoint token — instead of the runtime's admin password | `cmd/mxcli/testrunner/runner_attach.go` (`attach`), `cmd/mxcli/testrunner/handshake.go` (`Handshake`) | Carry `AdminPass` in the handshake alongside `Token` and pass that to `M2EEOptions`. The hosting `run --local` publishes it via `docker.LocalAppInfo` (the resolved value, not the package default, so a `--admin-pass` override still works). Whenever one process drives another's M2EE API, check which credential is being passed — `defaultLocalAdminPass` and any app-level token are unrelated | | `alter page … set Editable = [expr]` (or `set Visible`) writes a project Studio Pro refuses to open: `StorageLoadException: Conditional editability settings has an invalid value '' for property Attribute`. `mxcli check` ✓ and `mx check` ✓ — neither inspects the stored value. The identical settings written by `create page` load fine | The ALTER path builds the `Forms$Conditional{Visibility,Editability}Settings` node by hand and wrote `Attribute: null`. `Attribute` is a **BY_NAME** `AttributeIdentifier`, so its unset value is the empty string, not null — exactly what the CREATE path already encodes via `codec.RegisterTypeDefaults(..., EmptyStringFields: []string{"Attribute"})`, whose comment records this same StorageLoadException from #627. Only the hand-built ALTER node missed it | `mdl/backend/pagemutator/mutator.go` (`setWidgetConditionalSettingMut`) | Write `{Key: "Attribute", Value: ""}`, not `nil`. **General rule: when one path hand-builds BSON that another path builds through the codec, diff the two encodings rather than eyeballing the hand-built one** — `mxcli bson dump --type page --object M.P` on a CREATE-authored and an ALTER-authored widget makes the divergence a one-line diff (key sets and values were otherwise identical). `SourceVariable` stays `nil`: it is BY_ID, where null *is* the absent value, so "null is wrong" is per-field, not a blanket rule. Test `TestSetWidgetConditionalSetting_AttributeIsEmptyString`; repro `mdl-examples/bug-tests/851-alter-page-conditional-attribute.mdl`. Issue #851 | | A widget conditional using a function whose name is also an MDL lexer keyword — `visible: [trim($currentObject/Slug) != '']`, `[length(…) > 0]`, `empty`/`count`/`find` — **silently drops the whole property**; `mxcli check` ✓, `mx check` ✓, and the widget renders unconditionally visible. `toUpperCase`/`isMatch`/`contains` in the same position work | `xpathFunctionName` (MDLPage.g4) enumerated only `IDENTIFIER \| HYPHENATED_ID \| NOT \| TRUE \| FALSE \| CONTAINS`, so `trim(` never matched `xpathFunctionCall`. The enclosing `[...]` then failed to parse as an `xpathConstraint` and matched the generic `propertyValueV3` alternative instead, so the visitor set `Visible` (an array) rather than `VisibleIf`, and the builder's `else if pages.StaticVisibleExpression(...)` — which reads only bool/string — never fired | `mdl/grammar/domains/MDLPage.g4` (`xpathFunctionName`) + `mdl/executor/validate_widgets.go` (`validateConsumableConditional`) | Define `xpathFunctionName : xpathWord \| NOT` — `xpathWord` is a negated token set, so it self-maintains as the lexer gains keywords; an enumerated list reacquires this bug with the next promoted function name. Safe because `xpathFunctionCall` requires a following LPAREN and no `xpathStepValue` may be followed by one, so bare `empty` still parses as a path word. `NOT` is spelled out (xpathWord excludes it). **Also add the general guard**: MDL-WIDGET19 errors when `Visible`/`Editable` holds a value that is neither routed to `VisibleIf`/`EditableIf` nor a bool/string — that is the residue signature of any conditional the visitor could not build, so the next one fails loudly instead of vanishing. `make grammar` regenerates the parser (not committed). **Verify in a browser, not at `mx check`** — a dropped property is still a valid model, so `mx check` reports 0 errors before AND after; the symptom only exists at render time (see `verify-in-runtime.md`). The repro script carries a `Bug852.Verify` page for this: `Slug` is three spaces, so `trim()` changes the outcome and a dropped `Visible` renders (Mendix defaults to visible). Pre-fix all 5 markers render; post-fix only the 3 that should. **One rule, two contexts**: `xpathConstraint` serves both `Visible:`/`Editable:` (a Mendix *client expression* — trim/length/toUpperCase/find) and a datasource `where` (real *XPath* — contains/starts-with/ends-with/string-length/not, `length()` = list length, aggregates Java-only, and `empty`/`NULL` are KEYWORDS not calls). The sets differ, so the grammar must not enumerate either; mxbuild adjudicates. Regression-test the XPath side when touching this rule — `[Name = empty]`, `[Name = NULL]`, `not()`, `contains()`, `starts-with()`, `string-length()` all still parse and `mx check` clean. Tests `TestConditionalVisibility_KeywordFunctionNames`, `TestValidateStaticWidget_UnconsumableConditional`; repro `mdl-examples/bug-tests/852-conditional-keyword-functions.mdl`. Issue #852 | | `download file $Doc;` is accepted by `mxcli check` and `mxcli exec` ("Created microflow") but the activity lands with **no action at all** — `describe` renders `-- Empty action` and `mx check` fails `[CE0008] "No action defined."`. Same for `download file $Doc show in browser;` | `microflowActionToGen` (the modelsdk write path) had no `*microflows.DownloadFileAction` case, so it hit `default: return nil` and the enclosing ActionActivity was serialized with a nil Action. Grammar, visitor, flow builder, read path and DESCRIBE formatter were all already in place, so the statement passed every stage that reports anything and vanished at the one that does not | `mdl/backend/modelsdk/microflow_write.go` (`microflowActionToGen`) | Add the case, setting `FileDocumentVariableName`, `ShowFileInBrowser` and `ErrorHandlingType` (Rollback default). **The storage key is `ShowFileInBrowser`, not `ShowInBrowser`** — the gen setter binds the right one; legacy's `parseDownloadFileAction` reads the wrong key. **Test at the round trip, not the reader**: a reader-only test starts from BSON the writer never had to produce, so `TestActionFromGen_DownloadFile` was green throughout. `roundTripMicroflow` (model→gen→codec→model) is the harness; assert the ActionActivity's `Action` is non-nil, which is the CE0008 shape itself. This is the same silent-drop mechanism as the `microflowObjectToGen` default branch (#791) — when auditing, diff the write switch's cases against `sdk/mpr/writer_microflow_actions.go`. Test `TestMicroflowRoundTrip_DownloadFile`; repro `mdl-examples/bug-tests/850-download-file-action.mdl`. Issue #850 | diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index 8e3974a91..81e8f4eb6 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -133,6 +133,7 @@ Launch `run --local` as the **sole** command in its invocation (don't chain a tr | `--screenshot-path` / `--screenshot-url` | `.mxcli/run-local.png` / app root | Screenshot output / page (URL or `/path`) | | `--screenshot-user` / `--screenshot-password` | — | Log in once, reuse session (pages behind login) | | `--runtime-log` | `.mxcli/runtime.log` | Runtime log file: JVM stdout/stderr **and** the application log (microflow `LOG` output + server stack traces, via an attached file log subscriber). `-` disables. | +| `--test-endpoint` | off | Host mxcli's token-guarded test endpoint so `mxcli test … --attach` can run a suite against this app with no boot of its own. Installed **before** the boot (the handler registers from after-startup), your own after-startup microflow is chained not displaced, and both are removed on exit. See `test-microflows.md`. | | `--debug` | off | Enable the microflow debugger at boot + start a session, so `mxcli debug break/paused/…` works from another terminal (see `debug-microflows.md`). No breakpoints = no behaviour change; disabled on shutdown. | | `--debug-pass` | `mxdebug` | Debugger password when `--debug` is set | | `--metrics` | off | Register a Prometheus meter registry at boot; the runtime serves metrics at `http://127.0.0.1:/prometheus` | diff --git a/.claude/skills/mendix/test-microflows.md b/.claude/skills/mendix/test-microflows.md index 3a2ce469a..8d1534bab 100644 --- a/.claude/skills/mendix/test-microflows.md +++ b/.claude/skills/mendix/test-microflows.md @@ -121,20 +121,132 @@ mxcli test tests/ -p app.mpr --verbose ## How It Works -The test runner uses the **after-startup microflow** pattern: +There are two mechanisms. `--local` uses the **test endpoint**; Docker uses the +older **after-startup microflow** pattern. + +### `--local`: the test endpoint + +1. Parses test files and extracts test blocks with annotations +2. Records the project's current after-startup microflow, and whether an `MxTest` + module already exists +3. Generates **one `MxTest.Test_` microflow per test**, plus a Java action + that registers an HTTP endpoint, and points after-startup at a microflow whose + only job is to call it — **no test runs during startup** +4. Builds and boots the app once +5. Invokes each test by name over HTTP; each returns its own verdict in the + response +6. Restores the original after-startup setting and removes everything generated +7. Outputs results (console, JUnit XML) + +Two consequences worth knowing when reading a failing run: + +- **A test that throws fails only itself.** It is reported as `ERROR` with the + root-cause message, and the next test still runs. Under the after-startup + mechanism an uncaught error ends the whole flow — and because that flow *is* + the startup action, it also fails the boot. +- **Results are returned, not scraped**, so a test cannot be lost to log + buffering or a runtime that stopped echoing to the console. + +Each test is a separate microflow with its own variable scope, so `$result` in +one test never collides with `$result` in another. + +#### `--watch`: keep the runtime warm + +```bash +mxcli test tests/ -p app.mpr --local --watch +``` + +The first run pays the cold boot; after that the runtime and the build server +stay up, and the suite re-runs on every change — to a test file **or** to the +project's model. Measured on an 11.13.0 app: + +| | | +|---|---| +| First run (cold boot) | ~30s | +| Edit a test → verdict on screen | **~2s** | +| Edit a microflow → verdict on screen | **~2s** | +| The tests themselves | 20–70ms | + +Editing a microflow and seeing straight away whether it still passes is the loop +this exists for. Ctrl-C stops watching and restores the project — the shutdown +prints `project restored` when it has. + +Adding, editing and deleting tests all work mid-session: the suite is re-parsed +on every change, and a deleted test's microflow is dropped rather than left +behind reporting a stale pass. + +`--watch` requires `--local`. The Docker and `--legacy-runner` paths can only +re-run tests by restarting, which is the thing being avoided. + +#### `--attach`: no boot at all + +If you already have the app running, tests can skip the boot entirely. The dev +loop has to opt into hosting the endpoint, because the handler is registered by +the after-startup microflow and so cannot be added to an app that is already up: + +```bash +# terminal 1 — the app you are working in +mxcli run --local --test-endpoint -p app.mpr + +# terminal 2 — runs in ~2s, no boot, repeatable +mxcli test tests/ -p app.mpr --attach +mxcli test tests/ -p app.mpr --attach --watch # ...and re-run on every change +``` + +The hosting app chains your project's own after-startup microflow rather than +displacing it, so it still boots normally. The endpoint and the handshake file +(`.mxcli/test-endpoint.json`, mode 0600) are removed when the app stops. + +Three things to know before reaching for it: + +- **Tests run against the running app's database**, not a scratch one, so they + can leave data behind in the app you are looking at. `--local` uses a separate + `_test` database; `--attach` does not. +- **An attach only owns its own test microflows.** The endpoint and the + after-startup setting belong to the app hosting them, and cleanup never + touches them. +- **A change needing a runtime restart is refused** — a new entity or + association. That runtime belongs to the other process. Restart it, or drop + `--attach`. + +| | Boot | Database | Owns the runtime | +|---|---|---|---| +| `--local` | ~30s each run | `_test` | yes | +| `--local --watch` | ~30s once, then ~2s | `_test` | yes | +| `--attach` | none | the running app's | no | + +#### Security of the endpoint + +It executes microflows under a system context, so it is gated four ways: + +| Guard | Behaviour | +|---|---| +| No `MXCLI_TEST_TOKEN` in the runtime's environment | The handler is **not registered at all** (404) | +| Missing or wrong `X-MxTest-Token` header | 401, compared in constant time | +| Non-loopback caller | 403 | +| `mf` outside `MxTest.Test_*` | 403 — it is not a general microflow-invocation API | + +The token is generated per run and reaches the runtime through its **environment**, +never written into the project. Combined with fail-closed registration, that means +a project which kept the `MxTest` module through a failed cleanup exposes nothing +when deployed anywhere else. + +### Docker: the after-startup microflow 1. Parses test files and extracts test blocks with annotations 2. Records the project's current after-startup microflow, and whether an `MxTest` module already exists -3. Generates a `MxTest.TestRunner` microflow with assertion logic and points - after-startup at it -4. Builds the project and restarts the runtime (Docker, or local with `--local`) +3. Generates a single `MxTest.TestRunner` microflow containing every test, and + points after-startup at it +4. Builds the project and restarts the container 5. Captures structured `MXTEST:` log lines for pass/fail 6. Restores the original after-startup setting and removes the generated runner — the whole `MxTest` module when the runner created it, otherwise just the `TestRunner` microflow 7. Outputs results (console, JUnit XML) +### Both mechanisms + The project's **Security Level is not modified**. The after-startup microflow runs in an administrative context and is not subject to it, and forcing it off breaks projects whose published REST/OData services use custom authentication. If a diff --git a/CLAUDE.md b/CLAUDE.md index faac2e8d3..088cd61be 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -496,7 +496,7 @@ go build -o bin/mxcli ./cmd/mxcli | **Full-text search** | `search 'keyword'` | Search across all strings and source | | **Linting** | `mxcli lint -p app.mpr [--format json\|sarif]` | 15 built-in rules + 27 Starlark rules (MDL, SEC, QUAL, ARCH, DESIGN, CONV) | | **Report** | `mxcli report -p app.mpr [--format markdown\|json\|html]` | Scored best practices report with category breakdown | -| **Testing** | `mxcli test tests/ -p app.mpr [--local]` | `.test.mdl` / `.test.md` files; `--local` runs on mxcli's own runtime (no Docker daemon), on its own ports + `_test` database | +| **Testing** | `mxcli test tests/ -p app.mpr [--local] [--watch] [--attach]` | `.test.mdl` / `.test.md` files. `--local` runs on mxcli's own runtime (no Docker daemon), on its own ports + `_test` database, driving a **token-guarded test endpoint** (one microflow per test, invoked over HTTP — a throwing test fails only itself, results are returned not log-scraped). `--watch` keeps the runtime warm (~30s first run, then ~2s). `--attach` runs against an app already up under `run --local --test-endpoint` (no boot; uses **that app's** database) | | **Diff** | `mxcli diff -p app.mpr changes.mdl` | Compare script against project state | | **Diff local** | `mxcli diff-local -p app.mpr --ref head` | Git diff for MPR v2 projects | | **Diff revisions** | `mxcli diff-local -p app.mpr --ref main..feature` | Compare two arbitrary git revisions | @@ -610,6 +610,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - LSP server with hover, go-to-definition, completion, diagnostics, symbols, folding - VS Code extension (`vscode-mdl`) with context menu commands (Run/Check/Selection) - Docker build integration (`mxcli docker build`) with PAD patching (Phase 1) +- Warm test loop (`mxcli test --local [--watch]`, `--attach`, `run --local --test-endpoint`): local test runs go through a **token-guarded HTTP endpoint** registered by a generated Java custom request handler, instead of compiling the suite into the project's after-startup microflow. One microflow per test, resolved by name at request time from `Core.getMicroflowNames()` and invoked with `Core.microflowCall(...).execute(...)` — so a throwing test fails only itself (not the boot), results are returned rather than scraped from the runtime log, and each test has its own variable scope. The handler **survives `reload_model`** (after-startup does not re-run, the JVM is unchanged), which is what makes `--watch` possible: ~30s first run, then ~2s from an edit — to a test *or* to the microflow under test — to a verdict. `--attach` skips even that boot by running against an app already up under `run --local --test-endpoint`, driving that process's serve + admin APIs over loopback; it uses **that app's database**, only ever adds/removes its own test microflows, and refuses a change needing a restart. Security: the handler is **not registered at all** without `MXCLI_TEST_TOKEN` in the runtime env (so a project that kept the `MxTest` module through a failed cleanup is inert in production), the token is constant-time compared, non-loopback callers are refused, `/list` is clamped to the test namespace, and only `MxTest.Test_*` may be invoked. The token reaches the runtime via its environment and is never written into the project. Docker keeps the after-startup runner (`--legacy-runner` selects it locally). Packages: `cmd/mxcli/testrunner/` (`endpoint.go`, `client.go`, `watch.go`, `host.go`, `handshake.go`). See `docs/15-testing/SPIKE_test_endpoint_request_handler.md` - Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--ensure-db` provisions the local Postgres + app database if missing; `--setup` does the non-blocking prerequisites (cache mxbuild+runtime, ensure DB) and exits — `mxcli init` wires it into a Claude Code SessionStart hook so a fresh/reaped web session self-bootstraps, and `docs-site/src/tools/bootstrap-prompt.md` is the empty-repo seed prompt. `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links (repeatable for multi-page sets, one PNG per page) and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` - External browser preview (`mxcli run --hub ` + `mxcli tunnel-hub`): the app stays local and reverse-tunnels out over a single 443 connection (embedded chisel) to a static relay, so it is reachable in a browser at a public URL — works from egress-only environments (Claude Code web), verified live through the session's MITM egress proxy. `run --hub` implies `--local`, boots the runtime with `ApplicationRootUrl` set to the assigned URL (so the SPA/`originURI` work under the public origin), resolves the control proxy honouring `NO_PROXY`, and retries forever. `mxcli tunnel-hub --domain ` is the **multi-tenant** relay: a registry keyed by prefix/project/solution/branch/worktree (stable URLs on reconnect) fronts many previews at per-subdomain hosts (`[prefix-]project[-branch].`; main collapses to the project) over one 443 with per-subdomain autocert, a registration API (`/api/register|status|deregister|backends|sessions`), and an availability overview at `hub./` **grouped by Claude Code session** (`/api/sessions`): each session lists the endpoints it exposed and links back to its `claude.ai/code` conversation. Client identity flags: `--hub-prefix`/`--hub-project`/`--hub-solution`/`--hub-branch`/`--hub-worktree` (project + branch auto-detected); `--hub-session` groups a session's endpoints (auto-detected from `CLAUDE_CODE_REMOTE_SESSION_ID`). Past sessions are retained: a durable per-session endpoint history (`--sessions-file`, default `~/.mxcli/hub-sessions.json`) survives restarts and reaping, and is pruned after `--session-retention` (default 30d) — so the overview shows offline sessions too (`SessionLog` in `cmd/mxcli/tunnelhub/sessions.go`). Package: `cmd/mxcli/tunnelhub/`. See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` (slices 3–4) - Tunnel-hub GitHub authentication (opt-in, gated on `--github-oauth-client-id`; absent = today's open hub): **viewer plane** — GitHub OAuth web flow + HMAC-signed SSO session cookie (`Domain=.`), owner-checked previews (`--require-auth` default on → 302 to login / 403 non-owner; soft mode filters the listing only), `/api/backends` filtered to the viewer (unauthenticated → 401), admin "signed in as" via `/api/whoami`. **Registration plane** — durable, hashed hub API keys (`--keys-file`, default `~/.mxcli/hub-keys.json`, survive restarts) presented as `X-Hub-Key` → stamps `Backend.Owner`; shared `X-Hub-Secret` still works as an owner-less fallback. **Key issuance** — the hub's `/cli` browser page mints a key from the session cookie (no PAT; the device flow was removed as Claude Code containers block GitHub's device endpoints), rotate-by-default + count + revoke-all; `mxcli auth hub login --token ` is the headless path; `run --hub` reads `MXCLI_HUB_KEY` (env → `~/.mxcli/auth.json`) and degrades to local-only if registration fails. Append-only JSONL audit trail (`--audit-log`, no secrets). Packages: `cmd/mxcli/tunnelhub/` (+`audit/`), `cmd/mxcli/hubauth/`. See `docs/11-proposals/PROPOSAL_hub_authentication.md` diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index c2df3b6d7..6f28d7418 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -9,6 +9,7 @@ import ( "github.com/mendixlabs/mxcli/cmd/mxcli/docker" "github.com/mendixlabs/mxcli/cmd/mxcli/hubauth" + "github.com/mendixlabs/mxcli/cmd/mxcli/testrunner" "github.com/spf13/cobra" ) @@ -47,6 +48,20 @@ subscriber after start, so the application log lands there too (a standalone runtime attaches no subscriber by default). The path is printed at boot; override with --runtime-log , or "-" to disable. +With --test-endpoint, the app hosts mxcli's token-guarded test endpoint, so +'mxcli test -p --attach' runs a suite against this already-warm +app instead of booting a runtime of its own — a couple of seconds instead of ~30. +The endpoint has to be installed before the boot (its handler is registered by +the after-startup microflow, which only runs at startup), so it cannot be added +to an app that is already up. Your project's own after-startup microflow is +chained, not displaced, so the app still boots the way you expect. The endpoint +is removed and the project restored when the app stops. + +Two things to know: tests then run against THIS app's database, not a scratch +one, and while the app is up its model carries a microflow-executing endpoint — +guarded by a per-run token, loopback-only, and limited to the generated +MxTest.Test_* microflows, but present. Leave the flag off for a normal dev loop. + With --debug, the microflow debugger is enabled at boot and a session is started, so 'mxcli debug break/paused/step/continue' works from another terminal (use the same -p). No breakpoints exist until you set one, so --debug alone does not change @@ -66,6 +81,7 @@ custom OpenTelemetry span filters. Examples: mxcli run --local -p app.mpr mxcli run --local -p app.mpr --watch + mxcli run --local -p app.mpr --test-endpoint # then: mxcli test tests/ -p app.mpr --attach mxcli run --local -p app.mpr --debug # then: mxcli debug break … -p app.mpr mxcli run --local -p app.mpr --app-port 8081 --db-name myapp mxcli run --hub https://hub.example.com -p app.mpr # browser preview @@ -109,6 +125,7 @@ Examples: } watch, _ := cmd.Flags().GetBool("watch") + testEndpoint, _ := cmd.Flags().GetBool("test-endpoint") ensureDB, _ := cmd.Flags().GetBool("ensure-db") setupOnly, _ := cmd.Flags().GetBool("setup") appPort, _ := cmd.Flags().GetInt("app-port") @@ -175,8 +192,40 @@ Examples: Stderr: os.Stderr, } + // --test-endpoint installs the token-guarded test endpoint into the project + // so `mxcli test --attach` can run tests against this app without booting + // its own runtime. It must be installed before the boot (the handler is + // registered by the after-startup microflow, which only runs at startup) + // and removed on the way out. + var hosted *testrunner.HostedEndpoint + if testEndpoint { + if setupOnly { + fmt.Fprintln(os.Stderr, "Error: --test-endpoint has nothing to do with --setup (which never boots the app)") + os.Exit(1) + } + var err error + hosted, err = testrunner.InstallHostedEndpoint(projectPath, os.Stdout) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + // Ctrl-C is the normal way to stop a dev loop, and it must not leave the + // endpoint in the project. RunLocal returns on SIGINT, so the deferred + // removal runs — but os.Exit below would skip it, hence the explicit + // removal on the error path too. Remove is idempotent. + defer hosted.Remove() + opts.Env = append(opts.Env, hosted.Env...) + opts.OnReady = func(info docker.LocalAppInfo) { + if err := hosted.Publish(info); err != nil { + fmt.Fprintf(os.Stderr, "Warning: could not publish the test-endpoint handshake: %v\n", err) + } + } + } + if err := docker.RunLocal(opts); err != nil { fmt.Fprintf(os.Stderr, "Error: %v\n", err) + // os.Exit skips deferred calls, so remove explicitly here. + hosted.Remove() os.Exit(1) } }, @@ -193,6 +242,7 @@ func init() { runCmd.Flags().String("hub-worktree", "", "Worktree label to distinguish multiple worktrees of one branch") runCmd.Flags().String("hub-session", "", "Session id to group this preview under in the hub overview (default: CLAUDE_CODE_REMOTE_SESSION_ID / MXCLI_HUB_SESSION)") runCmd.Flags().Bool("watch", false, "Rebuild and hot-apply on every project change") + runCmd.Flags().Bool("test-endpoint", false, "Host mxcli's token-guarded test endpoint so 'mxcli test --attach' can run tests against this app without booting its own runtime (removed on exit)") runCmd.Flags().Bool("ensure-db", false, "Provision the local Postgres + app database if missing (fresh-session bootstrap)") runCmd.Flags().Bool("setup", false, "Prepare prerequisites (cache MxBuild+runtime, ensure DB) and exit without booting — for a SessionStart hook") runCmd.Flags().Int("app-port", 0, "HTTP port for the app (default 8080)") diff --git a/cmd/mxcli/cmd_test_run.go b/cmd/mxcli/cmd_test_run.go index 74254dccf..2e86c1a35 100644 --- a/cmd/mxcli/cmd_test_run.go +++ b/cmd/mxcli/cmd_test_run.go @@ -27,19 +27,54 @@ Tests use MDL syntax with javadoc-style annotations for expectations: ); / -The test runner: -1. Parses test files and extracts test blocks with @test/@expect annotations -2. Generates a TestRunner microflow -3. Injects it into the project as after-startup microflow -4. Builds and restarts the Mendix runtime (Docker, or --local) -5. Captures structured log output to determine pass/fail -6. Restores original project settings - With --local the app runs on mxcli's own runtime instead of a container — the same boot as 'mxcli run --local', so no Docker daemon is needed. It uses its own ports (8081/8091) and its own '_test' database, so a warm 'run --local' loop can keep serving the same project while tests run. +--local also uses a different, better mechanism to run the tests: + + 1. Parses test files and extracts test blocks with @test/@expect annotations + 2. Generates one microflow per test, plus a Java action that registers a + token-guarded HTTP endpoint + 3. Boots the app once — startup only registers the endpoint, it runs no tests + 4. Invokes each test by name over HTTP; the verdict comes back in the response + 5. Restores original project settings + +Because each test is its own microflow invoked on its own, a test that throws +fails only itself instead of ending the run, and results are returned rather +than recovered from the runtime log. + +The endpoint is only reachable from loopback, only with a per-run token passed +to the runtime through its environment (never written into your project), and +will only ever invoke the generated MxTest.Test_* microflows. With no token in +the environment it is not registered at all, so a project that kept the MxTest +module through a failed cleanup exposes nothing when deployed elsewhere. + +--local --watch keeps the runtime and the build server up between runs and +re-runs the suite on every change — to a test file, or to the project's model. +The first run pays the cold boot (~30s); each one after it is a warm rebuild +(~1-4s) plus the tests themselves (milliseconds). Editing a microflow and seeing +whether it still passes is the loop this exists for. Ctrl-C stops watching and +restores the project. + +--attach skips the boot entirely and runs against an app already started with +'mxcli run --local --test-endpoint'. That app's runtime is already warm, so a run +costs only the test-microflow injection, a warm rebuild, and the tests: about two +seconds, with no cold boot at all. It combines with --watch. + +The trade is deliberate and worth knowing: the tests run against the running +app's database, not a scratch one, so they can leave data behind in the app you +are looking at. An attach only ever adds and removes its own test microflows — +the endpoint and the after-startup setting belong to the app hosting them. A +change that needs a runtime restart (a new entity or association) is refused, +since that runtime belongs to the other process. + +Without --local the Docker path is used instead: the suite is compiled into a +single after-startup microflow, the container is restarted, and results are +parsed out of its log. Pass --legacy-runner to use that mechanism on a local run +too, if the endpoint ever misbehaves. + Supports two file formats: .test.mdl — Pure MDL test blocks separated by / .test.md — Markdown specification with embedded mdl-test code blocks @@ -60,6 +95,15 @@ Examples: # Run without Docker, on mxcli's own local runtime mxcli test tests/ -p app.mpr --local + # Keep the runtime warm and re-run on every change + mxcli test tests/ -p app.mpr --local --watch + + # Run against an app already up (mxcli run --local --test-endpoint) — no boot + mxcli test tests/ -p app.mpr --attach + + # ...and re-run on every change, still without owning the runtime + mxcli test tests/ -p app.mpr --attach --watch + # Skip build (reuse existing deployment) mxcli test tests/ -p app.mpr --skip-build @@ -73,6 +117,9 @@ Examples: junitOutput, _ := cmd.Flags().GetString("junit") skipBuild, _ := cmd.Flags().GetBool("skip-build") local, _ := cmd.Flags().GetBool("local") + legacyRunner, _ := cmd.Flags().GetBool("legacy-runner") + watch, _ := cmd.Flags().GetBool("watch") + attach, _ := cmd.Flags().GetBool("attach") verbose, _ := cmd.Flags().GetBool("verbose") color, _ := cmd.Flags().GetBool("color") timeoutStr, _ := cmd.Flags().GetString("timeout") @@ -99,16 +146,19 @@ Examples: } opts := testrunner.RunOptions{ - ProjectPath: projectPath, - TestFiles: args, - SkipBuild: skipBuild, - Local: local, - Timeout: timeout, - JUnitOutput: junitOutput, - Verbose: verbose, - Color: color, - Stdout: os.Stdout, - Stderr: os.Stderr, + ProjectPath: projectPath, + TestFiles: args, + SkipBuild: skipBuild, + Local: local, + LegacyRunner: legacyRunner, + Watch: watch, + Attach: attach, + Timeout: timeout, + JUnitOutput: junitOutput, + Verbose: verbose, + Color: color, + Stdout: os.Stdout, + Stderr: os.Stderr, } result, err := testrunner.Run(opts) @@ -117,6 +167,11 @@ Examples: os.Exit(1) } + // A --watch session interrupted before any run completed has no result to + // report. Exiting 0 is right: nothing failed, the user just stopped watching. + if result == nil { + return + } if !result.AllPassed() { os.Exit(1) } diff --git a/cmd/mxcli/docker/localapp.go b/cmd/mxcli/docker/localapp.go index cae6a7429..d68874178 100644 --- a/cmd/mxcli/docker/localapp.go +++ b/cmd/mxcli/docker/localapp.go @@ -36,6 +36,10 @@ type LocalAppOptions struct { // RuntimeLogPath tees the runtime JVM output and the runtime's own // application log to this file. RuntimeLogPath string + // Env are extra "KEY=value" entries for the runtime JVM (see + // LocalRuntimeOptions.Env) — how a secret reaches the runtime without being + // written to disk. + Env []string // Stdout/Stderr receive progress messages. Stdout io.Writer Stderr io.Writer @@ -55,6 +59,16 @@ type LocalApp struct { } func (o *LocalAppOptions) applyDefaults() { + // Resolve the project path before anything is derived from it: DeployDir + // below, and the runtime's own working directory, both hang off it, and a + // relative value would leave them relative to whatever cwd the caller + // happened to have. ServeServer.Build absolutizes too — that is the backstop + // for MxBuild's own requirement; this is so the paths around it agree. + if o.ProjectPath != "" && !filepath.IsAbs(o.ProjectPath) { + if abs, err := filepath.Abs(o.ProjectPath); err == nil { + o.ProjectPath = abs + } + } if o.DeployDir == "" { o.DeployDir = filepath.Join(filepath.Dir(o.ProjectPath), "deployment") } @@ -166,6 +180,7 @@ func StartLocalApp(opts LocalAppOptions) (*LocalApp, error) { AdminPass: opts.AdminPass, DB: opts.DB, RuntimeLogPath: opts.RuntimeLogPath, + Env: opts.Env, Stdout: opts.Stdout, Stderr: opts.Stderr, }) @@ -177,6 +192,40 @@ func StartLocalApp(opts LocalAppOptions) (*LocalApp, error) { return app, nil } +// Rebuild rebuilds the project through the warm serve server and applies the +// result to the running runtime, returning whether that was a hot reload or a +// restart. This is the warm loop: the serve server keeps the model loaded, so a +// rebuild is ~1s instead of the ~15s cold build, and the runtime is only +// restarted when the build says the metamodel changed. +// +// A restart re-spawns the JVM from the same options, so anything passed via Env +// — notably the test runner's endpoint token — survives it. +// +// Returns an error if the app was started with SkipBuild: there is no serve +// server to rebuild through. +func (a *LocalApp) Rebuild(projectPath string) (ApplyAction, *BuildResult, error) { + if a.serve == nil { + return ActionReload, nil, fmt.Errorf("this app was started without a build server (SkipBuild); nothing to rebuild through") + } + if a.Runtime == nil { + return ActionReload, nil, fmt.Errorf("the runtime is not running") + } + build, err := a.serve.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: projectPath}) + if err != nil { + return ActionReload, nil, err + } + if !build.OK() { + return ActionReload, build, fmt.Errorf("build failed: %s", build.Message) + } + action, err := a.Runtime.Controller().ApplyBuild(build, a.Runtime.Restart) + return action, build, err +} + +// ProjectSourceMTime is the newest modification time across a project's model +// source — the change signal a warm loop polls. Exported for callers outside +// this package that run their own watch loop (the test runner). +func ProjectSourceMTime(projectPath string) time.Time { return projectSourceMTime(projectPath) } + // Stop shuts down the runtime and the build server. Safe to call more than once // and on a partially-started app. func (a *LocalApp) Stop() error { diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go index 08cfa3f9c..ded08b3a2 100644 --- a/cmd/mxcli/docker/localboot.go +++ b/cmd/mxcli/docker/localboot.go @@ -88,6 +88,12 @@ type LocalRuntimeOptions struct { // ReadyTimeout bounds how long StartLocalRuntime waits for the admin API // (default 90s). ReadyTimeout time.Duration + // Env are extra "KEY=value" entries layered onto the runtime JVM's + // environment, last-wins over the inherited process environment. Used to hand + // the runtime a secret that must not be written to disk — the test runner + // passes its per-run endpoint token this way rather than baking it into the + // generated Java source, which would land in the user's javasource/ tree. + Env []string // Stdout/Stderr receive progress messages (default os.Stdout/os.Stderr). Stdout io.Writer Stderr io.Writer @@ -161,14 +167,17 @@ func (o *LocalRuntimeOptions) jvmArgs() []string { // localRuntimeEnv builds the environment for the runtime JVM, layered on the // current process environment. PrepareMxCommand later adds the FreeType fix. +// o.Env is appended last so a caller-supplied value wins over both the inherited +// environment and these defaults. func localRuntimeEnv(o LocalRuntimeOptions) []string { - return append(os.Environ(), + env := append(os.Environ(), "M2EE_ADMIN_PASS="+o.AdminPass, fmt.Sprintf("M2EE_ADMIN_PORT=%d", o.AdminPort), "M2EE_ADMIN_LISTEN_ADDRESSES="+o.ListenAddr, "MX_INSTALL_PATH="+o.InstallPath, "MX_LOG_LEVEL=i", ) + return append(env, o.Env...) } // otelAgentJar locates the OpenTelemetry Java agent bundled with the runtime diff --git a/cmd/mxcli/docker/mxserve.go b/cmd/mxcli/docker/mxserve.go index b19557eda..af78aeabb 100644 --- a/cmd/mxcli/docker/mxserve.go +++ b/cmd/mxcli/docker/mxserve.go @@ -184,10 +184,26 @@ func (s *ServeServer) waitReady(timeout time.Duration) error { // Build sends a build request to the warm server and parses the result. The // caller inspects RestartRequired to decide reload_model vs restart. +// +// A relative ProjectFilePath is resolved here rather than by each caller. +// MxBuild rejects one outright ("the project file path should be an absolute +// path") and answers with a page of Windows sample requests, which tells a user +// who typed `-p app.mpr` nothing about what to do. `mxcli run` learned to +// absolutize at the CLI layer (findings #17), but that left the requirement +// unenforced for every other caller — and `mxcli test --local` then hit exactly +// the same error through StartLocalApp. Doing it at the one place that talks to +// MxBuild is what stops a third caller finding it again. func (s *ServeServer) Build(req BuildRequest) (*BuildResult, error) { if req.Target == "" { req.Target = TargetDeploy } + if req.ProjectFilePath != "" && !filepath.IsAbs(req.ProjectFilePath) { + abs, err := filepath.Abs(req.ProjectFilePath) + if err != nil { + return nil, fmt.Errorf("resolving project path %q: %w", req.ProjectFilePath, err) + } + req.ProjectFilePath = abs + } bodyBytes, err := json.Marshal(req) if err != nil { return nil, fmt.Errorf("marshaling build request: %w", err) diff --git a/cmd/mxcli/docker/mxserve_test.go b/cmd/mxcli/docker/mxserve_test.go index 3c0f774a6..f991569a1 100644 --- a/cmd/mxcli/docker/mxserve_test.go +++ b/cmd/mxcli/docker/mxserve_test.go @@ -4,7 +4,8 @@ package docker import ( "encoding/json" - "net" + "fmt" + "io" "net/http" "net/http/httptest" "net/url" @@ -14,138 +15,113 @@ import ( "testing" ) -// newTestServe returns a ServeServer wired to an httptest server, so the HTTP -// client (Build) can be tested without spawning mxbuild. -func newTestServe(t *testing.T, handler http.HandlerFunc) *ServeServer { +// fakeServe stands in for `mxbuild --serve`, recording the build request it was +// sent so a test can assert on what actually went over the wire. +func fakeServe(t *testing.T) (*ServeServer, *BuildRequest) { t.Helper() - ts := httptest.NewServer(handler) - t.Cleanup(ts.Close) - u, err := url.Parse(ts.URL) + var got BuildRequest + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(body, &got) + w.Header().Set("Content-Type", "application/json") + fmt.Fprint(w, `{"status":"Success"}`) + })) + t.Cleanup(srv.Close) + + u, err := url.Parse(srv.URL) if err != nil { - t.Fatalf("parse test URL: %v", err) + t.Fatalf("parsing test server URL: %v", err) } - host, portStr, err := net.SplitHostPort(u.Host) + port, err := strconv.Atoi(u.Port()) if err != nil { - t.Fatalf("split host:port: %v", err) + t.Fatalf("parsing test server port: %v", err) } - port, _ := strconv.Atoi(portStr) - return &ServeServer{Host: host, Port: port} + return &ServeServer{Host: u.Hostname(), Port: port}, &got } -func TestServeBuild_DeployRestartRequired(t *testing.T) { - s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/build" { - t.Errorf("path = %q, want /build", r.URL.Path) - } - if r.Method != http.MethodPost { - t.Errorf("method = %q, want POST", r.Method) - } - var req BuildRequest - if err := json.NewDecoder(r.Body).Decode(&req); err != nil { - t.Errorf("decode request: %v", err) - } - if req.Target != TargetDeploy { - t.Errorf("target = %q, want Deploy (default)", req.Target) - } - if req.ProjectFilePath != "/x/App.mpr" { - t.Errorf("projectFilePath = %q", req.ProjectFilePath) - } - _, _ = w.Write([]byte(`{"restartRequired": true, "status": "Success"}`)) - }) +// TestBuildAbsolutizesProjectPath pins the fix for MxBuild's "the project file +// path should be an absolute path" rejection. `mxcli run` used to absolutize at +// the CLI layer, which left `mxcli test --local` hitting the raw error through +// StartLocalApp; doing it in Build covers every caller. +func TestBuildAbsolutizesProjectPath(t *testing.T) { + dir := t.TempDir() + mpr := filepath.Join(dir, "App.mpr") + if err := os.WriteFile(mpr, []byte("x"), 0o600); err != nil { + t.Fatalf("writing fixture: %v", err) + } - res, err := s.Build(BuildRequest{ProjectFilePath: "/x/App.mpr"}) // Target empty -> Deploy + // Run from the project directory so "App.mpr" is a valid relative path. + cwd, err := os.Getwd() if err != nil { - t.Fatalf("Build: %v", err) - } - if !res.OK() { - t.Errorf("OK() = false, status = %q", res.Status) + t.Fatalf("getwd: %v", err) } - if !res.RestartRequired { - t.Error("RestartRequired = false, want true (domain/view-entity change)") + if err := os.Chdir(dir); err != nil { + t.Fatalf("chdir: %v", err) } -} + t.Cleanup(func() { os.Chdir(cwd) }) -func TestServeBuild_HotReloadable(t *testing.T) { - s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { - _, _ = w.Write([]byte(`{"restartRequired": false, "status": "Success"}`)) - }) - res, err := s.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: "/x/App.mpr"}) - if err != nil { + srv, got := fakeServe(t) + if _, err := srv.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: "App.mpr"}); err != nil { t.Fatalf("Build: %v", err) } - if !res.OK() { - t.Errorf("OK() = false, status = %q", res.Status) + + if !filepath.IsAbs(got.ProjectFilePath) { + t.Fatalf("MxBuild was sent a relative path %q; it rejects those", got.ProjectFilePath) } - if res.RestartRequired { - t.Error("RestartRequired = true, want false (microflow/page change -> reload_model)") + // EvalSymlinks because macOS /tmp is a symlink to /private/tmp. + wantResolved, _ := filepath.EvalSymlinks(mpr) + gotResolved, _ := filepath.EvalSymlinks(got.ProjectFilePath) + if gotResolved != wantResolved { + t.Errorf("sent %q, want %q", gotResolved, wantResolved) } } -func TestServeBuild_Failure(t *testing.T) { - s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { - w.WriteHeader(http.StatusBadRequest) - _, _ = w.Write([]byte(`{"status": "Failure", "message": "Specified 'target' is invalid."}`)) - }) - res, err := s.Build(BuildRequest{Target: "Bogus", ProjectFilePath: "/x/App.mpr"}) - if err != nil { - t.Fatalf("Build should parse the failure envelope, got transport error: %v", err) - } - if res.OK() { - t.Error("OK() = true, want false") +// TestBuildLeavesAnAbsolutePathAlone guards against the resolution mangling a +// path that was already correct. +func TestBuildLeavesAnAbsolutePathAlone(t *testing.T) { + abs := filepath.Join(t.TempDir(), "App.mpr") + srv, got := fakeServe(t) + if _, err := srv.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: abs}); err != nil { + t.Fatalf("Build: %v", err) } - if res.Message == "" { - t.Error("Message empty, want the failure message") + if got.ProjectFilePath != abs { + t.Errorf("absolute path was rewritten: got %q, want %q", got.ProjectFilePath, abs) } } -func TestServeBuild_PackageTargetSendsMdaPath(t *testing.T) { - s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { - var req BuildRequest - _ = json.NewDecoder(r.Body).Decode(&req) - if req.Target != TargetPackage { - t.Errorf("target = %q, want Package", req.Target) - } - if req.MdaFilePath != "/out/app.mda" { - t.Errorf("mdaFilePath = %q, want /out/app.mda", req.MdaFilePath) - } - _, _ = w.Write([]byte(`{"restartRequired": true, "status": "Success"}`)) - }) - if _, err := s.Build(BuildRequest{Target: TargetPackage, ProjectFilePath: "/x/App.mpr", MdaFilePath: "/out/app.mda"}); err != nil { +// TestBuildDefaultsTargetToDeploy pins the pre-existing default, which the +// absolutization now sits next to. +func TestBuildDefaultsTargetToDeploy(t *testing.T) { + srv, got := fakeServe(t) + if _, err := srv.Build(BuildRequest{ProjectFilePath: filepath.Join(t.TempDir(), "App.mpr")}); err != nil { t.Fatalf("Build: %v", err) } + if got.Target != TargetDeploy { + t.Errorf("Target = %q, want %q", got.Target, TargetDeploy) + } } -func TestVerifyMxBuildCache(t *testing.T) { - // layout: /modeler/mxbuild and /runtime - cache := t.TempDir() - modeler := filepath.Join(cache, "modeler") - if err := os.MkdirAll(modeler, 0o755); err != nil { - t.Fatal(err) +// TestLocalAppOptionsAbsolutizeProjectPath pins that DeployDir is not derived +// from a relative project path. +func TestLocalAppOptionsAbsolutizeProjectPath(t *testing.T) { + dir := t.TempDir() + cwd, err := os.Getwd() + if err != nil { + t.Fatalf("getwd: %v", err) } - mxbuild := filepath.Join(modeler, "mxbuild") - if err := os.WriteFile(mxbuild, []byte("#!/bin/sh\n"), 0o755); err != nil { - t.Fatal(err) + if err := os.Chdir(dir); err != nil { + t.Fatalf("chdir: %v", err) } + t.Cleanup(func() { os.Chdir(cwd) }) - // runtime/ missing -> error - if err := verifyMxBuildCache(mxbuild); err == nil { - t.Error("expected error when runtime/ dir is missing") - } + o := LocalAppOptions{ProjectPath: "App.mpr"} + o.applyDefaults() - // runtime/ present -> ok - if err := os.MkdirAll(filepath.Join(cache, "runtime"), 0o755); err != nil { - t.Fatal(err) - } - if err := verifyMxBuildCache(mxbuild); err != nil { - t.Errorf("expected no error when runtime/ present, got %v", err) + if !filepath.IsAbs(o.ProjectPath) { + t.Errorf("ProjectPath = %q, want an absolute path", o.ProjectPath) } -} - -func TestServeBuild_BadJSONBody(t *testing.T) { - s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { - _, _ = w.Write([]byte(`not json`)) - }) - if _, err := s.Build(BuildRequest{ProjectFilePath: "/x/App.mpr"}); err == nil { - t.Error("expected an error decoding a non-JSON body") + if !filepath.IsAbs(o.DeployDir) { + t.Errorf("DeployDir = %q, want an absolute path", o.DeployDir) } } diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index b91ea0d2d..fdf7fb2aa 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -29,6 +29,17 @@ import ( // serve build's restartRequired flag decides which. See // docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md. +// LocalAppInfo is what a running local app exposes to another process: the +// loopback ports it serves on, and the admin password needed to drive its M2EE +// API. The admin password is separate from any application-level credential — +// conflating the two is an authentication failure at the first admin call. +type LocalAppInfo struct { + AppPort int + AdminPort int + ServePort int + AdminPass string +} + // LocalRunOptions configures RunLocal. type LocalRunOptions struct { // ProjectPath is the .mpr file. @@ -120,6 +131,15 @@ type LocalRunOptions struct { // charts, since the console exporter omits timestamps/parent span IDs. // Implies Trace. TraceOTLP string + // Env are extra "KEY=value" entries for the runtime JVM (see + // LocalRuntimeOptions.Env) — how a secret reaches the runtime without being + // written to disk. `--test-endpoint` passes the test-endpoint token this way. + Env []string + // OnReady, when set, is called once the app is serving, with everything a + // second process needs to drive it. Used by `--test-endpoint` to publish its + // handshake only after there is something for `mxcli test --attach` to + // connect to. + OnReady func(LocalAppInfo) // RuntimeSettings are raw "Key=Value" runtime settings merged into the boot // update_configuration payload (Value is parsed as JSON, else a string), e.g. // 'Metrics.Registries=[{"type":"otlp"}]' or @@ -692,6 +712,7 @@ func RunLocal(opts LocalRunOptions) error { Trace: opts.Trace, TraceServiceName: traceService, TraceOTLPEndpoint: opts.TraceOTLP, + Env: opts.Env, Stdout: w, Stderr: stderr, }) @@ -700,6 +721,15 @@ func RunLocal(opts LocalRunOptions) error { } defer rt.Stop() + if opts.OnReady != nil { + opts.OnReady(LocalAppInfo{ + AppPort: opts.AppPort, + AdminPort: opts.AdminPort, + ServePort: opts.ServePort, + AdminPass: opts.AdminPass, + }) + } + fmt.Fprintf(w, "\nApp is running at %s\n", rt.AppURL()) // The local runtime boots with the live-preview dev flags (see // LocalRuntimeOptions.jvmArgs), so `mxcli oql` can query it directly — and it diff --git a/cmd/mxcli/main.go b/cmd/mxcli/main.go index 3703823a3..be87beadd 100644 --- a/cmd/mxcli/main.go +++ b/cmd/mxcli/main.go @@ -371,6 +371,9 @@ func init() { testRunCmd.Flags().StringP("junit", "j", "", "Write JUnit XML results to file") testRunCmd.Flags().BoolP("skip-build", "s", false, "Skip build step (reuse existing deployment)") testRunCmd.Flags().Bool("local", false, "Run on mxcli's local runtime instead of Docker (no daemon needed)") + testRunCmd.Flags().Bool("legacy-runner", false, "With --local, run tests from the after-startup microflow and parse the log, instead of over the test endpoint") + testRunCmd.Flags().BoolP("watch", "w", false, "With --local, keep the runtime warm and re-run the suite on every test or model change (Ctrl-C to stop)") + testRunCmd.Flags().Bool("attach", false, "Run against an app already started with 'mxcli run --local --test-endpoint' instead of booting one (tests hit that app's database)") testRunCmd.Flags().BoolP("verbose", "v", false, "Show all runtime log output") testRunCmd.Flags().BoolP("color", "", false, "Use colored output") testRunCmd.Flags().StringP("timeout", "t", "5m", "Timeout for runtime startup and test execution") diff --git a/cmd/mxcli/syntax/features_misc.go b/cmd/mxcli/syntax/features_misc.go index 131a885de..1a3f4f108 100644 --- a/cmd/mxcli/syntax/features_misc.go +++ b/cmd/mxcli/syntax/features_misc.go @@ -265,18 +265,25 @@ SEARCH 'word*';`, Register(SyntaxFeature{ Path: "test", - Summary: "Microflow testing — run .test.mdl or .test.md files against a Mendix project in Docker", + Summary: "Microflow testing — run .test.mdl or .test.md files against a Mendix project (local warm loop, or Docker)", Keywords: []string{ "test", "testing", "microflow test", "nanoflow test", "test.mdl", "test.md", "junit", "docker", "@test", "@expect", "@throws", "@cleanup", + "watch", "attach", "test endpoint", "warm", }, Syntax: `mxcli test -p app.mpr [flags] Flags: -l, --list List tests without executing -j, --junit FILE Write JUnit XML results - -s, --skip-build Skip Docker build (reuse existing) + -s, --skip-build Skip the build (reuse existing deployment) + --local Run on mxcli's own runtime — no Docker daemon needed + -w, --watch With --local: keep the runtime warm and re-run on + every test or model change (Ctrl-C to stop) + --attach Run against an app already started with + 'mxcli run --local --test-endpoint' — no boot at all + --legacy-runner With --local: use the old after-startup runner -v, --verbose Show runtime log lines -t, --timeout DUR Runtime startup timeout (default: 5m) @@ -285,7 +292,17 @@ Annotations: @expect $var = value Assert variable equals value @expect $obj/Attr = val Assert entity attribute @throws 'message' Expect error - @cleanup rollback|none Cleanup strategy (default: rollback)`, + @cleanup rollback|none Cleanup strategy (default: rollback) + +How --local runs tests: one microflow per test, invoked by name over a +token-guarded HTTP endpoint the app registers at boot. A test that throws +fails only itself, and results are returned rather than scraped from the log. +Docker still uses the older after-startup runner. + +Cost of a run: + cold (--local) ~30s boots a runtime on its own ports + DB + warm (--local --watch) ~2s runtime stays up between runs + attached (--attach) ~2s no boot; uses the running app's database`, Example: `-- .test.mdl file format /** * @test String concatenation @@ -297,8 +314,14 @@ $result = CALL MICROFLOW MyModule.ConcatNames( / -- Run tests -mxcli test tests/ -p app.mpr -mxcli test tests/ -p app.mpr --junit results.xml`, +mxcli test tests/ -p app.mpr -- Docker +mxcli test tests/ -p app.mpr --local -- no Docker daemon +mxcli test tests/ -p app.mpr --local --watch -- warm loop, re-runs on change +mxcli test tests/ -p app.mpr --junit results.xml + +-- Or attach to an app you already have running: +mxcli run --local --test-endpoint -p app.mpr -- terminal 1 +mxcli test tests/ -p app.mpr --attach -- terminal 2`, }) // ── Errors ────────────────────────────────────────────────────────── diff --git a/cmd/mxcli/testrunner/client.go b/cmd/mxcli/testrunner/client.go new file mode 100644 index 000000000..27253f415 --- /dev/null +++ b/cmd/mxcli/testrunner/client.go @@ -0,0 +1,187 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "encoding/json" + "fmt" + "io" + "net" + "net/http" + "net/url" + "strings" + "time" +) + +// endpointClient talks to the test endpoint registered inside the running app. +type endpointClient struct { + baseURL string + token string + http *http.Client +} + +// newEndpointClient returns a client for the app serving on port. +// +// The transport deliberately takes no proxy: the address is always loopback, and +// an HTTP_PROXY in the environment (this is common in container and CI images) +// would otherwise send the token to the proxy. +func newEndpointClient(port int, token string) *endpointClient { + return &endpointClient{ + baseURL: fmt.Sprintf("http://127.0.0.1:%d/%s", port, endpointPath), + token: token, + http: &http.Client{ + Timeout: 10 * time.Minute, + Transport: &http.Transport{ + Proxy: nil, + DialContext: (&net.Dialer{Timeout: 5 * time.Second}).DialContext, + TLSHandshakeTimeout: 5 * time.Second, + }, + }, + } +} + +// runResponse is the endpoint's reply to a run request. +type runResponse struct { + MF string `json:"mf"` + OK bool `json:"ok"` + DurationMicros int64 `json:"durationMicros"` + Result string `json:"result"` + Error string `json:"error"` +} + +// listResponse is the endpoint's reply to a list request. +type listResponse struct { + Microflows []string `json:"microflows"` + Error string `json:"error"` +} + +// get performs one authenticated GET and decodes the JSON body into out. +func (c *endpointClient) get(route string, params url.Values, out any) error { + u := c.baseURL + route + if len(params) > 0 { + u += "?" + params.Encode() + } + req, err := http.NewRequest(http.MethodGet, u, nil) + if err != nil { + return err + } + req.Header.Set(endpointTokenHeader, c.token) + + resp, err := c.http.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + body, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20)) + if err != nil { + return fmt.Errorf("reading response: %w", err) + } + + // 401/403 mean the gate rejected us, which is a bug in how mxcli passed the + // token rather than anything to do with the tests. Say so plainly instead of + // letting it surface as an unmarshalling error. + switch resp.StatusCode { + case http.StatusUnauthorized: + return fmt.Errorf("test endpoint rejected the token (is another app serving port %s?)", portOf(c.baseURL)) + case http.StatusForbidden: + return fmt.Errorf("test endpoint refused the request: %s", strings.TrimSpace(string(body))) + } + + if err := json.Unmarshal(body, out); err != nil { + return fmt.Errorf("decoding response (HTTP %d): %w: %s", resp.StatusCode, err, truncate(string(body), 200)) + } + return nil +} + +// ping reports whether the endpoint is up and accepting our token. +func (c *endpointClient) ping() error { + var lr listResponse + return c.get("list", nil, &lr) +} + +// list returns the test microflows the running app knows about. +func (c *endpointClient) list() ([]string, error) { + var lr listResponse + if err := c.get("list", url.Values{"prefix": {testFlowPrefix}}, &lr); err != nil { + return nil, err + } + if lr.Error != "" { + return nil, fmt.Errorf("%s", lr.Error) + } + return lr.Microflows, nil +} + +// run executes one test microflow and returns the endpoint's reply. +func (c *endpointClient) run(mf string) (*runResponse, error) { + var rr runResponse + if err := c.get("run", url.Values{"mf": {mf}}, &rr); err != nil { + return nil, err + } + return &rr, nil +} + +// waitReady polls until the endpoint answers or the deadline passes. The runtime +// reports itself started before the after-startup action has necessarily +// finished registering the handler, so a first call can legitimately 404. +func (c *endpointClient) waitReady(timeout time.Duration) error { + deadline := time.Now().Add(timeout) + var last error + for { + if err := c.ping(); err == nil { + return nil + } else { + last = err + } + if time.Now().After(deadline) { + return fmt.Errorf("test endpoint did not come up within %s: %w", timeout, last) + } + time.Sleep(200 * time.Millisecond) + } +} + +// toResult maps an endpoint reply onto the test's result. +// +// Three outcomes are distinguished, and the distinction matters when reading a +// failing run: the test decided it failed (an assertion), the microflow threw +// (StatusError — the test did not reach a verdict), or the verdict came back in +// a shape this runner does not recognise. +func toResult(tc TestCase, rr *runResponse) TestResult { + res := TestResult{ + ID: tc.ID, + Name: tc.Name, + Duration: time.Duration(rr.DurationMicros) * time.Microsecond, + } + switch { + case !rr.OK: + res.Status = StatusError + res.Message = rr.Error + if res.Message == "" { + res.Message = "microflow threw, but the runtime reported no message" + } + case rr.Result == verdictPass: + res.Status = StatusPass + case strings.HasPrefix(rr.Result, verdictFailPrefix): + res.Status = StatusFail + res.Message = strings.TrimPrefix(rr.Result, verdictFailPrefix) + default: + res.Status = StatusError + res.Message = fmt.Sprintf("unrecognised verdict from the test microflow: %q", truncate(rr.Result, 200)) + } + return res +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] + "…" +} + +// portOf pulls the port back out of a base URL for error messages. +func portOf(baseURL string) string { + u, err := url.Parse(baseURL) + if err != nil { + return "?" + } + return u.Port() +} diff --git a/cmd/mxcli/testrunner/client_test.go b/cmd/mxcli/testrunner/client_test.go new file mode 100644 index 000000000..a95b660f1 --- /dev/null +++ b/cmd/mxcli/testrunner/client_test.go @@ -0,0 +1,217 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +// fakeEndpoint stands in for the handler the Java action registers, enforcing +// the same token gate. It lets the Go side of the contract — header name, +// routes, response shape, status codes — be tested without a Mendix runtime. +type fakeEndpoint struct { + token string + flows map[string]runResponse + // seenTokens records what each request presented, so a test can assert the + // client actually sends the token rather than the server merely allowing it. + seenTokens []string +} + +func (f *fakeEndpoint) handler() http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + presented := r.Header.Get(endpointTokenHeader) + f.seenTokens = append(f.seenTokens, presented) + w.Header().Set("Content-Type", "application/json") + + if presented != f.token { + w.WriteHeader(http.StatusUnauthorized) + fmt.Fprint(w, `{"error":"unauthorized"}`) + return + } + + switch { + case strings.HasSuffix(r.URL.Path, "/list"): + names := []string{} + for name := range f.flows { + if p := r.URL.Query().Get("prefix"); p == "" || strings.HasPrefix(name, p) { + names = append(names, name) + } + } + json.NewEncoder(w).Encode(listResponse{Microflows: names}) + case strings.HasSuffix(r.URL.Path, "/run"): + mf := r.URL.Query().Get("mf") + resp, ok := f.flows[mf] + if !ok { + w.WriteHeader(http.StatusNotFound) + fmt.Fprintf(w, `{"error":"unknown microflow","mf":%q}`, mf) + return + } + json.NewEncoder(w).Encode(resp) + default: + w.WriteHeader(http.StatusNotFound) + fmt.Fprint(w, `{"error":"no such route"}`) + } + }) +} + +// newFakeEndpoint starts the fake and returns a client pointed at it. +func newFakeEndpoint(t *testing.T, token string, flows map[string]runResponse) (*fakeEndpoint, *endpointClient) { + t.Helper() + fake := &fakeEndpoint{token: token, flows: flows} + srv := httptest.NewServer(fake.handler()) + t.Cleanup(srv.Close) + + c := newEndpointClient(0, token) + c.baseURL = srv.URL + "/" + endpointPath + return fake, c +} + +func TestClientSendsTheToken(t *testing.T) { + fake, c := newFakeEndpoint(t, "s3cret", map[string]runResponse{ + testFlowPrefix + "test_1": {OK: true, Result: verdictPass}, + }) + + if _, err := c.list(); err != nil { + t.Fatalf("list: %v", err) + } + if len(fake.seenTokens) != 1 || fake.seenTokens[0] != "s3cret" { + t.Errorf("server saw tokens %q, want one request presenting %q", fake.seenTokens, "s3cret") + } +} + +// TestClientReportsAnUnauthorizedGateClearly pins that a rejected token is +// reported as a token problem, not as a JSON decoding failure — the gate +// rejecting mxcli is a bug in how the token was passed, and the message has to +// say so. +func TestClientReportsAnUnauthorizedGateClearly(t *testing.T) { + _, c := newFakeEndpoint(t, "the-real-token", nil) + c.token = "the-wrong-token" + + _, err := c.list() + if err == nil { + t.Fatal("list with a wrong token succeeded") + } + if !strings.Contains(err.Error(), "rejected the token") { + t.Errorf("error %q does not explain that the token was rejected", err) + } +} + +func TestClientListFiltersToTestFlows(t *testing.T) { + _, c := newFakeEndpoint(t, "t", map[string]runResponse{ + testFlowPrefix + "test_1": {}, + "MyModule.SomethingElse": {}, + testFlowPrefix + "test_222": {}, + }) + + names, err := c.list() + if err != nil { + t.Fatalf("list: %v", err) + } + for _, n := range names { + if !strings.HasPrefix(n, testFlowPrefix) { + t.Errorf("list returned a non-test microflow: %q", n) + } + } + if len(names) != 2 { + t.Errorf("got %d test microflows, want 2: %q", len(names), names) + } +} + +func TestToResult(t *testing.T) { + tc := TestCase{ID: "test_1", Name: "a test"} + + tests := []struct { + name string + resp runResponse + wantStatus TestStatus + wantMsg string + }{ + { + name: "pass", + resp: runResponse{OK: true, Result: verdictPass, DurationMicros: 1500}, + wantStatus: StatusPass, + }, + { + name: "assertion failure is a FAIL", + resp: runResponse{OK: true, Result: verdictFailPrefix + "expected $r = 'x'"}, + wantStatus: StatusFail, + wantMsg: "expected $r = 'x'", + }, + { + name: "a thrown microflow is an ERROR, not a FAIL", + resp: runResponse{OK: false, Error: "NullPointerException"}, + wantStatus: StatusError, + wantMsg: "NullPointerException", + }, + { + name: "a throw with no message still says something", + resp: runResponse{OK: false}, + wantStatus: StatusError, + wantMsg: "microflow threw, but the runtime reported no message", + }, + { + name: "an unrecognised verdict is an ERROR", + resp: runResponse{OK: true, Result: "who knows"}, + wantStatus: StatusError, + wantMsg: `unrecognised verdict from the test microflow: "who knows"`, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := toResult(tc, &tt.resp) + if got.Status != tt.wantStatus { + t.Errorf("status = %v, want %v", got.Status, tt.wantStatus) + } + if got.Message != tt.wantMsg { + t.Errorf("message = %q, want %q", got.Message, tt.wantMsg) + } + if got.ID != tc.ID || got.Name != tc.Name { + t.Errorf("identity not carried over: got %q/%q", got.ID, got.Name) + } + }) + } +} + +func TestToResultCarriesDuration(t *testing.T) { + got := toResult(TestCase{ID: "test_1"}, &runResponse{OK: true, Result: verdictPass, DurationMicros: 2500}) + if got.Duration != 2500*time.Microsecond { + t.Errorf("duration = %v, want 2.5ms", got.Duration) + } +} + +// TestClientUsesNoProxy pins that the loopback call cannot be diverted. An +// HTTP_PROXY in the environment is common in container and CI images, and would +// otherwise send the token to the proxy. +func TestClientUsesNoProxy(t *testing.T) { + c := newEndpointClient(8081, "tok") + tr, ok := c.http.Transport.(*http.Transport) + if !ok { + t.Fatalf("transport is %T, want *http.Transport", c.http.Transport) + } + if tr.Proxy != nil { + t.Error("the client honours a proxy; the token could leave the machine") + } +} + +func TestWaitReadyGivesUp(t *testing.T) { + // Port 1 on loopback: nothing listens, and connections fail fast. + c := newEndpointClient(1, "tok") + start := time.Now() + err := c.waitReady(600 * time.Millisecond) + if err == nil { + t.Fatal("waitReady succeeded against a dead port") + } + if elapsed := time.Since(start); elapsed > 10*time.Second { + t.Errorf("waitReady took %v; it should honour its timeout", elapsed) + } + if !strings.Contains(err.Error(), "did not come up") { + t.Errorf("error %q does not explain the endpoint never came up", err) + } +} diff --git a/cmd/mxcli/testrunner/endpoint.go b/cmd/mxcli/testrunner/endpoint.go new file mode 100644 index 000000000..86a2fea14 --- /dev/null +++ b/cmd/mxcli/testrunner/endpoint.go @@ -0,0 +1,265 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "crypto/rand" + "encoding/hex" + "fmt" + "strings" +) + +const ( + // endpointPath is the path the request handler is registered at. Mendix + // matches on the leading segment, so the trailing slash is part of the name. + endpointPath = "mxtest/" + // endpointTokenEnv is the environment variable the runtime JVM reads its + // per-run token from. It is passed via the process environment and never + // written into the project, so a cleanup that fails cannot leave a working + // credential behind in javasource/. + endpointTokenEnv = "MXCLI_TEST_TOKEN" + // endpointTokenHeader carries the token on each request. + endpointTokenHeader = "X-MxTest-Token" + + // endpointRegisterAction is the Java action that registers the handler. + endpointRegisterAction = mxTestModule + ".RegisterTestEndpoint" + // endpointStartupFlow is the after-startup microflow that calls it. It only + // registers the endpoint — unlike the log-scraping runner it replaces, no test + // ever executes during startup. + endpointStartupFlow = mxTestModule + ".RegisterEndpoint" + // testFlowPrefix prefixes every generated per-test microflow. + testFlowPrefix = mxTestModule + ".Test_" +) + +// newEndpointToken returns a fresh 256-bit token as hex. +func newEndpointToken() (string, error) { + var b [32]byte + if _, err := rand.Read(b[:]); err != nil { + return "", fmt.Errorf("generating endpoint token: %w", err) + } + return hex.EncodeToString(b[:]), nil +} + +// testFlowName is the microflow generated for a test case. +func testFlowName(tc TestCase) string { return testFlowPrefix + tc.ID } + +// GenerateEndpointMDL returns the MDL that installs the test endpoint: a Java +// action registering a request handler, plus the after-startup microflow that +// calls it once at boot. +// +// The handler is generic — it resolves microflows by name from +// Core.getMicroflowNames() at request time — so this MDL does not mention any +// test and never has to be regenerated when tests change. That is what lets a +// re-run be an HTTP call instead of a restart. +// +// Three properties make an endpoint that executes arbitrary microflows under a +// system context safe enough to install in a developer's project: +// +// 1. It fails closed. With no token in the environment the handler is not +// registered at all, so a project whose cleanup failed and still carries the +// MxTest module exposes nothing when deployed anywhere else. +// 2. Every request must present the token, compared with a length-independent +// constant-time equality so a wrong guess leaks no timing signal. +// 3. Non-loopback callers are refused outright. mxcli always talks to +// 127.0.0.1; nothing legitimate reaches this handler from off-box. +// +// chainAfterStartup, when non-empty, is a microflow the generated startup flow +// calls after registering the endpoint — the project's own after-startup +// microflow, which this one displaces. +// +// The test runner passes "" : a test run wants a known starting state, and the +// suite is the only thing that should execute. A dev loop hosting the endpoint +// (`run --local --test-endpoint`) passes the real one, because the developer's +// app must still seed its data and do whatever else it does at boot. +func GenerateEndpointMDL(chainAfterStartup string) string { + var b strings.Builder + + b.WriteString("CREATE MODULE " + mxTestModule + ";\n\n") + b.WriteString("/** Registers the mxcli test endpoint. Called once at startup. */\n") + b.WriteString("CREATE OR REPLACE JAVA ACTION " + endpointRegisterAction + "() RETURNS Boolean\n") + b.WriteString("AS $$\n") + b.WriteString(endpointJava) + b.WriteString("\n$$;\n/\n\n") + + b.WriteString("/** Registers the mxcli test endpoint at boot. Runs no tests. */\n") + b.WriteString("CREATE OR REPLACE MICROFLOW " + endpointStartupFlow + " ()\n") + b.WriteString("RETURNS Boolean AS $Registered\n") + b.WriteString("BEGIN\n") + b.WriteString(" $Registered = CALL JAVA ACTION " + endpointRegisterAction + "();\n") + if chainAfterStartup != "" { + // Register first, then hand over: if the project's own startup microflow + // fails, the endpoint is already up and the failure is diagnosable over + // HTTP instead of only in the log. + b.WriteString(" $Chained = CALL MICROFLOW " + chainAfterStartup + "();\n") + } + b.WriteString(" RETURN $Registered;\n") + b.WriteString("END;\n") + b.WriteString("/\n") + + return b.String() +} + +// endpointJava is the body of the registration Java action. It is a constant, +// not a template: nothing about a particular run is interpolated into it, and in +// particular the token is read from the environment rather than baked in. +// +// Fully-qualified type names throughout — the generated .java file's import list +// is fixed by the Java-action scaffold and cannot be extended from MDL. +const endpointJava = `final com.mendix.logging.ILogNode log = com.mendix.core.Core.getLogger("MxTest"); + +// Fail closed: no token in the environment means this is not an mxcli test run, +// so the endpoint is never exposed. A project that kept the MxTest module +// through a failed cleanup is inert everywhere else, including production. +final String expectedToken = System.getenv("` + endpointTokenEnv + `"); +if (expectedToken == null || expectedToken.isEmpty()) { + log.info("MxTest: no ` + endpointTokenEnv + ` in the environment; test endpoint NOT registered"); + return true; +} + +com.mendix.core.Core.addRequestHandler("` + endpointPath + `", new com.mendix.externalinterface.connector.RequestHandler() { + + private String esc(String s) { + if (s == null) return "null"; + StringBuilder b = new StringBuilder("\""); + for (int i = 0; i < s.length(); i++) { + char c = s.charAt(i); + switch (c) { + case '"': b.append("\\\""); break; + case '\\': b.append("\\\\"); break; + case '\n': b.append("\\n"); break; + case '\r': b.append("\\r"); break; + case '\t': b.append("\\t"); break; + default: + if (c < 0x20) b.append(String.format("\\u%04x", (int) c)); + else b.append(c); + } + } + return b.append('"').toString(); + } + + // Constant-time comparison. String.equals returns early on the first + // differing byte; MessageDigest.isEqual does not, and is also safe when the + // lengths differ. + private boolean tokenOK(String presented) { + if (presented == null) return false; + return java.security.MessageDigest.isEqual( + presented.getBytes(java.nio.charset.StandardCharsets.UTF_8), + expectedToken.getBytes(java.nio.charset.StandardCharsets.UTF_8)); + } + + private boolean isLoopback(String addr) { + if (addr == null || addr.isEmpty()) return false; + try { + return java.net.InetAddress.getByName(addr).isLoopbackAddress(); + } catch (java.net.UnknownHostException e) { + return false; + } + } + + @Override + protected void processRequest(com.mendix.m2ee.api.IMxRuntimeRequest request, + com.mendix.m2ee.api.IMxRuntimeResponse response, + String path) throws Exception { + response.setContentType("application/json"); + java.io.Writer out = response.getWriter(); + + // mxcli always calls 127.0.0.1. Anything else is not a test run. + if (!isLoopback(request.getRemoteAddr())) { + log.warn("MxTest: refused non-loopback request from " + request.getRemoteAddr()); + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.FORBIDDEN); + out.write("{\"error\":\"forbidden\"}"); + out.flush(); + return; + } + if (!tokenOK(request.getHeader("` + endpointTokenHeader + `"))) { + log.warn("MxTest: refused request with a missing or incorrect token"); + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.UNAUTHORIZED); + out.write("{\"error\":\"unauthorized\"}"); + out.flush(); + return; + } + + java.util.Set known = com.mendix.core.Core.getMicroflowNames(); + + if ("list".equals(path)) { + // Never widen past the test namespace. An unfiltered list would hand + // back the app's entire microflow inventory, which this endpoint has + // no business disclosing — it will not run those microflows either. + // A caller-supplied prefix can only narrow further. + String prefix = request.getParameter("prefix"); + if (prefix == null || !prefix.startsWith("` + testFlowPrefix + `")) { + prefix = "` + testFlowPrefix + `"; + } + java.util.List names = new java.util.ArrayList(); + for (String n : known) { + if (n.startsWith(prefix)) names.add(n); + } + java.util.Collections.sort(names); + StringBuilder b = new StringBuilder("{\"microflows\":["); + for (int i = 0; i < names.size(); i++) { + if (i > 0) b.append(','); + b.append(esc(names.get(i))); + } + b.append("]}"); + out.write(b.toString()); + out.flush(); + return; + } + + if (!"run".equals(path)) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.NOT_FOUND); + out.write("{\"error\":\"no such route\",\"path\":" + esc(path) + "}"); + out.flush(); + return; + } + + String mf = request.getParameter("mf"); + if (mf == null || mf.isEmpty()) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.BAD_REQUEST); + out.write("{\"error\":\"missing mf parameter\"}"); + out.flush(); + return; + } + // Only ever run a microflow this runner generated. Even behind the token + // this handler should not be a way to invoke the rest of the app. + if (!mf.startsWith("` + testFlowPrefix + `")) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.FORBIDDEN); + out.write("{\"error\":\"not a test microflow\",\"mf\":" + esc(mf) + "}"); + out.flush(); + return; + } + if (!known.contains(mf)) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.NOT_FOUND); + out.write("{\"error\":\"unknown microflow\",\"mf\":" + esc(mf) + "}"); + out.flush(); + return; + } + + long t0 = System.nanoTime(); + com.mendix.systemwideinterfaces.core.IContext ctx = com.mendix.core.Core.createSystemContext(); + Object result = null; + String error = null; + try { + result = com.mendix.core.Core.microflowCall(mf).execute(ctx); + } catch (Throwable t) { + Throwable root = t; + while (root.getCause() != null && root.getCause() != root) root = root.getCause(); + String msg = root.getMessage(); + error = (msg == null || msg.isEmpty()) ? root.getClass().getName() : msg; + } + long micros = (System.nanoTime() - t0) / 1000L; + + StringBuilder b = new StringBuilder("{"); + b.append("\"mf\":").append(esc(mf)); + b.append(",\"ok\":").append(error == null); + b.append(",\"durationMicros\":").append(micros); + b.append(",\"result\":").append(result == null ? "null" : esc(String.valueOf(result))); + if (error != null) b.append(",\"error\":").append(esc(error)); + b.append('}'); + out.write(b.toString()); + out.flush(); + } +}); + +log.info("MxTest: test endpoint registered at /` + endpointPath + `"); +return true;` diff --git a/cmd/mxcli/testrunner/endpoint_test.go b/cmd/mxcli/testrunner/endpoint_test.go new file mode 100644 index 000000000..7e8f3ff8e --- /dev/null +++ b/cmd/mxcli/testrunner/endpoint_test.go @@ -0,0 +1,167 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "strings" + "testing" +) + +func TestNewEndpointTokenIsUniqueAndLongEnough(t *testing.T) { + seen := make(map[string]bool) + for i := 0; i < 100; i++ { + tok, err := newEndpointToken() + if err != nil { + t.Fatalf("newEndpointToken: %v", err) + } + if len(tok) != 64 { + t.Fatalf("token %q is %d hex chars, want 64 (256 bits)", tok, len(tok)) + } + if seen[tok] { + t.Fatalf("newEndpointToken returned a duplicate: %q", tok) + } + seen[tok] = true + } +} + +// TestEndpointJavaFailsClosed pins the property that makes it safe to leave the +// MxTest module in a project: with no token in the environment the handler is +// never registered, so there is nothing to reach. +func TestEndpointJavaFailsClosed(t *testing.T) { + guard := strings.Index(endpointJava, `System.getenv("`+endpointTokenEnv+`")`) + if guard < 0 { + t.Fatal("the handler does not read the token from the environment") + } + register := strings.Index(endpointJava, "Core.addRequestHandler") + if register < 0 { + t.Fatal("the handler is never registered") + } + if guard > register { + t.Error("the token is read after the handler is registered; it must gate registration") + } + + // The early return between them is what makes the guard load-bearing. + between := endpointJava[guard:register] + if !strings.Contains(between, "return true;") { + t.Error("no early return between reading the token and registering: an empty token would still register the handler") + } +} + +// TestEndpointJavaNeverEmbedsASecret pins that the token reaches the runtime +// through the environment only. Interpolating it into the generated Java would +// write a live credential into the user's javasource/ tree, where a failed +// cleanup leaves it behind. +func TestEndpointJavaNeverEmbedsASecret(t *testing.T) { + tok, err := newEndpointToken() + if err != nil { + t.Fatalf("newEndpointToken: %v", err) + } + mdl := GenerateEndpointMDL("") + if strings.Contains(mdl, tok) { + t.Fatal("the generated MDL contains the token") + } + // GenerateEndpointMDL takes no token argument at all, so the only way one + // could appear is via the environment read. + if !strings.Contains(mdl, endpointTokenEnv) { + t.Errorf("the generated MDL does not reference %s", endpointTokenEnv) + } +} + +func TestEndpointJavaChecksTheToken(t *testing.T) { + // Match the return statement, not the word: an earlier version of this test + // looked for "MessageDigest.isEqual" anywhere in the source and was satisfied + // by the comment above the method, so it stayed green when the body was + // swapped for String.equals. + if !strings.Contains(endpointJava, "return java.security.MessageDigest.isEqual(") { + t.Error("token comparison is not constant-time (use MessageDigest.isEqual, not String.equals)") + } + if strings.Contains(endpointJava, "presented.equals(") { + t.Error("the token is compared with String.equals, which returns early on the first differing byte") + } + if !strings.Contains(endpointJava, endpointTokenHeader) { + t.Errorf("the handler does not read the %s header", endpointTokenHeader) + } + if !strings.Contains(endpointJava, "return java.net.InetAddress.getByName(addr).isLoopbackAddress();") { + t.Error("the handler does not refuse non-loopback callers") + } +} + +// TestEndpointJavaOnlyRunsTestMicroflows pins that the endpoint is not a general +// microflow-invocation API even for a caller holding the token. +func TestEndpointJavaOnlyRunsTestMicroflows(t *testing.T) { + if !strings.Contains(endpointJava, `mf.startsWith("`+testFlowPrefix+`")`) { + t.Errorf("the handler does not restrict execution to %s* microflows", testFlowPrefix) + } +} + +// TestEndpointListCannotEnumerateTheApp pins that /list is clamped to the test +// namespace. Found by probing the live runtime: an absent prefix returned every +// microflow in the app, Administration.* included. The endpoint will not run +// those, so it must not disclose them either. +func TestEndpointListCannotEnumerateTheApp(t *testing.T) { + if !strings.Contains(endpointJava, `!prefix.startsWith("`+testFlowPrefix+`")`) { + t.Error("a caller-supplied prefix is not clamped to the test namespace") + } + if strings.Contains(endpointJava, "if (prefix == null || n.startsWith(prefix))") { + t.Error("a null prefix still lists every microflow in the app") + } +} + +// TestEndpointRejectsBeforeItActs pins the ordering of the two gates: both the +// loopback check and the token check must precede any use of the request. +func TestEndpointRejectsBeforeItActs(t *testing.T) { + loopback := strings.Index(endpointJava, "if (!isLoopback(") + token := strings.Index(endpointJava, "if (!tokenOK(") + execute := strings.Index(endpointJava, "Core.microflowCall(mf).execute") + names := strings.Index(endpointJava, "Core.getMicroflowNames()") + + for _, tc := range []struct { + name string + gate, work int + }{ + {"loopback check precedes listing microflow names", loopback, names}, + {"token check precedes listing microflow names", token, names}, + {"loopback check precedes execution", loopback, execute}, + {"token check precedes execution", token, execute}, + } { + if tc.gate < 0 || tc.work < 0 { + t.Fatalf("%s: a landmark is missing (gate=%d work=%d)", tc.name, tc.gate, tc.work) + } + if tc.gate > tc.work { + t.Errorf("%s: gate at %d comes after the work at %d", tc.name, tc.gate, tc.work) + } + } +} + +func TestGenerateEndpointMDLShape(t *testing.T) { + mdl := GenerateEndpointMDL("") + for _, want := range []string{ + "CREATE MODULE " + mxTestModule + ";", + "CREATE OR REPLACE JAVA ACTION " + endpointRegisterAction + "() RETURNS Boolean", + "CREATE OR REPLACE MICROFLOW " + endpointStartupFlow + " ()", + "RETURNS Boolean AS $Registered", + } { + if !strings.Contains(mdl, want) { + t.Errorf("generated MDL is missing %q", want) + } + } + // The startup microflow must return Boolean or Mendix fails the build with + // CE0142 on a void after-startup microflow. + if !strings.Contains(mdl, "RETURN $Registered;") { + t.Error("the startup microflow does not return a Boolean (CE0142)") + } +} + +// TestGenerateEndpointMDLIsTestIndependent pins the property the whole design +// rests on: the endpoint MDL does not mention any test, so it never has to be +// regenerated when tests change. +func TestGenerateEndpointMDLIsTestIndependent(t *testing.T) { + a := GenerateEndpointMDL("") + b := GenerateEndpointMDL("") + if a != b { + t.Fatal("GenerateEndpointMDL is not deterministic") + } + if strings.Contains(a, "test_1") { + t.Error("the endpoint MDL references a specific test") + } +} diff --git a/cmd/mxcli/testrunner/generator_endpoint.go b/cmd/mxcli/testrunner/generator_endpoint.go new file mode 100644 index 000000000..92e649ca3 --- /dev/null +++ b/cmd/mxcli/testrunner/generator_endpoint.go @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "fmt" + "strings" +) + +// Verdict protocol. A test microflow returns one string: either verdictPass, or +// verdictFailPrefix followed by the reason. The endpoint hands that string back +// as the HTTP response's "result" field, so a result is a returned value rather +// than something recovered from the runtime log. +const ( + verdictPass = "PASS" + verdictFailPrefix = "FAIL:" +) + +// GenerateTestFlows returns the MDL declaring one microflow per test case. +// +// This is the endpoint path's counterpart to GenerateTestRunner, which compiles +// the whole suite into a single after-startup microflow. One microflow per test +// buys three things that the monolith cannot give: +// +// - Each test can be invoked, re-invoked, or skipped on its own, so --filter +// and single-test runs are a matter of which URL is called. +// - A test that throws fails only itself. In the monolith an uncaught error +// ends the whole flow, and because that flow is the after-startup action it +// also fails the boot. +// - Every test gets its own variable scope, so the suffix-renaming the +// monolith needs to keep `$result` in test 1 from colliding with `$result` +// in test 2 is simply not required here. +func GenerateTestFlows(suite *TestSuite) string { + var b strings.Builder + b.WriteString("CREATE MODULE " + mxTestModule + ";\n\n") + for _, tc := range suite.Tests { + writeTestFlow(&b, tc) + b.WriteString("\n") + } + return b.String() +} + +// writeTestFlow writes one test's microflow. +func writeTestFlow(b *strings.Builder, tc TestCase) { + fmt.Fprintf(b, "/** %s */\n", escapeMDLComment(tc.Name)) + fmt.Fprintf(b, "CREATE OR REPLACE MICROFLOW %s ()\n", testFlowName(tc)) + b.WriteString("RETURNS String AS $Verdict\n") + b.WriteString("BEGIN\n") + fmt.Fprintf(b, " DECLARE $Verdict String = '%s';\n", verdictPass) + + if tc.Throws != "" { + writeThrowsFlowBody(b, tc) + } else { + writeExpectFlowBody(b, tc) + } + + b.WriteString(" RETURN $Verdict;\n") + b.WriteString("END;\n") + b.WriteString("/\n") +} + +// writeExpectFlowBody writes the body of a normal test: run the MDL, then check +// each @expect. An error during the body short-circuits to a FAIL verdict. +func writeExpectFlowBody(b *strings.Builder, tc TestCase) { + for _, line := range rewriteBodyForVerdict(strings.Split(tc.MDL, "\n"), tc) { + b.WriteString(" ") + b.WriteString(line) + b.WriteString("\n") + } + for _, exp := range tc.Expects { + writeExpectCheck(b, exp) + } +} + +// writeThrowsFlowBody writes the body of an @throws test: the verdict starts as +// a failure and only the error handler can clear it, so a body that completes +// without throwing fails — which is the point of the annotation. +func writeThrowsFlowBody(b *strings.Builder, tc TestCase) { + fmt.Fprintf(b, " SET $Verdict = '%s';\n", + escapeMDLString(verdictFailPrefix+"expected an exception but none was thrown")) + for _, line := range rewriteBodyForThrows(strings.Split(tc.MDL, "\n")) { + b.WriteString(" ") + b.WriteString(line) + b.WriteString("\n") + } +} + +// writeExpectCheck writes one @expect assertion. +// +// Only the pass condition is expressed with `=`; a `<>` expectation is compiled +// as the same equality with the branches swapped. That is deliberate and +// inherited from the monolithic generator: `<>` in a generated Mendix expression +// produced expression errors, so the operator never reaches the model. +func writeExpectCheck(b *strings.Builder, exp Expect) { + equal := fmt.Sprintf("%s = %s", exp.Variable, exp.Value) + failMsg := escapeMDLString(fmt.Sprintf("%sexpected %s %s %s", + verdictFailPrefix, exp.Variable, exp.Operator, exp.Value)) + + // An earlier statement may already have failed the test; never overwrite an + // existing failure with a later assertion's result. + fmt.Fprintf(b, " IF $Verdict = '%s' THEN\n", verdictPass) + if exp.Operator == "<>" { + fmt.Fprintf(b, " IF %s THEN\n", equal) + fmt.Fprintf(b, " SET $Verdict = '%s';\n", failMsg) + b.WriteString(" END IF;\n") + } else { + fmt.Fprintf(b, " IF %s THEN\n", equal) + b.WriteString(" ELSE\n") + fmt.Fprintf(b, " SET $Verdict = '%s';\n", failMsg) + b.WriteString(" END IF;\n") + } + b.WriteString(" END IF;\n") +} + +// rewriteBodyForVerdict attaches an ON ERROR handler to every CALL in the test +// body, turning a thrown error into a FAIL verdict and an early return. +func rewriteBodyForVerdict(lines []string, tc TestCase) []string { + handler := []string{ + fmt.Sprintf(" SET $Verdict = '%s';", + escapeMDLString(verdictFailPrefix+"exception during execution")), + " RETURN $Verdict;", + } + return attachOnError(lines, handler) +} + +// rewriteBodyForThrows attaches an ON ERROR handler that clears the pre-set +// failure verdict — the error is the expected outcome. +func rewriteBodyForThrows(lines []string) []string { + handler := []string{fmt.Sprintf(" SET $Verdict = '%s';", verdictPass)} + return attachOnError(lines, handler) +} + +// attachOnError appends `ON ERROR { ... }` to each CALL statement in the body, +// joining a statement that spans several lines first. +func attachOnError(lines, handler []string) []string { + var out []string + for i := 0; i < len(lines); i++ { + trimmed := strings.TrimSpace(lines[i]) + if !containsCallMicroflow(trimmed) { + out = append(out, lines[i]) + continue + } + + stmt := lines[i] + for !strings.HasSuffix(strings.TrimSpace(stmt), ";") && i+1 < len(lines) { + i++ + stmt += "\n" + lines[i] + } + stmt = strings.TrimSuffix(strings.TrimSpace(stmt), ";") + + out = append(out, stmt+" ON ERROR {") + out = append(out, handler...) + out = append(out, "};") + } + return out +} + +// escapeMDLComment keeps a test name from closing the javadoc block it sits in. +func escapeMDLComment(s string) string { + return strings.ReplaceAll(s, "*/", "* /") +} diff --git a/cmd/mxcli/testrunner/generator_endpoint_test.go b/cmd/mxcli/testrunner/generator_endpoint_test.go new file mode 100644 index 000000000..dedfd9ba7 --- /dev/null +++ b/cmd/mxcli/testrunner/generator_endpoint_test.go @@ -0,0 +1,225 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "strings" + "testing" +) + +func TestGenerateTestFlowsOneMicroflowPerTest(t *testing.T) { + suite := &TestSuite{ + Name: "suite", + Tests: []TestCase{ + {ID: "test_1", Name: "first", MDL: "$r = CALL MICROFLOW Mod.A();"}, + {ID: "test_2", Name: "second", MDL: "$r = CALL MICROFLOW Mod.B();"}, + }, + } + mdl := GenerateTestFlows(suite) + + for _, want := range []string{ + "CREATE OR REPLACE MICROFLOW MxTest.Test_test_1 ()", + "CREATE OR REPLACE MICROFLOW MxTest.Test_test_2 ()", + } { + if !strings.Contains(mdl, want) { + t.Errorf("generated MDL is missing %q", want) + } + } + if n := strings.Count(mdl, "CREATE OR REPLACE MICROFLOW"); n != 2 { + t.Errorf("got %d microflows, want one per test (2)", n) + } +} + +// TestGenerateTestFlowsNoVariableRenaming pins the simplification that per-test +// microflows buy. The monolithic runner has to suffix every variable to keep +// test 1's $result apart from test 2's; separate microflows have separate +// scopes, so the same name in two tests must survive unmangled. +func TestGenerateTestFlowsNoVariableRenaming(t *testing.T) { + suite := &TestSuite{ + Tests: []TestCase{ + {ID: "test_1", Name: "a", MDL: "$result = CALL MICROFLOW Mod.A();", + Expects: []Expect{{Variable: "$result", Operator: "=", Value: "'x'"}}}, + {ID: "test_2", Name: "b", MDL: "$result = CALL MICROFLOW Mod.B();", + Expects: []Expect{{Variable: "$result", Operator: "=", Value: "'y'"}}}, + }, + } + mdl := GenerateTestFlows(suite) + + if strings.Contains(mdl, "$result_1") || strings.Contains(mdl, "$result_2") { + t.Error("variables were suffix-renamed; per-test microflows have their own scope") + } + if n := strings.Count(mdl, "$result"); n < 4 { + t.Errorf("expected $result to survive in both tests, found %d references", n) + } +} + +func TestGenerateTestFlowsExpectAssertion(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "equality", MDL: "$r = CALL MICROFLOW Mod.A();", + Expects: []Expect{{Variable: "$r", Operator: "=", Value: "'John'"}}, + }}} + mdl := GenerateTestFlows(suite) + + if !strings.Contains(mdl, "IF $r = 'John' THEN") { + t.Errorf("missing the equality check:\n%s", mdl) + } + if !strings.Contains(mdl, verdictFailPrefix+"expected $r = ''John''") { + t.Errorf("missing the failure verdict with the expected value:\n%s", mdl) + } +} + +// TestGenerateTestFlowsNotEqualIsCompiledAsEquality pins the inherited +// constraint: `<>` produced Mendix expression errors, so it must never reach the +// model — a <> expectation is the same equality with the branches swapped. +func TestGenerateTestFlowsNotEqualIsCompiledAsEquality(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "inequality", MDL: "$r = CALL MICROFLOW Mod.A();", + Expects: []Expect{{Variable: "$r", Operator: "<>", Value: "'John'"}}, + }}} + mdl := GenerateTestFlows(suite) + + if strings.Contains(mdl, "$r <> 'John'") { + t.Error("the <> operator reached the generated Mendix expression") + } + if !strings.Contains(mdl, "IF $r = 'John' THEN") { + t.Errorf("<> was not compiled as a swapped equality:\n%s", mdl) + } +} + +func TestGenerateTestFlowsWrapsCallsWithErrorHandling(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "throwing", MDL: "$r = CALL MICROFLOW Mod.A();", + }}} + mdl := GenerateTestFlows(suite) + + if !strings.Contains(mdl, "ON ERROR {") { + t.Errorf("the CALL was not wrapped in ON ERROR:\n%s", mdl) + } + if !strings.Contains(mdl, verdictFailPrefix+"exception during execution") { + t.Errorf("the error handler does not set a FAIL verdict:\n%s", mdl) + } +} + +// TestGenerateTestFlowsThrowsTestStartsFailed pins that an @throws test whose +// body completes normally fails: the verdict is pre-set to a failure and only +// the error handler clears it. +func TestGenerateTestFlowsThrowsTestStartsFailed(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "expects a throw", MDL: "$r = CALL MICROFLOW Mod.A();", + Throws: "boom", + }}} + mdl := GenerateTestFlows(suite) + + failIdx := strings.Index(mdl, verdictFailPrefix+"expected an exception") + handlerIdx := strings.Index(mdl, "ON ERROR {") + if failIdx < 0 { + t.Fatalf("no pre-set failure verdict:\n%s", mdl) + } + if handlerIdx < 0 { + t.Fatalf("no error handler:\n%s", mdl) + } + if failIdx > handlerIdx { + t.Error("the failure verdict is set after the handler; a non-throwing body would pass") + } + if !strings.Contains(mdl[handlerIdx:], "SET $Verdict = '"+verdictPass+"';") { + t.Error("the error handler does not clear the failure verdict") + } +} + +func TestGenerateTestFlowsMultiLineCall(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "multiline", + MDL: "$r = CALL MICROFLOW Mod.A(\n FirstName = 'John',\n LastName = 'Doe'\n);", + }}} + mdl := GenerateTestFlows(suite) + + if !strings.Contains(mdl, ") ON ERROR {") { + t.Errorf("a statement spanning lines was not joined before ON ERROR was attached:\n%s", mdl) + } + if strings.Count(mdl, "ON ERROR {") != 1 { + t.Errorf("expected exactly one handler for one call:\n%s", mdl) + } +} + +// TestGenerateTestFlowsEscapesNameInComment pins that a test name cannot close +// the javadoc block it is written into and break the generated MDL. +func TestGenerateTestFlowsEscapesNameInComment(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "ends the comment */ CREATE MODULE Evil;", MDL: "", + }}} + mdl := GenerateTestFlows(suite) + + head := mdl[:strings.Index(mdl, "CREATE OR REPLACE MICROFLOW")] + if strings.Count(head, "*/") != 1 { + t.Errorf("the test name closed the javadoc block early:\n%s", head) + } +} + +func TestEndpointCleanupCommands(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}}} + + tests := []struct { + name string + state projectState + present bool + want []string + }{ + { + name: "drops the whole module when the runner created it", + state: projectState{afterStartup: "Mod.ASU", createdMxTest: true}, + present: true, + want: []string{ + "ALTER SETTINGS MODEL AfterStartupMicroflow = 'Mod.ASU'", + "DROP MODULE MxTest", + }, + }, + { + name: "drops only the generated documents from a user's module", + state: projectState{createdMxTest: false}, + present: true, + want: []string{ + "ALTER SETTINGS MODEL AfterStartupMicroflow = ''", + "DROP MICROFLOW MxTest.Test_test_1", + "DROP MICROFLOW MxTest.Test_test_2", + "DROP MICROFLOW " + endpointStartupFlow, + "DROP JAVA ACTION " + endpointRegisterAction, + }, + }, + { + name: "drops nothing when the module never landed", + state: projectState{afterStartup: "Mod.ASU", createdMxTest: true}, + present: false, + want: []string{"ALTER SETTINGS MODEL AfterStartupMicroflow = 'Mod.ASU'"}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := endpointCleanupCommands(tc.state, suite, tc.present) + if len(got) != len(tc.want) { + t.Fatalf("got %d commands %q, want %d %q", len(got), got, len(tc.want), tc.want) + } + for i := range got { + if got[i] != tc.want[i] { + t.Errorf("command %d: got %q, want %q", i, got[i], tc.want[i]) + } + } + }) + } +} + +// TestEndpointCleanupRestoreIsAlwaysFirst pins the ordering: after-startup must +// stop pointing at the startup microflow before that microflow is dropped. +func TestEndpointCleanupRestoreIsAlwaysFirst(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ID: "test_1"}}} + for _, st := range []projectState{ + {createdMxTest: true}, + {createdMxTest: false}, + {afterStartup: "Mod.ASU", createdMxTest: true}, + } { + cmds := endpointCleanupCommands(st, suite, true) + if !strings.HasPrefix(cmds[0], "ALTER SETTINGS MODEL AfterStartupMicroflow") { + t.Errorf("state %+v: first command is %q, want the after-startup restore", st, cmds[0]) + } + } +} diff --git a/cmd/mxcli/testrunner/handshake.go b/cmd/mxcli/testrunner/handshake.go new file mode 100644 index 000000000..e46f68683 --- /dev/null +++ b/cmd/mxcli/testrunner/handshake.go @@ -0,0 +1,134 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "syscall" + "time" +) + +// syscallSignalZero is signal 0: delivered to no one, but still performs the +// process-exists and permission checks. The idiom for "is this pid alive?". +const syscallSignalZero = syscall.Signal(0) + +// HandshakeFile is where a `run --local --test-endpoint` session publishes what +// `mxcli test --attach` needs to reach it. It lives beside the project rather +// than in a shared temp directory so two projects cannot collide. +const handshakeName = "test-endpoint.json" + +// Handshake is the contract between a dev loop hosting the test endpoint and a +// test run attaching to it. +// +// It carries a live credential, so it is written 0600 and removed when the dev +// loop exits. It is not a secret store: the token it holds only works against a +// loopback endpoint on this machine, and only until that runtime stops. +type Handshake struct { + // Project is the .mpr the dev loop is serving, so an attach can refuse a + // handshake left behind by a different project. + Project string `json:"project"` + // PID of the hosting `mxcli run --local` process, used to detect a stale file. + PID int `json:"pid"` + // AppPort is where the test endpoint is reachable. + AppPort int `json:"appPort"` + // AdminPort is the M2EE admin API, used to reload the model after injecting + // test microflows. + AdminPort int `json:"adminPort"` + // AdminPass authenticates against that admin API. It is NOT the endpoint + // token: the two are different secrets, and using one for the other fails + // with "Authentication failed" at the first reload. + AdminPass string `json:"adminPass"` + // ServePort is the mxbuild serve API, used to rebuild after injecting. + ServePort int `json:"servePort"` + // Token authenticates against the endpoint. + Token string `json:"token"` + // Started is when the dev loop published this, for a clearer stale message. + Started time.Time `json:"started"` +} + +// HandshakePath is the handshake file's location for a project. +func HandshakePath(projectPath string) string { + return filepath.Join(filepath.Dir(projectPath), ".mxcli", handshakeName) +} + +// WriteHandshake publishes the handshake, replacing any existing one. +// +// Written via a temp file and renamed so an attach can never read a +// half-written file, and created 0600 because it carries the endpoint token. +func WriteHandshake(projectPath string, h Handshake) error { + path := HandshakePath(projectPath) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return fmt.Errorf("creating %s: %w", filepath.Dir(path), err) + } + body, err := json.MarshalIndent(h, "", " ") + if err != nil { + return err + } + tmp := path + ".tmp" + if err := os.WriteFile(tmp, body, 0o600); err != nil { + return fmt.Errorf("writing %s: %w", tmp, err) + } + if err := os.Rename(tmp, path); err != nil { + os.Remove(tmp) + return fmt.Errorf("publishing %s: %w", path, err) + } + return nil +} + +// RemoveHandshake deletes the handshake. Safe when it is not there. +func RemoveHandshake(projectPath string) { + os.Remove(HandshakePath(projectPath)) +} + +// ReadHandshake loads the handshake for a project and rejects a stale one. +// +// Staleness matters more than it looks: a dev loop killed with SIGKILL leaves +// the file behind, and attaching to a dead runtime would fail with a confusing +// connection error several steps later. Checking the recorded PID turns that +// into one clear message at the start. +func ReadHandshake(projectPath string) (*Handshake, error) { + path := HandshakePath(projectPath) + body, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + return nil, fmt.Errorf("no test endpoint is being hosted for this project\n"+ + " --attach needs an app already running with the endpoint. Start one with:\n"+ + " mxcli run --local --test-endpoint -p %s\n"+ + " (expected the handshake at %s)", projectPath, path) + } + return nil, fmt.Errorf("reading %s: %w", path, err) + } + + var h Handshake + if err := json.Unmarshal(body, &h); err != nil { + return nil, fmt.Errorf("%s is not readable as a handshake: %w", path, err) + } + if h.Token == "" || h.AppPort == 0 { + return nil, fmt.Errorf("%s is incomplete; stop and restart the hosting 'mxcli run --local --test-endpoint'", path) + } + if !processAlive(h.PID) { + return nil, fmt.Errorf("the app that published %s (pid %d, started %s) is no longer running\n"+ + " Start one with: mxcli run --local --test-endpoint -p %s", + path, h.PID, h.Started.Format(time.RFC3339), projectPath) + } + return &h, nil +} + +// processAlive reports whether a pid names a live process. Signal 0 performs the +// existence and permission checks without delivering anything. +func processAlive(pid int) bool { + if pid <= 0 { + return false + } + proc, err := os.FindProcess(pid) + if err != nil { + return false + } + return proc.Signal(syscallSignalZero) == nil +} + +// nowFunc is time.Now, indirected so a test can pin the timestamp. +var nowFunc = time.Now diff --git a/cmd/mxcli/testrunner/handshake_test.go b/cmd/mxcli/testrunner/handshake_test.go new file mode 100644 index 000000000..26d5ad503 --- /dev/null +++ b/cmd/mxcli/testrunner/handshake_test.go @@ -0,0 +1,245 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func tempProject(t *testing.T) string { + t.Helper() + dir := t.TempDir() + mpr := filepath.Join(dir, "App.mpr") + if err := os.WriteFile(mpr, []byte("x"), 0o600); err != nil { + t.Fatalf("writing project fixture: %v", err) + } + return mpr +} + +func TestHandshakeRoundTrip(t *testing.T) { + mpr := tempProject(t) + want := Handshake{ + Project: mpr, PID: os.Getpid(), + AppPort: 8080, AdminPort: 8090, ServePort: 6543, + Token: "tok", Started: time.Now(), + } + if err := WriteHandshake(mpr, want); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + + got, err := ReadHandshake(mpr) + if err != nil { + t.Fatalf("ReadHandshake: %v", err) + } + if got.Token != want.Token || got.AppPort != want.AppPort || + got.AdminPort != want.AdminPort || got.ServePort != want.ServePort { + t.Errorf("round trip lost data: got %+v", got) + } +} + +// TestHandshakeIsNotWorldReadable pins the file mode: the handshake carries a +// live token for an endpoint that executes microflows. +func TestHandshakeIsNotWorldReadable(t *testing.T) { + mpr := tempProject(t) + if err := WriteHandshake(mpr, Handshake{PID: os.Getpid(), AppPort: 8080, Token: "tok"}); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + info, err := os.Stat(HandshakePath(mpr)) + if err != nil { + t.Fatalf("stat: %v", err) + } + if perm := info.Mode().Perm(); perm != 0o600 { + t.Errorf("handshake mode is %o, want 600 — it holds a live token", perm) + } +} + +// TestHandshakeLeavesNoTempFile pins that the write-then-rename never leaves the +// intermediate behind, which would also be a token on disk nobody cleans up. +func TestHandshakeLeavesNoTempFile(t *testing.T) { + mpr := tempProject(t) + if err := WriteHandshake(mpr, Handshake{PID: os.Getpid(), AppPort: 8080, Token: "tok"}); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + if _, err := os.Stat(HandshakePath(mpr) + ".tmp"); !os.IsNotExist(err) { + t.Error("the temp file used for the atomic write was left behind") + } +} + +func TestReadHandshakeMissingExplainsHowToStartOne(t *testing.T) { + mpr := tempProject(t) + _, err := ReadHandshake(mpr) + if err == nil { + t.Fatal("reading a missing handshake succeeded") + } + if !strings.Contains(err.Error(), "--test-endpoint") { + t.Errorf("error %q does not say how to start a hosting app", err) + } +} + +// TestReadHandshakeRejectsADeadHost pins the staleness check. A dev loop killed +// with SIGKILL leaves the file behind; without this the attach would fail much +// later with a confusing connection error. +func TestReadHandshakeRejectsADeadHost(t *testing.T) { + mpr := tempProject(t) + // PID 0x7FFFFFFF is above any real pid_max, so it cannot be live. + if err := WriteHandshake(mpr, Handshake{PID: 0x7FFFFFFF, AppPort: 8080, Token: "tok", Started: time.Now()}); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + _, err := ReadHandshake(mpr) + if err == nil { + t.Fatal("a handshake naming a dead process was accepted") + } + if !strings.Contains(err.Error(), "no longer running") { + t.Errorf("error %q does not identify the host as dead", err) + } +} + +func TestReadHandshakeRejectsIncomplete(t *testing.T) { + mpr := tempProject(t) + body, _ := json.Marshal(Handshake{PID: os.Getpid(), AppPort: 8080}) // no token + path := HandshakePath(mpr) + os.MkdirAll(filepath.Dir(path), 0o755) + if err := os.WriteFile(path, body, 0o600); err != nil { + t.Fatalf("write: %v", err) + } + if _, err := ReadHandshake(mpr); err == nil { + t.Fatal("a handshake with no token was accepted") + } +} + +func TestRemoveHandshake(t *testing.T) { + mpr := tempProject(t) + if err := WriteHandshake(mpr, Handshake{PID: os.Getpid(), AppPort: 8080, Token: "tok"}); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + RemoveHandshake(mpr) + if _, err := os.Stat(HandshakePath(mpr)); !os.IsNotExist(err) { + t.Error("the handshake survived RemoveHandshake") + } + RemoveHandshake(mpr) // must not panic when already gone +} + +// TestGenerateEndpointMDLChainsAfterStartup pins that hosting the endpoint in a +// dev app does not silently drop the app's own startup logic — seed data, for +// instance — which displacing after-startup would. +func TestGenerateEndpointMDLChainsAfterStartup(t *testing.T) { + mdl := GenerateEndpointMDL("MyModule.ASU_Startup") + if !strings.Contains(mdl, "CALL MICROFLOW MyModule.ASU_Startup()") { + t.Errorf("the project's own after-startup is not chained:\n%s", mdl) + } + + register := strings.Index(mdl, "CALL JAVA ACTION "+endpointRegisterAction) + chained := strings.Index(mdl, "CALL MICROFLOW MyModule.ASU_Startup()") + if register > chained { + t.Error("the endpoint is registered after the chained microflow; a failure in that microflow would then leave no endpoint to diagnose it with") + } +} + +func TestGenerateEndpointMDLNoChainWhenNone(t *testing.T) { + mdl := GenerateEndpointMDL("") + if strings.Contains(mdl, "$Chained") { + t.Errorf("a chained call was emitted with no microflow to chain:\n%s", mdl) + } +} + +func TestDropTestFlows(t *testing.T) { + got := dropTestFlows(&TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}}}) + want := []string{"DROP MICROFLOW MxTest.Test_test_1", "DROP MICROFLOW MxTest.Test_test_2"} + if len(got) != len(want) { + t.Fatalf("got %q, want %q", got, want) + } + for i := range got { + if got[i] != want[i] { + t.Errorf("command %d: got %q, want %q", i, got[i], want[i]) + } + } +} + +// TestDropTestFlowsNeverTouchesTheEndpoint pins the ownership boundary: an +// attach adds only test microflows, so it must remove only those. The endpoint +// and the after-startup setting belong to the dev loop hosting them. +func TestDropTestFlowsNeverTouchesTheEndpoint(t *testing.T) { + for _, cmd := range dropTestFlows(&TestSuite{Tests: []TestCase{{ID: "test_1"}}}) { + for _, forbidden := range []string{"DROP MODULE", endpointStartupFlow, endpointRegisterAction, "AfterStartupMicroflow"} { + if strings.Contains(cmd, forbidden) { + t.Errorf("attach cleanup would remove %q, which the hosting dev loop owns: %q", forbidden, cmd) + } + } + } +} + +func TestValidateOptionsAttach(t *testing.T) { + tests := []struct { + name string + opts RunOptions + wantErr string + }{ + {name: "attach alone is fine", opts: RunOptions{Attach: true}}, + {name: "attach with watch is fine", opts: RunOptions{Attach: true, Watch: true}}, + { + name: "attach with the legacy runner", + opts: RunOptions{Attach: true, LegacyRunner: true}, + wantErr: "--attach cannot be combined with --legacy-runner", + }, + { + name: "attach with skip-build", + opts: RunOptions{Attach: true, SkipBuild: true}, + wantErr: "--attach cannot be combined with --skip-build", + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := validateOptions(tt.opts) + if tt.wantErr == "" { + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + return + } + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("error = %v, want it to contain %q", err, tt.wantErr) + } + }) + } +} + +// TestAttachDoesNotRequireLocal pins that --attach implies a local app: needing +// --local as well would be noise, since there is nothing else to attach to. +func TestAttachDoesNotRequireLocal(t *testing.T) { + if err := validateOptions(RunOptions{Attach: true, Local: false}); err != nil { + t.Errorf("--attach without --local was rejected: %v", err) + } +} + +// TestAttachUsesTheAdminPasswordNotTheEndpointToken pins a bug found by running +// --attach against a live app: the M2EE admin API and the test endpoint use +// different secrets, and passing the endpoint token to the admin API fails with +// "Authentication failed" at the first reload — after the test microflows have +// already been injected. +func TestAttachUsesTheAdminPasswordNotTheEndpointToken(t *testing.T) { + mpr := tempProject(t) + hs := Handshake{ + Project: mpr, PID: os.Getpid(), + AppPort: 8080, AdminPort: 8090, ServePort: 6543, + AdminPass: "the-admin-password", + Token: "the-endpoint-token", + } + if err := WriteHandshake(mpr, hs); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + got, err := ReadHandshake(mpr) + if err != nil { + t.Fatalf("ReadHandshake: %v", err) + } + if got.AdminPass != "the-admin-password" { + t.Fatalf("the handshake does not carry the admin password (got %q)", got.AdminPass) + } + if got.AdminPass == got.Token { + t.Error("the admin password and the endpoint token are the same value; they are different secrets") + } +} diff --git a/cmd/mxcli/testrunner/host.go b/cmd/mxcli/testrunner/host.go new file mode 100644 index 000000000..b70e0812a --- /dev/null +++ b/cmd/mxcli/testrunner/host.go @@ -0,0 +1,119 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "fmt" + "io" + "os" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// HostedEndpoint is a test endpoint installed into a project for the lifetime of +// a dev loop, so `mxcli test --attach` can run tests against it without booting +// its own runtime. +// +// It exists because the endpoint cannot be added to a running app: the handler +// is registered by the after-startup microflow, which only runs at boot, and its +// token comes from the runtime's environment. Whoever boots the app therefore +// has to opt in — which is also the right place for the decision, since hosting +// it means the developer's own app carries a microflow-executing endpoint and +// runs tests against the database they are looking at. +type HostedEndpoint struct { + // Token the runtime must be given as MXCLI_TEST_TOKEN. + Token string + // Env is the entry to add to the runtime process environment. + Env []string + + projectPath string + state projectState + out io.Writer + removed bool +} + +// InstallHostedEndpoint injects the test endpoint into the project and returns +// what the caller needs to boot with it. The caller must Remove it on shutdown. +// +// The project's own after-startup microflow is chained rather than displaced, so +// the app still boots the way the developer expects. +func InstallHostedEndpoint(projectPath string, w io.Writer) (*HostedEndpoint, error) { + token, err := newEndpointToken() + if err != nil { + return nil, err + } + + state, err := captureProjectState(projectPath) + if err != nil { + return nil, fmt.Errorf("capturing project state: %w", err) + } + + h := &HostedEndpoint{ + Token: token, + Env: []string{endpointTokenEnv + "=" + token}, + projectPath: projectPath, + state: state, + out: w, + } + + fmt.Fprintln(w, "Installing the mxcli test endpoint (for 'mxcli test --attach')...") + if err := execMDLScript(projectPath, GenerateEndpointMDL(state.afterStartup), "mxtest-endpoint-*.mdl"); err != nil { + h.Remove() + return nil, fmt.Errorf("injecting the test endpoint: %w", err) + } + if err := execMxcliCmd(projectPath, "ALTER SETTINGS MODEL AfterStartupMicroflow = "+quoteMDLString(endpointStartupFlow)); err != nil { + h.Remove() + return nil, fmt.Errorf("pointing after-startup at the endpoint: %w", err) + } + if state.afterStartup != "" { + fmt.Fprintf(w, " after-startup chained: %s runs, then %s\n", endpointStartupFlow, state.afterStartup) + } + return h, nil +} + +// Publish writes the handshake that `mxcli test --attach` reads. Called once the +// app is actually serving, so an attach never finds a handshake for a runtime +// that has not come up. +func (h *HostedEndpoint) Publish(info docker.LocalAppInfo) error { + if h == nil { + return nil + } + err := WriteHandshake(h.projectPath, Handshake{ + Project: h.projectPath, + PID: os.Getpid(), + AppPort: info.AppPort, + AdminPort: info.AdminPort, + ServePort: info.ServePort, + AdminPass: info.AdminPass, + Token: h.Token, + Started: nowFunc(), + }) + if err != nil { + return err + } + fmt.Fprintf(h.out, " test endpoint ready — run 'mxcli test -p %s --attach' from another terminal\n", h.projectPath) + return nil +} + +// Remove withdraws the handshake and restores the project. +// +// Best-effort by design: this runs on the dev loop's shutdown path, where the +// user is trying to stop the app, and a hard failure there helps nobody. What it +// must not do is stay silent — a project left carrying the endpoint is something +// the developer has to know about before they commit. +func (h *HostedEndpoint) Remove() { + // Idempotent and nil-safe: the caller both defers this and calls it on the + // os.Exit path (which skips defers), so it must tolerate running twice. + if h == nil || h.removed { + return + } + h.removed = true + RemoveHandshake(h.projectPath) + if err := cleanupEndpoint(h.projectPath, h.state, &TestSuite{}, h.out); err != nil { + fmt.Fprintf(h.out, "\nERROR: could not remove the test endpoint — the project has been left modified:\n%v\n", err) + fmt.Fprintf(h.out, "Check the after-startup microflow and the %s module before committing.\n", mxTestModule) + return + } + removeGeneratedJavaSource(h.projectPath, h.out) + fmt.Fprintln(h.out, " test endpoint removed; project restored") +} diff --git a/cmd/mxcli/testrunner/runner.go b/cmd/mxcli/testrunner/runner.go index b82fa06f9..ffc8bf459 100644 --- a/cmd/mxcli/testrunner/runner.go +++ b/cmd/mxcli/testrunner/runner.go @@ -38,9 +38,26 @@ type RunOptions struct { SkipBuild bool // Local runs the app with mxcli's own local runtime (`run --local`) instead - // of a Docker container. Everything else about the run is unchanged. + // of a Docker container, and drives the tests over the test endpoint. Local bool + // LegacyRunner forces the after-startup mechanism on a local run — the suite + // compiled into one startup microflow, results read back from the runtime + // log. An escape hatch for the case where the endpoint misbehaves; the + // Docker path uses this mechanism regardless. + LegacyRunner bool + + // Watch keeps the runtime and the build server up and re-runs the suite on + // every change to a test file or to the project's model, until interrupted. + // Requires the test endpoint, so it is incompatible with LegacyRunner and + // with the Docker path — both of which can only re-run by restarting. + Watch bool + + // Attach runs against an app already started with + // `mxcli run --local --test-endpoint`, skipping the boot entirely. The tests + // then run against that app's database rather than a scratch one. + Attach bool + // Timeout for runtime startup and test execution. Timeout time.Duration @@ -60,14 +77,22 @@ type RunOptions struct { Stderr io.Writer } -// Run executes the test suite using the after-startup pattern: -// 1. Parse test files -// 2. Generate TestRunner microflow -// 3. Inject into project, set as after-startup -// 4. Build and restart runtime -// 5. Parse logs for results -// 6. Cleanup (restore original settings) -// 7. Output results +// Run executes the test suite. +// +// There are two mechanisms, and which one is used follows from opts.Local: +// +// - Local runs go through the test endpoint (runEndpoint). Boot registers an +// HTTP handler and nothing else; each test is then invoked by name against a +// runtime that stays up, and returns its verdict in the response. +// - Docker runs go through the after-startup runner (runAfterStartup), which +// compiles the suite into the project's after-startup microflow, restarts the +// container, and recovers results from its log. +// +// The endpoint is the better mechanism — a re-run is an HTTP call rather than a +// restart, a failing test is a result rather than a failed boot, and results are +// returned rather than scraped. It is confined to the local path because it +// needs to hand the runtime a secret through its environment and to reach it on +// loopback, neither of which is wired through docker-compose yet. func Run(opts RunOptions) (*SuiteResult, error) { w := opts.Stdout if w == nil { @@ -78,12 +103,24 @@ func Run(opts RunOptions) (*SuiteResult, error) { stderr = os.Stderr } + if err := validateOptions(opts); err != nil { + return nil, err + } + timeout := opts.Timeout if timeout == 0 { timeout = 5 * time.Minute } - // Step 1: Parse test files + // Resolve the project path up front so everything derived from it — the + // runtime log path, the deployment directory, the paths named in error + // messages — is absolute and agrees. + if opts.ProjectPath != "" && !filepath.IsAbs(opts.ProjectPath) { + if abs, err := filepath.Abs(opts.ProjectPath); err == nil { + opts.ProjectPath = abs + } + } + fmt.Fprintln(w, "Parsing test files...") suite, err := parseTestFiles(opts.TestFiles) if err != nil { @@ -95,56 +132,166 @@ func Run(opts RunOptions) (*SuiteResult, error) { return nil, fmt.Errorf("no tests found in the provided files") } - // Step 2: Generate TestRunner microflow MDL - fmt.Fprintln(w, "Generating test runner microflow...") - runnerMDL := GenerateTestRunner(suite) + if opts.Attach { + return runAttached(opts, suite, timeout, w) + } + if opts.Local && !opts.LegacyRunner { + return runEndpoint(opts, suite, timeout, w) + } + return runAfterStartup(opts, suite, timeout, w) +} + +// validateOptions rejects combinations that cannot work, with a message that +// says what to do instead. Watching depends on re-invoking tests without a +// restart, which only the test endpoint can do. +func validateOptions(opts RunOptions) error { + if opts.Attach { + if opts.LegacyRunner { + return fmt.Errorf("--attach cannot be combined with --legacy-runner: the after-startup runner can only run tests by restarting, which is what attaching avoids") + } + if opts.SkipBuild { + return fmt.Errorf("--attach cannot be combined with --skip-build: the attached app must be rebuilt to pick up the test microflows") + } + // --attach implies a local app; requiring --local as well would be noise. + return nil + } + if !opts.Watch { + return nil + } + if !opts.Local { + return fmt.Errorf("--watch requires --local: the Docker path can only re-run tests by restarting the container") + } + if opts.LegacyRunner { + return fmt.Errorf("--watch cannot be combined with --legacy-runner: the after-startup runner can only re-run tests by restarting the runtime") + } + if opts.SkipBuild { + return fmt.Errorf("--watch cannot be combined with --skip-build: watching exists to rebuild on every change") + } + return nil +} + +// runEndpoint injects the test endpoint plus one microflow per test, boots the +// app once, and drives the suite over HTTP. +func runEndpoint(opts RunOptions, suite *TestSuite, timeout time.Duration, w io.Writer) (*SuiteResult, error) { + token, err := newEndpointToken() + if err != nil { + return nil, err + } + + fmt.Fprintln(w, "Generating test endpoint and test microflows...") + // "" : a test run wants a known starting state, so the project's own + // after-startup is not chained here (a hosted endpoint does chain it). + endpointMDL := GenerateEndpointMDL("") + flowsMDL := GenerateTestFlows(suite) if opts.Verbose { fmt.Fprintln(w, "--- Generated MDL ---") - fmt.Fprintln(w, runnerMDL) + fmt.Fprintln(w, endpointMDL) + fmt.Fprintln(w, flowsMDL) fmt.Fprintln(w, "--- End MDL ---") } - // Step 3: Save original settings and inject test runner - fmt.Fprintln(w, "Injecting test runner into project...") // Capture what cleanup will need to restore, before touching anything. This // must succeed: without it cleanup cannot tell an existing MxTest module from // the one it is about to create, nor restore the original after-startup. + fmt.Fprintln(w, "Injecting test endpoint into project...") state, err := captureProjectState(opts.ProjectPath) if err != nil { return nil, fmt.Errorf("capturing project state: %w", err) } - // Write the runner MDL to a temp file and execute it - tmpFile, err := os.CreateTemp("", "mxtest-runner-*.mdl") + // From here on the project is modified, so every exit runs cleanup. + // + // cleanupSuite, not the suite captured here: --watch re-parses on every + // change, so by the time cleanup runs the set of injected test microflows may + // differ from the one injected at boot. Dropping the wrong list would leave + // generated microflows in the user's project. + cleanupSuite := suite + finish := func(result *SuiteResult, runErr error) (*SuiteResult, error) { + fmt.Fprintln(w, "Cleaning up...") + cleanupErr := cleanupEndpoint(opts.ProjectPath, state, cleanupSuite, w) + removeGeneratedJavaSource(opts.ProjectPath, w) + reportCleanup(w, cleanupErr) + if cleanupErr == nil { + fmt.Fprintln(w, " project restored") + } + if runErr != nil { + return nil, runErr + } + if cleanupErr != nil { + return result, fmt.Errorf("cleanup failed, project left modified: %w", cleanupErr) + } + return result, nil + } + + if err := execMDLScript(opts.ProjectPath, endpointMDL, "mxtest-endpoint-*.mdl"); err != nil { + return finish(nil, fmt.Errorf("injecting test endpoint: %w", err)) + } + if err := execMDLScript(opts.ProjectPath, flowsMDL, "mxtest-flows-*.mdl"); err != nil { + return finish(nil, fmt.Errorf("injecting test microflows: %w", err)) + } + for _, cmd := range setupCommands(endpointStartupFlow) { + if err := execMxcliCmd(opts.ProjectPath, cmd); err != nil { + return finish(nil, fmt.Errorf("preparing project for the test run (%s): %w", cmd, err)) + } + } + fmt.Fprintf(w, " After-startup set to %s (registers the endpoint; runs no tests)\n", endpointStartupFlow) + + // --watch keeps the runtime and the build server up and re-runs on every + // change, so it owns the loop — including printing each run's results, which + // it must do before cleanup rather than after. It reports each re-injected + // suite back so cleanup drops what is actually in the project. + if opts.Watch { + return runEndpointWatch(opts, suite, token, timeout, w, finish, func(s *TestSuite) { cleanupSuite = s }) + } + + result, err := runViaEndpoint(opts, suite, token, timeout, w) if err != nil { - return nil, fmt.Errorf("creating temp file: %w", err) + return finish(nil, err) } - tmpPath := tmpFile.Name() - defer os.Remove(tmpPath) - if _, err := tmpFile.WriteString(runnerMDL); err != nil { - tmpFile.Close() - return nil, fmt.Errorf("writing runner MDL: %w", err) + result, err = finish(result, nil) + if result != nil { + PrintResults(w, result, opts.Color) + if jerr := writeJUnit(opts, result, w); jerr != nil && err == nil { + err = jerr + } } - tmpFile.Close() + return result, err +} + +// runAfterStartup is the original mechanism: compile the suite into the +// after-startup microflow, restart the runtime, and read results out of the log. +// Always the Docker path, and the local path under --legacy-runner. +func runAfterStartup(opts RunOptions, suite *TestSuite, timeout time.Duration, w io.Writer) (*SuiteResult, error) { + fmt.Fprintln(w, "Generating test runner microflow...") + runnerMDL := GenerateTestRunner(suite) - // Execute the MDL to create the TestRunner microflow - if err := execMxcli(opts.ProjectPath, "exec", tmpPath, "-p", opts.ProjectPath); err != nil { + if opts.Verbose { + fmt.Fprintln(w, "--- Generated MDL ---") + fmt.Fprintln(w, runnerMDL) + fmt.Fprintln(w, "--- End MDL ---") + } + + // Save original settings and inject the test runner. + fmt.Fprintln(w, "Injecting test runner into project...") + state, err := captureProjectState(opts.ProjectPath) + if err != nil { + return nil, fmt.Errorf("capturing project state: %w", err) + } + + if err := execMDLScript(opts.ProjectPath, runnerMDL, "mxtest-runner-*.mdl"); err != nil { return nil, fmt.Errorf("injecting test runner: %w", err) } // Set after-startup microflow - for _, cmd := range setupCommands() { + for _, cmd := range setupCommands(mxTestRunner) { if err := execMxcliCmd(opts.ProjectPath, cmd); err != nil { return nil, fmt.Errorf("preparing project for the test run (%s): %w", cmd, err) } } fmt.Fprintf(w, " After-startup set to %s\n", mxTestRunner) - // Steps 4+5: build, run the app, and capture the runner's log output. The - // local and Docker paths differ only in how the app is started and where its - // log is read from; everything before and after is shared. var logOutput string if opts.Local { logOutput, err = runLocalAndCapture(opts, timeout, w) @@ -156,29 +303,17 @@ func Run(opts RunOptions) (*SuiteResult, error) { return nil, err } - // Step 6: Parse results from logs fmt.Fprintln(w, "Parsing test results...") result := ParseLogResults(strings.NewReader(logOutput), suite) - // Step 7: Cleanup fmt.Fprintln(w, "Cleaning up...") cleanupErr := cleanup(opts.ProjectPath, state, w) reportCleanup(w, cleanupErr) - // Step 8: Output results PrintResults(w, result, opts.Color) - // Write JUnit XML if requested - if opts.JUnitOutput != "" { - f, err := os.Create(opts.JUnitOutput) - if err != nil { - return result, fmt.Errorf("creating JUnit output: %w", err) - } - defer f.Close() - if err := WriteJUnitXML(f, result); err != nil { - return result, fmt.Errorf("writing JUnit XML: %w", err) - } - fmt.Fprintf(w, "JUnit XML written to: %s\n", opts.JUnitOutput) + if err := writeJUnit(opts, result, w); err != nil { + return result, err } // A failed cleanup leaves the project modified, so the run must not be @@ -189,6 +324,23 @@ func Run(opts RunOptions) (*SuiteResult, error) { return result, nil } +// writeJUnit writes the JUnit XML report when one was asked for. +func writeJUnit(opts RunOptions, result *SuiteResult, w io.Writer) error { + if opts.JUnitOutput == "" { + return nil + } + f, err := os.Create(opts.JUnitOutput) + if err != nil { + return fmt.Errorf("creating JUnit output: %w", err) + } + defer f.Close() + if err := WriteJUnitXML(f, result); err != nil { + return fmt.Errorf("writing JUnit XML: %w", err) + } + fmt.Fprintf(w, "JUnit XML written to: %s\n", opts.JUnitOutput) + return nil +} + // ListTests parses test files and prints the test names without executing. func ListTests(files []string, w io.Writer) error { suite, err := parseTestFiles(files) @@ -360,17 +512,53 @@ func moduleExists(projectPath, name string) (bool, error) { } // setupCommands returns the MDL statements Run issues to put the project into its -// testing state. The project's Security Level is deliberately absent: the -// after-startup microflow runs in an administrative context and is not subject to -// it, so forcing it OFF bought nothing — while breaking any project with a -// published REST/OData service using custom authentication ("App security is off, -// but custom authentication is enabled for this service"), and the restore -// hardcoded PRODUCTION, silently changing projects that run at another level -// (mendixlabs/mxcli#802). -func setupCommands() []string { +// testing state, pointing after-startup at startupFlow. The project's Security +// Level is deliberately absent: the after-startup microflow runs in an +// administrative context and is not subject to it, so forcing it OFF bought +// nothing — while breaking any project with a published REST/OData service using +// custom authentication ("App security is off, but custom authentication is +// enabled for this service"), and the restore hardcoded PRODUCTION, silently +// changing projects that run at another level (mendixlabs/mxcli#802). +func setupCommands(startupFlow string) []string { return []string{ - "ALTER SETTINGS MODEL AfterStartupMicroflow = " + quoteMDLString(mxTestRunner), + "ALTER SETTINGS MODEL AfterStartupMicroflow = " + quoteMDLString(startupFlow), + } +} + +// runMDLCommands executes each statement, attempting all of them even after one +// fails, and joins the failures. Restores must not stop at the first error: a +// half-restored project is worse than a fully failed one, because it looks fine +// (#803). +func runMDLCommands(projectPath string, cmds []string) error { + var errs []error + for _, cmd := range cmds { + if err := execMxcliCmd(projectPath, cmd); err != nil { + errs = append(errs, fmt.Errorf("%s: %w", cmd, err)) + } + } + return errors.Join(errs...) +} + +// lowerModule maps a module name to the javasource/ directory Mendix generates +// for it, which is always lowercased. +func lowerModule(name string) string { return strings.ToLower(name) } + +// execMDLScript writes MDL to a temp file and executes it against the project. +func execMDLScript(projectPath, mdl, namePattern string) error { + f, err := os.CreateTemp("", namePattern) + if err != nil { + return fmt.Errorf("creating temp file: %w", err) + } + path := f.Name() + defer os.Remove(path) + + if _, err := f.WriteString(mdl); err != nil { + f.Close() + return fmt.Errorf("writing MDL: %w", err) } + f.Close() + + return execMxcli(projectPath, "exec", path, "-p", projectPath) } // cleanupCommands returns the MDL statements that put the project back the way it @@ -418,16 +606,7 @@ func cleanup(projectPath string, st projectState, w io.Writer) error { fmt.Fprintf(w, " %s module already existed; dropping only %s\n", mxTestModule, mxTestRunner) } - var errs []error - for _, cmd := range cleanupCommands(st, mxTestPresent) { - if err := execMxcliCmd(projectPath, cmd); err != nil { - errs = append(errs, fmt.Errorf("%s: %w", cmd, err)) - } - } - if len(errs) > 0 { - return errors.Join(errs...) - } - return nil + return runMDLCommands(projectPath, cleanupCommands(st, mxTestPresent)) } // reportCleanup prints a cleanup failure prominently. The project is left mutated, diff --git a/cmd/mxcli/testrunner/runner_attach.go b/cmd/mxcli/testrunner/runner_attach.go new file mode 100644 index 000000000..53565477c --- /dev/null +++ b/cmd/mxcli/testrunner/runner_attach.go @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "fmt" + "io" + "time" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// attachedApp is a running app someone else owns, reached through its handshake. +// +// It satisfies the same two needs bootForTests does — a client for the endpoint, +// and a way to apply a model change — but owns neither the runtime nor the build +// server, so it must never stop them. +type attachedApp struct { + client *endpointClient + serve *docker.ServeServer + ctrl *docker.RuntimeController + hs *Handshake +} + +// attach connects to an app already hosting the test endpoint. +func attach(opts RunOptions, w io.Writer) (*attachedApp, error) { + hs, err := ReadHandshake(opts.ProjectPath) + if err != nil { + return nil, err + } + + client := newEndpointClient(hs.AppPort, hs.Token) + if err := client.ping(); err != nil { + return nil, fmt.Errorf("the app on port %d is not answering the test endpoint: %w\n"+ + " The handshake looks live, so the app may still be starting. Retry in a moment.", hs.AppPort, err) + } + + fmt.Fprintf(w, "Attached to the app on port %d (pid %d) — no boot needed.\n", hs.AppPort, hs.PID) + // The dev app's database is the developer's, and these tests are about to + // write to it. That is the whole trade --attach makes, so say it every time + // rather than burying it in the docs. + fmt.Fprintln(w, " NOTE: tests run against the running app's database, not a scratch one.") + + return &attachedApp{ + client: client, + serve: &docker.ServeServer{Host: "127.0.0.1", Port: hs.ServePort}, + // The admin password, not the endpoint token — different secrets. + ctrl: docker.NewRuntimeController(docker.M2EEOptions{ + Host: "127.0.0.1", + Port: hs.AdminPort, + Token: hs.AdminPass, + }), + hs: hs, + }, nil +} + +// applyModelChange rebuilds the project through the dev loop's serve server and +// applies the result through its admin API. +// +// Driving the other process's services rather than requiring it to be in +// --watch: both are plain loopback APIs, so an attach can apply its own +// injections deterministically instead of waiting to see whether someone else's +// watcher noticed. A dev loop that *is* watching may also rebuild — harmless, +// since both produce the same deployment from the same source. +func (a *attachedApp) endpoint() *endpointClient { return a.client } + +func (a *attachedApp) applyModelChange(projectPath string) (string, error) { + build, err := a.serve.Build(docker.BuildRequest{Target: docker.TargetDeploy, ProjectFilePath: projectPath}) + if err != nil { + return "", fmt.Errorf("rebuilding through the attached app's build server on port %d: %w", a.hs.ServePort, err) + } + if !build.OK() { + return "", fmt.Errorf("build failed: %s", build.Message) + } + // No restart callback: the runtime belongs to the other process. A structural + // change is refused rather than half-applied — see the error below. + action, err := a.ctrl.ApplyBuild(build, nil) + if err != nil { + return action.String(), err + } + if action == docker.ActionRestart { + return action.String(), fmt.Errorf("this change needs a runtime restart (an entity or association changed), " + + "which --attach cannot do — the runtime belongs to the 'mxcli run --local' process.\n" + + " Restart that process, or drop --attach to run against a scratch runtime.") + } + return action.String(), nil +} + +// runAttached injects the test microflows into the already-running app, runs the +// suite, and removes them again. +// +// It deliberately does not touch the endpoint or the after-startup setting: the +// dev loop installed those and will remove them when it exits. An attach only +// owns the test microflows it adds. +func runAttached(opts RunOptions, suite *TestSuite, timeout time.Duration, w io.Writer) (*SuiteResult, error) { + app, err := attach(opts, w) + if err != nil { + return nil, err + } + + // Only the generated test microflows are ours to remove. + injected := suite + finish := func(result *SuiteResult, runErr error) (*SuiteResult, error) { + fmt.Fprintln(w, "Cleaning up...") + cleanupErr := runMDLCommands(opts.ProjectPath, dropTestFlows(injected)) + if cleanupErr == nil { + // Leave the app serving a model that matches the project on disk; + // otherwise the developer's next page load still runs the test flows. + if _, err := app.applyModelChange(opts.ProjectPath); err != nil { + fmt.Fprintf(w, " note: the app is still serving the test microflows until its next rebuild: %v\n", err) + } + fmt.Fprintln(w, " test microflows removed") + } else { + reportCleanup(w, cleanupErr) + } + if runErr != nil { + return nil, runErr + } + if cleanupErr != nil { + return result, fmt.Errorf("cleanup failed, project left modified: %w", cleanupErr) + } + return result, nil + } + + fmt.Fprintln(w, "Injecting test microflows...") + if err := execMDLScript(opts.ProjectPath, GenerateTestFlows(suite), "mxtest-flows-*.mdl"); err != nil { + return finish(nil, fmt.Errorf("injecting test microflows: %w", err)) + } + if _, err := app.applyModelChange(opts.ProjectPath); err != nil { + return finish(nil, err) + } + + if opts.Watch { + return runAttachedWatch(opts, app, suite, timeout, w, finish, func(s *TestSuite) { injected = s }) + } + + result, err := runSuite(app.client, suite, opts, w) + if err != nil { + return finish(nil, err) + } + result, err = finish(result, nil) + if result != nil { + PrintResults(w, result, opts.Color) + if jerr := writeJUnit(opts, result, w); jerr != nil && err == nil { + err = jerr + } + } + return result, err +} + +// dropTestFlows returns the DROP statements for a suite's generated microflows. +func dropTestFlows(suite *TestSuite) []string { + if suite == nil { + return nil + } + cmds := make([]string, 0, len(suite.Tests)) + for _, tc := range suite.Tests { + cmds = append(cmds, "DROP MICROFLOW "+testFlowName(tc)) + } + return cmds +} diff --git a/cmd/mxcli/testrunner/runner_cleanup_test.go b/cmd/mxcli/testrunner/runner_cleanup_test.go index 4809c0c04..0c1f0fbf5 100644 --- a/cmd/mxcli/testrunner/runner_cleanup_test.go +++ b/cmd/mxcli/testrunner/runner_cleanup_test.go @@ -107,10 +107,14 @@ func TestQuoteMDLString(t *testing.T) { } // TestNoSecurityLevelManipulation pins #802: the Security Level is the project's -// business. Neither setup nor cleanup may touch it. +// business. Neither setup nor cleanup may touch it — on either mechanism. func TestNoSecurityLevelManipulation(t *testing.T) { - all := append(setupCommands(), cleanupCommands(projectState{}, true)...) + suite := &TestSuite{Tests: []TestCase{{ID: "test_1"}}} + all := append(setupCommands(mxTestRunner), setupCommands(endpointStartupFlow)...) + all = append(all, cleanupCommands(projectState{}, true)...) all = append(all, cleanupCommands(projectState{afterStartup: "Mod.Flow", createdMxTest: true}, true)...) + all = append(all, endpointCleanupCommands(projectState{}, suite, true)...) + all = append(all, endpointCleanupCommands(projectState{afterStartup: "Mod.Flow", createdMxTest: true}, suite, true)...) for _, cmd := range all { if strings.Contains(strings.ToUpper(cmd), "SECURITY LEVEL") { t.Errorf("the runner still alters the project Security Level: %q (#802)", cmd) diff --git a/cmd/mxcli/testrunner/runner_endpoint.go b/cmd/mxcli/testrunner/runner_endpoint.go new file mode 100644 index 000000000..c3d67c0de --- /dev/null +++ b/cmd/mxcli/testrunner/runner_endpoint.go @@ -0,0 +1,241 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "fmt" + "io" + "os" + "path/filepath" + "time" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// testAppSession is a booted app plus a client for its test endpoint. It is what +// the warm loop keeps alive between runs. +type testAppSession struct { + app *docker.LocalApp + client *endpointClient + logPath string +} + +// bootForTests boots the app and waits for the test endpoint to register. +func bootForTests(opts RunOptions, token string, timeout time.Duration, w io.Writer) (*testAppSession, error) { + logPath := filepath.Join(filepath.Dir(opts.ProjectPath), ".mxcli", "test-runtime.log") + + fmt.Fprintln(w, "Starting local runtime (no Docker)...") + app, err := docker.StartLocalApp(docker.LocalAppOptions{ + ProjectPath: opts.ProjectPath, + AppPort: localTestAppPort, + AdminPort: localTestAdminPort, + ServePort: localTestServePort, + DB: docker.DBConfig{ + Name: docker.DeriveDBName(opts.ProjectPath) + localTestDBSuffix, + }, + EnsureDB: true, + SkipBuild: opts.SkipBuild, + // The token reaches the runtime through its environment and is never + // written to the project. See endpointTokenEnv. + Env: []string{endpointTokenEnv + "=" + token}, + RuntimeLogPath: logPath, + Stdout: w, + Stderr: w, + }) + if err != nil { + // Unlike the after-startup path, a boot failure here is never a test + // result — no test has run yet. It is always a real error. + return nil, fmt.Errorf("local runtime: %w", err) + } + + client := newEndpointClient(localTestAppPort, token) + if err := client.waitReady(endpointReadyTimeout(timeout)); err != nil { + app.Stop() + return nil, fmt.Errorf("%w\n hint: check %s for a registration failure", err, logPath) + } + return &testAppSession{app: app, client: client, logPath: logPath}, nil +} + +func (s *testAppSession) stop() { + if s != nil && s.app != nil { + s.app.Stop() + } +} + +// testTarget is an app the runner can run tests against and push model changes +// into. Two things satisfy it: a runtime the runner booted itself, and one +// already running that it attached to. The watch loop is written against this +// so it does not care which. +type testTarget interface { + // endpoint is the client for the app's test endpoint. + endpoint() *endpointClient + // applyModelChange rebuilds the project and applies it, returning a label for + // what it took ("reload"/"restart"). It returns only once the endpoint is + // reachable again, so the caller can invoke a test straight after. + applyModelChange(projectPath string) (string, error) +} + +func (s *testAppSession) endpoint() *endpointClient { return s.client } + +// applyModelChange rebuilds through the serve server this session owns. +// +// A restart is fine here — the session owns the runtime — but it re-runs +// after-startup, so the endpoint has to be waited for before the next test. +func (s *testAppSession) applyModelChange(projectPath string) (string, error) { + action, _, err := s.app.Rebuild(projectPath) + if err != nil { + return action.String(), err + } + if action == docker.ActionRestart { + if err := s.client.waitReady(endpointReadyTimeout(0)); err != nil { + return action.String(), fmt.Errorf("the test endpoint did not come back after a restart: %w", err) + } + } + return action.String(), nil +} + +// runViaEndpoint boots the app once and drives the suite over HTTP. +// +// The contrast with the after-startup path is the whole point: there, tests run +// during boot, so every re-run is a restart and a result is something recovered +// from the log. Here boot only registers the endpoint, and each test is a +// request against a runtime that stays up. +func runViaEndpoint(opts RunOptions, suite *TestSuite, token string, timeout time.Duration, w io.Writer) (*SuiteResult, error) { + sess, err := bootForTests(opts, token, timeout, w) + if err != nil { + return nil, err + } + defer sess.stop() + return runSuite(sess.client, suite, opts, w) +} + +// runSuite invokes every test in the suite against a booted app and collects the +// verdicts. It never returns an error for a test-level problem — a missing +// microflow or a failed request is that test's result, so one bad test cannot +// hide every other. +func runSuite(client *endpointClient, suite *TestSuite, opts RunOptions, w io.Writer) (*SuiteResult, error) { + // Ask the app which test microflows it actually has. A test whose microflow + // is missing is reported as an error against that test rather than failing + // the run. + present := make(map[string]bool) + names, err := client.list() + if err != nil { + return nil, fmt.Errorf("listing test microflows: %w", err) + } + for _, n := range names { + present[n] = true + } + + result := &SuiteResult{Name: suite.Name, Started: time.Now()} + fmt.Fprintf(w, "Running %d test(s) over the test endpoint...\n", len(suite.Tests)) + + for _, tc := range suite.Tests { + flow := testFlowName(tc) + if !present[flow] { + result.Tests = append(result.Tests, TestResult{ + ID: tc.ID, + Name: tc.Name, + Status: StatusError, + Message: fmt.Sprintf("microflow %s was not created — the test body may not have compiled", flow), + }) + continue + } + + rr, err := client.run(flow) + if err != nil { + // A transport failure is not a verdict. Report it against this test + // and keep going; if the runtime died the rest will say so too. + result.Tests = append(result.Tests, TestResult{ + ID: tc.ID, + Name: tc.Name, + Status: StatusError, + Message: fmt.Sprintf("calling the test endpoint: %v", err), + }) + continue + } + + res := toResult(tc, rr) + result.Tests = append(result.Tests, res) + if opts.Verbose { + fmt.Fprintf(w, " %s %s (%s)\n", res.Status, res.Name, res.Duration.Round(time.Millisecond)) + } + } + + result.Duration = time.Since(result.Started) + return result, nil +} + +// endpointReadyTimeout bounds the wait for the endpoint to register. It is +// capped well below the suite timeout: the handler is registered during the +// start action, so if it is not up shortly after the runtime reports started, it +// is not coming. +func endpointReadyTimeout(suiteTimeout time.Duration) time.Duration { + const cap = 60 * time.Second + if suiteTimeout > 0 && suiteTimeout < cap { + return suiteTimeout + } + return cap +} + +// endpointCleanupCommands returns the MDL that removes what the endpoint path +// injected, in order. +// +// It mirrors cleanupCommands but has more to take out: the registration Java +// action and startup microflow, plus one microflow per test. When Run created +// the MxTest module, dropping the module removes all of it in one statement; +// when the module was already the user's, each generated document is named +// explicitly so nothing of theirs is touched. +func endpointCleanupCommands(st projectState, suite *TestSuite, mxTestPresent bool) []string { + restore := "ALTER SETTINGS MODEL AfterStartupMicroflow = ''" + if st.afterStartup != "" { + restore = "ALTER SETTINGS MODEL AfterStartupMicroflow = " + quoteMDLString(st.afterStartup) + } + cmds := []string{restore} + if !mxTestPresent { + return cmds + } + if st.createdMxTest { + return append(cmds, "DROP MODULE "+mxTestModule) + } + for _, tc := range suite.Tests { + cmds = append(cmds, "DROP MICROFLOW "+testFlowName(tc)) + } + return append(cmds, + "DROP MICROFLOW "+endpointStartupFlow, + "DROP JAVA ACTION "+endpointRegisterAction, + ) +} + +// cleanupEndpoint restores the project after an endpoint run. +// +// As with cleanup, every statement is attempted even after one fails and the +// failures are returned rather than warned about: a half-restored project still +// carries a test endpoint and an after-startup pointing at it. +func cleanupEndpoint(projectPath string, st projectState, suite *TestSuite, w io.Writer) error { + mxTestPresent := true + if exists, err := moduleExists(projectPath, mxTestModule); err == nil { + mxTestPresent = exists + } + if mxTestPresent && !st.createdMxTest { + fmt.Fprintf(w, " %s module already existed; dropping only the generated documents\n", mxTestModule) + } + return runMDLCommands(projectPath, endpointCleanupCommands(st, suite, mxTestPresent)) +} + +// removeGeneratedJavaSource deletes the .java file the Java action generated. +// +// DROP JAVA ACTION removes the model document; the source file it wrote into +// javasource/ is not the model's to delete, so it is left behind. For a +// generated per-run artifact that is litter in the user's tree — and litter that +// still contains a request handler, which is exactly what should not be left +// lying around. Failure is not fatal: the file is inert without the model +// document, and a cleanup error here would mask the real ones. +func removeGeneratedJavaSource(projectPath string, w io.Writer) { + dir := filepath.Join(filepath.Dir(projectPath), "javasource", lowerModule(mxTestModule)) + if _, err := os.Stat(dir); err != nil { + return + } + if err := os.RemoveAll(dir); err != nil { + fmt.Fprintf(w, " note: could not remove generated Java source at %s: %v\n", dir, err) + } +} diff --git a/cmd/mxcli/testrunner/watch.go b/cmd/mxcli/testrunner/watch.go new file mode 100644 index 000000000..2457f4aab --- /dev/null +++ b/cmd/mxcli/testrunner/watch.go @@ -0,0 +1,231 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "fmt" + "io" + "os" + "os/signal" + "path/filepath" + "syscall" + "time" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// watchPollInterval is how often the loop checks for a change. Polling rather +// than inotify for the same reason `run --local --watch` polls: container +// filesystems do not reliably deliver inotify events for host-mounted paths. +const watchPollInterval = 1 * time.Second + +// runEndpointWatch keeps the runtime and the build server up across runs, +// re-running the suite on every change to a test file or to the app's model. +// +// This is what the endpoint was for. A cold boot is ~30s and the one-shot runner +// pays it on every invocation; here it is paid once and each subsequent run is a +// warm rebuild plus a few HTTP calls. +// +// The loop has one hazard the dev loop does not: **the runner writes to the +// project it is watching**. Injecting the test microflows changes model source, +// which is the very signal being polled, so every baseline is taken *after* the +// injection and rebuild have settled. Getting that wrong is an infinite rebuild +// loop, not a subtle bug. +func runEndpointWatch(opts RunOptions, suite *TestSuite, token string, timeout time.Duration, w io.Writer, finish finishFunc, onInject func(*TestSuite)) (*SuiteResult, error) { + sess, err := bootForTests(opts, token, timeout, w) + if err != nil { + return finish(nil, err) + } + // Belt and braces: the exits below all stop the app explicitly, before + // cleanup rewrites the project out from under it. This catches a panic. + defer sess.stop() + + // shutdown stops the app and then restores the project, in that order — + // nothing should still be serving a model that is about to have the test + // endpoint removed from it. + shutdown := func(result *SuiteResult, runErr error) (*SuiteResult, error) { + fmt.Fprintln(w, "Stopping the runtime...") + sess.stop() + return finish(result, runErr) + } + return watchLoop(opts, sess, suite, w, shutdown, onInject) +} + +// runAttachedWatch is the same loop against an app someone else owns. Nothing is +// stopped on the way out — only the injected test microflows are removed. +func runAttachedWatch(opts RunOptions, app *attachedApp, suite *TestSuite, timeout time.Duration, w io.Writer, finish finishFunc, onInject func(*TestSuite)) (*SuiteResult, error) { + return watchLoop(opts, app, suite, w, finish, onInject) +} + +// watchLoop re-runs the suite on every change until interrupted. +func watchLoop(opts RunOptions, target testTarget, suite *TestSuite, w io.Writer, shutdown finishFunc, onInject func(*TestSuite)) (*SuiteResult, error) { + // Ctrl-C has to reach the cleanup, not kill the process with the project + // still carrying the test endpoint and an after-startup pointing at it. + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + defer signal.Stop(sigCh) + + ticker := time.NewTicker(watchPollInterval) + defer ticker.Stop() + + injected := suite + var last *SuiteResult + gen := 0 + + for { + gen++ + result, err := runSuite(target.endpoint(), injected, opts, w) + if err != nil { + // The endpoint stopped answering — the runtime is probably gone, and + // nothing further will work. Bail rather than spin. + return shutdown(last, err) + } + last = result + PrintResults(w, result, opts.Color) + if err := writeJUnit(opts, result, w); err != nil { + fmt.Fprintf(w, " %v\n", err) + } + + // Baseline AFTER the run, so an edit made while tests were executing is + // still caught on the next tick. + baseline := watchMTime(opts) + fmt.Fprintf(w, "\nWatching tests + model for changes (run #%d; Ctrl-C to stop)...\n", gen) + + changed := false + for !changed { + select { + case <-sigCh: + fmt.Fprintln(w, "\nShutting down...") + return shutdown(last, nil) + case <-ticker.C: + if now := watchMTime(opts); now.After(baseline) { + changed = true + } + } + } + + fmt.Fprintln(w, "Change detected, rebuilding...") + start := time.Now() + + // Re-parse: a test may have been added, edited, or deleted. + reparsed, err := parseTestFiles(opts.TestFiles) + if err != nil { + fmt.Fprintf(w, " test files do not parse: %v\n", err) + continue + } + + // Report the new set BEFORE injecting: if the injection fails partway, + // cleanup must still know about the microflows that did land. Dropping a + // microflow that was never created is harmless; leaving one behind is not. + onInject(reparsed) + if err := reinjectTests(opts, injected, reparsed, w); err != nil { + fmt.Fprintf(w, " injecting tests: %v\n", err) + injected = reparsed + continue + } + injected = reparsed + + action, err := target.applyModelChange(opts.ProjectPath) + if err != nil { + // Not fatal: a build error is usually the edit that just happened, and + // the next save is likely to fix it. Report and keep watching. + fmt.Fprintf(w, " %v\n", err) + continue + } + fmt.Fprintf(w, " rebuilt and applied via %s in %s\n", action, time.Since(start).Round(time.Millisecond)) + } +} + +// finishFunc restores the project and decides the final result. runEndpointWatch +// takes it rather than owning cleanup so that every exit — a clean Ctrl-C, a +// dead runtime, a boot failure — goes through the same restore as the one-shot +// path. +type finishFunc func(result *SuiteResult, runErr error) (*SuiteResult, error) + +// watchMTime is the change signal: the newer of the test files' and the model's +// modification times. +// +// Both matter, and for different reasons. A test file changing means the +// assertions changed. The model changing means the code under test changed — +// which is the case a developer actually cares about, editing a microflow and +// wanting to know immediately whether it still passes. +func watchMTime(opts RunOptions) time.Time { + newest := docker.ProjectSourceMTime(opts.ProjectPath) + if t := testFilesMTime(opts.TestFiles); t.After(newest) { + newest = t + } + return newest +} + +// testFilesMTime is the newest modification time across the test paths, which +// may be individual files or directories. +// +// A directory is walked rather than stat'ed: on Linux a directory's own mtime +// changes when an entry is created or removed but *not* when an existing file is +// edited in place, which is the common case this loop exists to catch. +func testFilesMTime(paths []string) time.Time { + var newest time.Time + for _, p := range paths { + info, err := os.Stat(p) + if err != nil { + continue + } + if !info.IsDir() { + if info.ModTime().After(newest) { + newest = info.ModTime() + } + continue + } + filepath.WalkDir(p, func(path string, d os.DirEntry, err error) error { + if err != nil || d.IsDir() || !isTestFile(d.Name()) { + return nil //nolint:nilerr // an unreadable entry is not a change + } + if fi, err := d.Info(); err == nil && fi.ModTime().After(newest) { + newest = fi.ModTime() + } + return nil + }) + // Catch a deletion too: the directory's own mtime moves when an entry + // goes away, and the walk above cannot see a file that is no longer there. + if info.ModTime().After(newest) { + newest = info.ModTime() + } + } + return newest +} + +// reinjectTests updates the project's generated test microflows to match a +// re-parsed suite. +// +// CREATE OR REPLACE covers a test that was added or edited, but says nothing +// about one that was deleted: its microflow would linger and keep being invoked, +// reporting a stale pass for a test that no longer exists. So the flows for tests +// that are gone are dropped explicitly. +func reinjectTests(opts RunOptions, old, new *TestSuite, w io.Writer) error { + if drops := staleTestFlows(old, new); len(drops) > 0 { + fmt.Fprintf(w, " dropping %d removed test microflow(s)\n", len(drops)) + if err := runMDLCommands(opts.ProjectPath, drops); err != nil { + return err + } + } + return execMDLScript(opts.ProjectPath, GenerateTestFlows(new), "mxtest-flows-*.mdl") +} + +// staleTestFlows returns DROP statements for test microflows in old that no +// longer have a counterpart in new. +func staleTestFlows(old, new *TestSuite) []string { + if old == nil { + return nil + } + keep := make(map[string]bool, len(new.Tests)) + for _, tc := range new.Tests { + keep[testFlowName(tc)] = true + } + var drops []string + for _, tc := range old.Tests { + if flow := testFlowName(tc); !keep[flow] { + drops = append(drops, "DROP MICROFLOW "+flow) + } + } + return drops +} diff --git a/cmd/mxcli/testrunner/watch_test.go b/cmd/mxcli/testrunner/watch_test.go new file mode 100644 index 000000000..4566dc329 --- /dev/null +++ b/cmd/mxcli/testrunner/watch_test.go @@ -0,0 +1,200 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestValidateOptions(t *testing.T) { + tests := []struct { + name string + opts RunOptions + wantErr string + }{ + {name: "no watch is always fine", opts: RunOptions{}}, + {name: "watch with local", opts: RunOptions{Watch: true, Local: true}}, + { + name: "watch without local", + opts: RunOptions{Watch: true}, + wantErr: "--watch requires --local", + }, + { + name: "watch with the legacy runner", + opts: RunOptions{Watch: true, Local: true, LegacyRunner: true}, + wantErr: "--watch cannot be combined with --legacy-runner", + }, + { + name: "watch with skip-build", + opts: RunOptions{Watch: true, Local: true, SkipBuild: true}, + wantErr: "--watch cannot be combined with --skip-build", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := validateOptions(tt.opts) + if tt.wantErr == "" { + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + return + } + if err == nil { + t.Fatalf("expected an error containing %q", tt.wantErr) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("error = %q, want it to contain %q", err, tt.wantErr) + } + }) + } +} + +// writeTestFile writes a test file with a controlled mtime. +func writeTestFile(t *testing.T, path string, mtime time.Time) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir: %v", err) + } + if err := os.WriteFile(path, []byte("/** @test x */\n"), 0o600); err != nil { + t.Fatalf("write: %v", err) + } + if err := os.Chtimes(path, mtime, mtime); err != nil { + t.Fatalf("chtimes: %v", err) + } +} + +// TestTestFilesMTimeSeesAnEditInsideADirectory pins the reason the directory is +// walked rather than stat'ed: on Linux a directory's own mtime does not move +// when an existing entry is edited in place, which is the common case. +func TestTestFilesMTimeSeesAnEditInsideADirectory(t *testing.T) { + dir := t.TempDir() + old := time.Now().Add(-2 * time.Hour) + f := filepath.Join(dir, "a.test.mdl") + writeTestFile(t, f, old) + // Pin the directory itself to the old time, so only the file can move. + if err := os.Chtimes(dir, old, old); err != nil { + t.Fatalf("chtimes dir: %v", err) + } + + before := testFilesMTime([]string{dir}) + + edited := time.Now() + writeTestFile(t, f, edited) + if err := os.Chtimes(dir, old, old); err != nil { + t.Fatalf("chtimes dir: %v", err) + } + + after := testFilesMTime([]string{dir}) + if !after.After(before) { + t.Errorf("editing a file in a watched directory produced no change signal (before=%v after=%v)", before, after) + } +} + +func TestTestFilesMTimeAcceptsAFilePath(t *testing.T) { + dir := t.TempDir() + f := filepath.Join(dir, "a.test.mdl") + want := time.Now().Add(-time.Hour).Truncate(time.Second) + writeTestFile(t, f, want) + + if got := testFilesMTime([]string{f}); !got.Truncate(time.Second).Equal(want) { + t.Errorf("mtime = %v, want %v", got, want) + } +} + +// TestTestFilesMTimeIgnoresNonTestFiles keeps an unrelated file in the tests +// directory — a README, an editor swap file — from re-triggering the loop. +func TestTestFilesMTimeIgnoresNonTestFiles(t *testing.T) { + dir := t.TempDir() + old := time.Now().Add(-2 * time.Hour) + writeTestFile(t, filepath.Join(dir, "a.test.mdl"), old) + if err := os.Chtimes(dir, old, old); err != nil { + t.Fatalf("chtimes dir: %v", err) + } + before := testFilesMTime([]string{dir}) + + noise := filepath.Join(dir, "notes.md") + if err := os.WriteFile(noise, []byte("scratch"), 0o600); err != nil { + t.Fatalf("write: %v", err) + } + now := time.Now() + if err := os.Chtimes(noise, now, now); err != nil { + t.Fatalf("chtimes: %v", err) + } + // Hold the directory back so only the noise file could move the signal. + if err := os.Chtimes(dir, old, old); err != nil { + t.Fatalf("chtimes dir: %v", err) + } + + if after := testFilesMTime([]string{dir}); after.After(before) { + t.Error("a non-test file in the tests directory moved the change signal") + } +} + +// TestTestFilesMTimeSeesADeletion pins that removing a test is a change. The +// walk cannot see a file that is gone, so the directory's own mtime — which does +// move on a deletion — has to be folded in. +func TestTestFilesMTimeSeesADeletion(t *testing.T) { + dir := t.TempDir() + old := time.Now().Add(-2 * time.Hour) + keep := filepath.Join(dir, "a.test.mdl") + gone := filepath.Join(dir, "b.test.mdl") + writeTestFile(t, keep, old) + writeTestFile(t, gone, old) + if err := os.Chtimes(dir, old, old); err != nil { + t.Fatalf("chtimes dir: %v", err) + } + before := testFilesMTime([]string{dir}) + + if err := os.Remove(gone); err != nil { + t.Fatalf("remove: %v", err) + } + + if after := testFilesMTime([]string{dir}); !after.After(before) { + t.Errorf("deleting a test file produced no change signal (before=%v after=%v)", before, after) + } +} + +// TestStaleTestFlowsDropsRemovedTests pins the correctness hazard of re-running: +// CREATE OR REPLACE updates a test that changed but says nothing about one that +// was deleted, whose microflow would otherwise linger and keep reporting a pass. +func TestStaleTestFlowsDropsRemovedTests(t *testing.T) { + old := &TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}, {ID: "test_3"}}} + new := &TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}}} + + got := staleTestFlows(old, new) + want := []string{"DROP MICROFLOW MxTest.Test_test_3"} + if len(got) != len(want) || (len(got) > 0 && got[0] != want[0]) { + t.Errorf("got %q, want %q", got, want) + } +} + +func TestStaleTestFlowsNoneWhenSuiteGrew(t *testing.T) { + old := &TestSuite{Tests: []TestCase{{ID: "test_1"}}} + new := &TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}}} + + if got := staleTestFlows(old, new); len(got) != 0 { + t.Errorf("got %q, want no drops when tests were only added", got) + } +} + +// TestStaleTestFlowsDropsAllWhenEmptied covers deleting the last test in a file: +// every previously-injected flow has to come out. +func TestStaleTestFlowsDropsAllWhenEmptied(t *testing.T) { + old := &TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}}} + new := &TestSuite{} + + if got := staleTestFlows(old, new); len(got) != 2 { + t.Errorf("got %d drops %q, want 2", len(got), got) + } +} + +func TestStaleTestFlowsHandlesNoPriorSuite(t *testing.T) { + if got := staleTestFlows(nil, &TestSuite{Tests: []TestCase{{ID: "test_1"}}}); got != nil { + t.Errorf("got %q, want nil for a first injection", got) + } +} diff --git a/docs-site/src/reference/capabilities.md b/docs-site/src/reference/capabilities.md index 1f6262931..34367323d 100644 --- a/docs-site/src/reference/capabilities.md +++ b/docs-site/src/reference/capabilities.md @@ -126,7 +126,7 @@ Everything mxcli can do, organized by use case. | Stdin piping | `echo "CMD" \| mxcli -p app.mpr` | Quiet mode, no prompts | | Docker build | `mxcli docker build -p app.mpr` | Build MDA in container | | Docker check | `mxcli docker check -p app.mpr` | Validate in container | -| Testing | `mxcli test tests/ -p app.mpr` | `.test.mdl` / `.test.md` | +| Testing | `mxcli test tests/ -p app.mpr [--local] [--watch] [--attach]` | `.test.mdl` / `.test.md`; `--local` needs no Docker, `--watch` keeps the runtime warm (~2s per run), `--attach` runs against an app already up | | SARIF output | `mxcli lint --format sarif` | For CI integration | | New project | `mxcli new --version X.Y.Z` | Create project from scratch with all tooling | | Init project | `mxcli init` | Set up `.claude/` with skills | diff --git a/docs-site/src/tools/run-local.md b/docs-site/src/tools/run-local.md index 43649d900..d5de0f008 100644 --- a/docs-site/src/tools/run-local.md +++ b/docs-site/src/tools/run-local.md @@ -79,6 +79,7 @@ so structural changes need a restart; behavioural changes do not. | `--screenshot-url` | app root | Page to shoot: full URL, or a path relative to the app root (e.g. `/p/customers`). Repeat for a multi-page set. | | `--screenshot-user` / `--screenshot-password` | — | Log in once (Mendix form auth) and reuse the session, so pages behind login render authenticated | | `--runtime-log` | `/.mxcli/runtime.log` | Runtime log file — JVM stdout/stderr **and** the application log (server stack traces + microflow `LOG` output); `-` disables | +| `--test-endpoint` | off | Host mxcli's token-guarded test endpoint so [`mxcli test … --attach`](running-tests.md) runs a suite against this app with no boot of its own. Installed before the boot (the handler registers from after-startup, so it cannot be added to a running app); your project's own after-startup microflow is chained, not displaced. Removed on exit. Tests then use **this app's database** | | `--debug` | off | Enable the microflow debugger at boot; then use [`mxcli debug`](debug-microflows.md) from another terminal. Behaviour-neutral until a breakpoint is set | | `--debug-pass` | `mxdebug` | Debugger password when `--debug` is set | | `--metrics` | off | Register a Prometheus meter registry; metrics served at `http://127.0.0.1:/prometheus` | diff --git a/docs-site/src/tools/running-tests.md b/docs-site/src/tools/running-tests.md index 14eeb1a4f..4d45072d2 100644 --- a/docs-site/src/tools/running-tests.md +++ b/docs-site/src/tools/running-tests.md @@ -4,24 +4,27 @@ The `mxcli test` command executes test files and reports results. ## Prerequisites -Running tests requires **Docker** for Mendix runtime validation. The test runner uses: +Tests need a Mendix runtime to execute against. There are two ways to get one, +and **Docker is only needed for the first**: -- `mx create-project` to create a fresh blank Mendix project -- `mx check` to validate the project after applying MDL changes +- **Docker** — the container path. Requires a running Docker daemon. +- **`--local`** — mxcli's own runtime, no daemon involved. It uses its own ports + (8081/8091) and its own `_test` database, so a `mxcli run --local` + dev loop can keep serving the same project while tests run. -The `mx` binary is located at: +`--local` also downloads what it needs on first use. To pre-cache it: + +```bash +mxcli setup mxbuild -p app.mpr +``` + +The `mx` binary, when you need it directly: | Environment | Path | |-------------|------| | Dev container | `~/.mxcli/mxbuild/{version}/modeler/mx` | | Repository | `reference/mxbuild/modeler/mx` | -To auto-download mxbuild for the project's Mendix version: - -```bash -mxcli setup mxbuild -p app.mpr -``` - ## Basic Usage ```bash @@ -35,12 +38,61 @@ mxcli test tests/sales.test.mdl -p app.mpr mxcli test tests/integration.test.md -p app.mpr ``` -## Test Execution Flow +## Choosing a mode + +| | Boot cost per run | Database | Needs Docker | +|---|---|---|---| +| `mxcli test …` | container restart | the container's | yes | +| `--local` | ~30s | `_test` | no | +| `--local --watch` | ~30s once, then ~2s | `_test` | no | +| `--attach` | none | **the running app's** | no | + +```bash +# No Docker daemon needed +mxcli test tests/ -p app.mpr --local + +# Keep the runtime warm; re-runs on every test or model change (Ctrl-C to stop) +mxcli test tests/ -p app.mpr --local --watch + +# Attach to an app you already have running — no boot at all +mxcli run --local --test-endpoint -p app.mpr # terminal 1 +mxcli test tests/ -p app.mpr --attach # terminal 2 +``` + +`--watch` is the everyday loop: edit a test *or* the microflow under test, and +the verdict lands in about two seconds. + +`--attach` skips even the first boot, at one cost worth knowing: the tests run +against the running app's database rather than a scratch one, so they can leave +data behind in the app you are looking at. It needs the dev loop to have been +started with `--test-endpoint`, because the endpoint's handler is registered by +the after-startup microflow and cannot be added to an app that is already up. + +## How tests execute + +**`--local` — the test endpoint.** One microflow is generated per test, plus a +Java action that registers a token-guarded HTTP endpoint. The app boots once; +startup only registers the endpoint and runs no tests. Each test is then invoked +by name over HTTP and returns its verdict in the response. + +Two consequences when reading a failing run: + +- A test that throws fails **only itself** and is reported as an error with the + root-cause message; the next test still runs. +- Results are **returned**, not recovered from the runtime log. + +The endpoint executes microflows under a system context, so it is gated: it is +not registered at all without a per-run token in the runtime's environment, +every request must present that token, non-loopback callers are refused, and it +will only ever invoke the generated `MxTest.Test_*` microflows. The token is +never written into your project. + +**Docker — the after-startup runner.** The whole suite is compiled into the +project's after-startup microflow, the container is restarted, and results are +parsed out of its log. `--legacy-runner` selects this on a local run too. -1. **Create project** -- A fresh Mendix project is created in a temporary directory using `mx create-project` -2. **Execute MDL** -- The test script is executed against the fresh project using `mxcli exec` -3. **Validate** -- `mx check` validates the resulting project for errors -4. **Report** -- Results are reported with pass/fail status per test case +Both paths restore the project when they finish, and both report loudly if that +restore fails — a modified project must never read as a clean pass. ## Isolated Testing Pattern diff --git a/docs-site/src/tools/testing.md b/docs-site/src/tools/testing.md index 338ff805b..65772d97a 100644 --- a/docs-site/src/tools/testing.md +++ b/docs-site/src/tools/testing.md @@ -13,7 +13,7 @@ The testing framework supports two test file formats: ## Prerequisites -Running tests requires **Docker** for Mendix runtime validation. The test runner: +Tests execute against a real Mendix runtime. **Docker is one way to get one, not the only one** — `--local` uses mxcli's own runtime with no daemon, and is the faster path (see [Running Tests](running-tests.md) for the mode comparison and the `--watch` / `--attach` loops). On the Docker path the test runner: 1. Creates a fresh Mendix project using `mx create-project` 2. Executes the MDL test script against the project diff --git a/docs/15-testing/SPIKE_test_endpoint_request_handler.md b/docs/15-testing/SPIKE_test_endpoint_request_handler.md new file mode 100644 index 000000000..c54ce9140 --- /dev/null +++ b/docs/15-testing/SPIKE_test_endpoint_request_handler.md @@ -0,0 +1,228 @@ +# Spike: custom request handler as a re-invokable test entry point + +**Status**: **shipped** for `mxcli test --local` — see `cmd/mxcli/testrunner/endpoint.go`. +Measured end-to-end on Mendix **11.13.0**. +**Reproduce the bare spike with**: [`mdl-examples/spikes/test-endpoint-request-handler.mdl`](../../mdl-examples/spikes/test-endpoint-request-handler.mdl) + +> The "open issues" at the bottom were what stood between the spike and the +> implementation. The token gate, the loopback check, the test-namespace +> restriction and the javasource cleanup are all now in `endpoint.go`; the +> remaining unresolved items are called out as still open. + +## Question + +`mxcli test --local` triggers tests from the project's **after-startup microflow**. +That is a boot hook, so re-running a suite requires a full runtime restart by +construction. Can a Java custom request handler give us a *re-invokable* entry +point instead, so a re-run costs an HTTP round trip rather than a restart? + +## Answer + +Yes, and it is better than the `check_health` idea it replaces — because +`Core.getMicroflowNames()` lets the handler resolve microflows **by name at +request time**. The Java is written once and never regenerated; tests can be +added, edited, and removed with no change to it. + +## What was built + +A single Java action, `MxTest.RegisterTestEndpoint`, authored inline from MDL +(`CREATE JAVA ACTION … AS $$ … $$`), registering one handler: + +```java +Core.addRequestHandler("mxtest/", new RequestHandler() { … }); +``` + +It is called once from a two-line after-startup microflow. After-startup is +still used — but only to *register the endpoint*, never to run tests. + +| Route | Behaviour | +|---|---| +| `GET /mxtest/list?prefix=` | Test discovery from `Core.getMicroflowNames()` | +| `GET /mxtest/run?mf=Module.Flow` | `Core.microflowCall(mf).execute(Core.createSystemContext())` | + +Results come back as JSON — return value, wall time, and on failure the +**root-cause** exception message. + +## Measurements + +All on the same machine, same project, Mendix 11.13.0, Postgres local. + +| Operation | Time | +|---|---| +| Cold boot to first test invocable (**today's cost per re-run**) | **30.55s** | +| Re-run whole suite, no model change (4 tests, 4 HTTP calls) | **0.084s** | +| Edit a test → watcher rebuild → hot reload → new result over HTTP | **4.29s** | +| Single test invocation, in-process | 0.6–26ms | + +So an unchanged re-run goes from ~30s to **0.08s (~360×)**, and an +edit-then-re-run from ~30s to **~4.3s (~7×)** — the 4.3s being almost entirely +the existing `--watch` rebuild, not the test mechanism. + +## The load-bearing finding: the handler survives `reload_model` + +This is what makes the warm loop actually work, and it was not obvious. + +- After-startup does **not** re-run on `reload_model`. Verified by grepping the + runtime log: exactly two `MxTest request handler registered` lines across two + *boots*, and none after the reload. +- The runtime **JVM PID is unchanged** across the reload (31312 before and + after), and `mxcli run --watch` reported `build #2 applied via reload`, not a + restart. +- The handler object registered by the *old* model resolves the *new* model's + microflows correctly. Proven with two probes in one reload: an **edited** + test returned its new value (`2+2=4` → `40+2=42`), and a test **created after + boot** appeared in `/mxtest/list` and ran — with no restart. + +## Consequences for the test runner + +Beyond the speed, three structural problems in `cmd/mxcli/testrunner/` dissolve: + +1. **The monolithic runner microflow goes away.** `generator.go` currently + compiles every test into one microflow, regex-renaming variables with `_N` + suffixes to avoid collisions. With per-name invocation each test is its own + microflow, so `--filter` and single-test runs are free and one throwing test + can no longer end the run. +2. **Results stop being scraped from JVM stdout.** They are the HTTP response + body. `results.go`'s log parsing is no longer on the critical path. +3. **A failing test stops being a failed boot.** Verified: a test throwing + `MendixRuntimeException` returns HTTP 200 with `ok:false` and the root-cause + message, and the runtime stays up and serves the next test. Compare + `runner_local.go:58-71`, which has to special-case boot failure today. + +## Issues found by the spike, and how they were resolved + +- **The endpoint was unauthenticated.** Verified in the spike: `curl` with no + cookies and no session executed a microflow. **Resolved** — four gates, each + verified against a live 11.13.0 runtime: + + | Guard | Verified behaviour | + |---|---| + | No `MXCLI_TEST_TOKEN` in the environment | Handler not registered; `/mxtest/list` → 404 | + | Missing / wrong `X-MxTest-Token` | 401 (constant-time compare) | + | Non-loopback caller | 403 | + | `mf` outside `MxTest.Test_*` | 403 | + + The token is generated per run and passed through the runtime's **environment** + (`LocalRuntimeOptions.Env`), never written into the project — so a failed + cleanup cannot leave a live credential in `javasource/`. + +- **`/list` disclosed the whole app.** Found only by probing the live runtime: + with no `prefix` the handler returned every microflow in the app, + `Administration.*` included. **Resolved** — the prefix is clamped to + `MxTest.Test_`; a caller-supplied prefix can only narrow further. The endpoint + will not run those microflows, so it must not enumerate them either. + +- **Path dispatch was loose** — anything that was not `list` was treated as + `run`. **Resolved**: exact matches, anything else 404. + +- **`DROP JAVA ACTION` leaves the `.java` file behind.** The model document goes; + the generated source in `javasource/mxtest/` is not the model's to delete. + **Resolved** — `removeGeneratedJavaSource` removes it, non-fatally. + +### Still open + +- **There is no `Core.removeRequestHandler`.** The API only offers + `addRequestHandler`, so the handler cannot be unregistered for the life of the + JVM. This is why registration is gated rather than reversed. Re-registering + the same path is still unexercised — after-startup does not re-run on + `reload_model`, so nothing in the current design hits it. +- **Test parameters and setup/teardown.** Only no-argument microflows are + invoked. `MicroflowCallBuilder.withParams(Map)` exists, and + `inTransaction(boolean)` looks directly relevant to rolling back a test's + database writes — the `@cleanup rollback` annotation is parsed but not yet + honoured by either mechanism. +- **The Docker path still uses after-startup.** The endpoint needs to hand the + runtime a secret through its environment and to be reached on loopback, + neither of which is wired through docker-compose. No Docker daemon was + available to verify a change there, so it was left alone rather than shipped + untested. +- ~~`--attach` to an already-running `run --local`~~ — **shipped**, see below. + +## The warm loop, realised (`--watch`) + +`mxcli test --local --watch` keeps the runtime and the build server up and +re-runs the suite on every change to a test file or to the model. Measured on +the same 11.13.0 app: + +| | | +|---|---| +| First run (cold boot) | ~30s | +| Edit a test → verdict on screen | **~2.0s** | +| Edit a microflow under test → verdict on screen | **~2.1s** | +| The tests themselves | 20–70ms | + +Every re-run in the session applied via **reload**, not restart — the property +the spike established. Verified live across a session that edited a test, +deleted a test, added a test, and changed the microflow under test. + +Two hazards this loop has that the `run --local` dev loop does not: + +1. **The runner writes to the project it is watching.** Injecting the test + microflows moves the very mtime being polled, so the baseline is taken after + the injection and rebuild settle. Getting it wrong is an infinite rebuild + loop; verified by idling a session and confirming the run counter does not + advance. +2. **The injected set changes during the session.** Cleanup drops what is + *currently* injected, not what was injected at boot — otherwise a test added + mid-session is left in the user's project. A deleted test's microflow is + dropped explicitly, since `CREATE OR REPLACE` says nothing about removal and + a lingering flow would keep reporting a stale pass. + +## `--attach`: no boot at all + +`mxcli test --attach` runs against an app already up, skipping the boot entirely. +Measured on the same 11.13.0 app: **2.83s** for the first attached run and +**2.30s** for a repeat, against ~30s cold — with the dev app still serving +throughout. + +### Why it has to be cooperative + +The obvious reading of "attach to a running app" does not work, and the reason is +worth recording because it constrains the design completely: + +- The handler is registered by the **after-startup microflow**, which runs only + at boot. It cannot be added to an app that is already up. +- Its token comes from the **runtime's environment**, which a second process + cannot change either. + +So the app has to opt in *before* it boots: `mxcli run --local --test-endpoint`. +That is also the right place for the decision, since hosting the endpoint means +the developer's own app carries a microflow-executing endpoint and tests will +write to the database they are looking at. + +What a second process *can* do, and does, is drive the dev loop's **serve server +and admin API** over loopback — both are plain HTTP. So an attach applies its own +injections deterministically instead of waiting to see whether someone else's +`--watch` noticed; `--attach` does not require the dev loop to be watching. + +### The handshake + +`run --local --test-endpoint` publishes `/.mxcli/test-endpoint.json` +(mode 0600, written-then-renamed) carrying the app/admin/serve ports, the +endpoint token, the admin password, and its own PID. `--attach` reads it and +refuses a stale one by checking the PID — a dev loop killed with SIGKILL leaves +the file behind, and without the check that surfaces much later as a confusing +connection error. + +The project's own after-startup microflow is **chained**, not displaced, so the +dev app still seeds its data and does whatever else it does at boot. + +### Two bugs the live run caught that review had not + +1. **The admin API and the endpoint use different secrets.** The first attempt + passed the endpoint token to the M2EE admin API, which failed with + `Authentication failed` — *after* the test microflows had already been + injected. The handshake now carries the admin password separately. +2. **`DROP JAVA ACTION` leaves the generated `.java` behind** (found earlier, in + the same family): the model document is the model's, the source file is not. + +### Ownership boundary + +An attach adds and removes only its own test microflows. The endpoint, the +after-startup setting and the `MxTest` module belong to the hosting dev loop and +are removed when *it* exits. Verified live: after an attached run the test +microflows were gone, `MxTest.RegisterEndpoint` was still installed, and the app +was still serving HTTP 200. + +A change needing a runtime restart (a new entity or association) is refused +rather than half-applied — that runtime belongs to the other process. diff --git a/mdl-examples/spikes/test-endpoint-request-handler.mdl b/mdl-examples/spikes/test-endpoint-request-handler.mdl new file mode 100644 index 000000000..726f345c2 --- /dev/null +++ b/mdl-examples/spikes/test-endpoint-request-handler.mdl @@ -0,0 +1,185 @@ +-- ============================================================================ +-- SPIKE: custom request handler as a re-invokable test entry point +-- ============================================================================ +-- Goal: prove that a Java custom request handler registered once at boot can +-- invoke ANY microflow by name over HTTP, so re-running tests costs one HTTP +-- round trip instead of a full runtime restart. +-- ============================================================================ + +create module MxTest; + +-- ---------------------------------------------------------------------------- +-- The handler. Registered once, at boot, from the after-startup microflow. +-- Generic: it resolves the microflow by name from Core.getMicroflowNames(), +-- so adding/renaming/removing tests never requires touching this Java code. +-- ---------------------------------------------------------------------------- +/** Registers the /mxtest/ request handler. Called once from after-startup. */ +create java action MxTest.RegisterTestEndpoint() returns boolean +as $$ +final com.mendix.logging.ILogNode log = com.mendix.core.Core.getLogger("MxTest"); + +com.mendix.core.Core.addRequestHandler("mxtest/", new com.mendix.externalinterface.connector.RequestHandler() { + + private String esc(String s) { + if (s == null) return "null"; + StringBuilder b = new StringBuilder("\""); + for (int i = 0; i < s.length(); i++) { + char c = s.charAt(i); + switch (c) { + case '"': b.append("\\\""); break; + case '\\': b.append("\\\\"); break; + case '\n': b.append("\\n"); break; + case '\r': b.append("\\r"); break; + case '\t': b.append("\\t"); break; + default: + if (c < 0x20) b.append(String.format("\\u%04x", (int) c)); + else b.append(c); + } + } + return b.append('"').toString(); + } + + @Override + protected void processRequest(com.mendix.m2ee.api.IMxRuntimeRequest request, + com.mendix.m2ee.api.IMxRuntimeResponse response, + String path) throws Exception { + response.setContentType("application/json"); + java.io.Writer out = response.getWriter(); + + java.util.Set known = com.mendix.core.Core.getMicroflowNames(); + + // GET /mxtest/list?prefix=MxTest.Test_ -> discover tests at runtime + if (path != null && path.startsWith("list")) { + String prefix = request.getParameter("prefix"); + java.util.List names = new java.util.ArrayList(); + for (String n : known) { + if (prefix == null || n.startsWith(prefix)) names.add(n); + } + java.util.Collections.sort(names); + StringBuilder b = new StringBuilder("{\"microflows\":["); + for (int i = 0; i < names.size(); i++) { + if (i > 0) b.append(','); + b.append(esc(names.get(i))); + } + b.append("]}"); + out.write(b.toString()); + out.flush(); + return; + } + + // GET /mxtest/run?mf=Module.Microflow + String mf = request.getParameter("mf"); + if (mf == null || mf.isEmpty()) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.BAD_REQUEST); + out.write("{\"error\":\"missing mf parameter\"}"); + out.flush(); + return; + } + if (!known.contains(mf)) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.NOT_FOUND); + out.write("{\"error\":\"unknown microflow\",\"mf\":" + esc(mf) + "}"); + out.flush(); + return; + } + + long t0 = System.nanoTime(); + com.mendix.systemwideinterfaces.core.IContext ctx = com.mendix.core.Core.createSystemContext(); + Object result = null; + String error = null; + try { + result = com.mendix.core.Core.microflowCall(mf).execute(ctx); + } catch (Throwable t) { + Throwable root = t; + while (root.getCause() != null && root.getCause() != root) root = root.getCause(); + error = root.getClass().getName() + ": " + root.getMessage(); + log.warn("test microflow " + mf + " threw: " + error); + } + long micros = (System.nanoTime() - t0) / 1000L; + + StringBuilder b = new StringBuilder("{"); + b.append("\"mf\":").append(esc(mf)); + b.append(",\"ok\":").append(error == null); + b.append(",\"durationMicros\":").append(micros); + b.append(",\"result\":").append(result == null ? "null" : esc(String.valueOf(result))); + if (error != null) b.append(",\"error\":").append(esc(error)); + b.append('}'); + out.write(b.toString()); + out.flush(); + } +}); + +log.info("MxTest request handler registered at /mxtest/"); +return true; +$$; +/ + +-- ---------------------------------------------------------------------------- +-- Boot hook: the ONLY thing that runs at startup. Registration, not tests. +-- ---------------------------------------------------------------------------- +/** Registers the test endpoint at boot. */ +create microflow MxTest.AfterStartup () +returns boolean as $Registered +begin + $Registered = call java action MxTest.RegisterTestEndpoint(); + return $Registered; +end; +/ + +alter settings model AfterStartupMicroflow = 'MxTest.AfterStartup'; + +-- ---------------------------------------------------------------------------- +-- Sample tests. Each is its own microflow -> per-test invocation and filtering +-- come for free, and a throwing test is an HTTP 200 with ok:false, not a +-- failed runtime boot. +-- ---------------------------------------------------------------------------- +create persistent entity MxTest.Widget ( + Label: string(100), + Amount: integer +); +/ + +/** Passing test: creates and reads back an object. */ +create microflow MxTest.Test_CreateWidget () +returns string as $Outcome +begin + declare $Outcome String = 'FAIL amount was not 41'; + $W = create MxTest.Widget (Label = 'alpha', Amount = 41); + commit $W; + if $W/Amount = 41 then + set $Outcome = 'PASS created widget with amount 41'; + end if; + return $Outcome; +end; +/ + +/** Passing test: pure arithmetic, no database. */ +create microflow MxTest.Test_Arithmetic () +returns string as $Outcome +begin + declare $Outcome String = ''; + declare $Sum Integer = 0; + set $Sum = 2 + 2; + if $Sum = 4 then + set $Outcome = 'PASS 2+2=4'; + else + set $Outcome = 'FAIL arithmetic broken'; + end if; + return $Outcome; +end; +/ + +/** Throws, to prove a failing test is reported not fatal. */ +create java action MxTest.Boom() returns string +as $$ +throw new com.mendix.systemwideinterfaces.MendixRuntimeException("deliberate failure from Test_Failing"); +$$; +/ + +/** Failing test: propagates a Java exception out of the microflow. */ +create microflow MxTest.Test_Failing () +returns string as $Outcome +begin + $Outcome = call java action MxTest.Boom(); + return $Outcome; +end; +/