Docs + two behavioural bugs found while evaluating a Cucumber → Varar migration

[aslakhellesoy]

Findings from evaluating a real Cucumber → Varar migration (a SvelteKit/Cloudflare-Workers app with 12 .feature files, ~31 scenarios, 103 step definitions) against v0.6.1 and the docs at varar.dev.

The evaluation went well — Varar's vitest plugin runs .feature files unchanged inside @cloudflare/vitest-pool-workers, which is a great story. Most of what follows is documentation, but two items are behavioural bugs I hit while reading the docs and expecting something else.

---

1. Bug: a sensor with ≥1 slot that returns undefined silently passes

core/dist/execute.js guards the comparison with if (!ex.rowChecks && returned !== undefined). So a one-slot sensor whose handler returns undefined performs no comparison at all and the example goes green.

// steps
const { stimulus, sensor } = steps(() => ({ name: '', total: 0 }))
stimulus('I set the name to {string}', (state, name) => ({ ...state, name }))
stimulus('I add {int}', (state, n) => ({ total: state.total + n }))  // drops `name`
sensor('The name is {string}', (state) => state.name)               // -> undefined

I set the name to "Ada". I add 2. The name is "Ada".

✓ I set the name to "Ada". I add 2. The name is "Ada" (1ms)
1 example, 1 passed, 0 failed

state.name is undefined at that point (confirmed by throwing from the sensor: STATE={"total":2}), yet the document claims "Ada" and the run is green.

The reference table only defines undefined as the return for the 0-slot case ("throw to fail, return nothing to pass"). For 1+ slots it says "the slot's value, bare" — it doesn't say undefined opts out. This is exactly the silent-coverage-loss failure mode the drift-detection page argues against, and a plain typo in a property access reaches it. Suggestion: a sensor with ≥1 slot returning undefined should be a ReturnShapeError, and the sensors reference should state the rule either way.

2. Bug: StimulusFn is typed Partial<C> but the runtime replaces state

// varar/dist/internal.d.ts
type StimulusFn<C, Custom> = <E extends string>(
  expression: E,
  handler: (state: DeepReadonly<C>, ...args) => Partial<C> | void | Promise<Partial<C> | void>,
) => void

execute.js is explicit that this is not a patch:

A stimulus REPLACES state ... There is no merge — a return with fewer keys shrinks the state.

So Partial<C> type-checks the one thing the semantics forbid, and everything downstream keeps the full DeepReadonly<C> type while the value has lost keys — silently, as in §1. It should be C | void. (The other ports get this right: the Rust/C#/Go samples carry a literal // FULL-REPLACEMENT state comment.)

The docs disagree with themselves here too:

• Stimuli: "returning the complete next state — full replacement ... a return that leaves a field out drops it rather than preserving it". ✅
• Your first spec from scratch: "The stimulus drives the software. It matches ... computes, and returns a patch to the state." ❌ — this is the first place a new user meets stimulus.

---

3. Undocumented: scannerPlugins, and that .feature files already run

gherkinTables / gherkinDocStrings and the scannerPlugins key of varar.config.json appear on no page of varar.dev (grepped all 20 URLs in the sitemap). They're only discoverable by reading @varar/core's dist.

This matters because Varar for Cucumber users says:

Our goal is to make Varar capable of running existing .feature files without any change to them. ... TODO: Finish the adapter implementation and test it extensively

…which reads as "not yet possible". But it already substantially works today. With:

{
  "docs": { "include": ["src/features/**/*.feature"], "exclude": [] },
  "steps": ["src/features/**/*.steps.ts"],
  "scannerPlugins": ["gherkinTables", "gherkinDocStrings"]
}

an untouched .feature file runs green: Feature:/Rule: lines fall out as prose, Given/When/Then/And keywords are narration the matcher ignores, each Example: becomes exactly one Varar example (its contiguous step lines are one paragraph), and indented Gherkin tables attach as data tables. Across the 12 real feature files the planner produced one example per scenario, as intended.

Worth a how-to page ("Run your existing .feature files"), plus a reference entry for scannerPlugins. Two caveats belong in it:

• Background: has no equivalent. It's a blank-line-separated paragraph, so it becomes its own example with its own fresh state rather than running before each scenario. It has to be inlined into each scenario by hand.
• Example names change. A Varar example is named after its whole paragraph, so a scenario is named Example: Simple basket Given the following users have been imported: — which breaks vitest -t "<scenario name>" filtering that Cucumber users rely on.

4. Undocumented: the whole drift-acceptance workflow

Examples › Drift detection explains drift at length and says the run "fails until you explicitly acknowledge the drift" — but never says how. Nowhere on the site do these appear:

• varar.lock.json — the baseline file (it's described only as "the recorded baseline"). Not named, not shown, no guidance on committing it.
• varar run --update — only in varar --help.
• VAR_UPDATE=1 / VAR_UPDATE=true — the only way to accept drift under the vitest plugin (vitest/dist/runtime.js reads process.env.VAR_UPDATE), since there's no --update flag there. Run specs through vitest doesn't mention drift at all.

Also note the env var is VAR_UPDATE, not VARAR_UPDATE — see §7.

5. varar init doesn't work out of the box (tutorial is unrunnable as written)

Following Get started on your computer verbatim in a fresh npm init -y project on Node v22.13.1:

$ pnpm add -D @varar/varar @varar/cli
$ npx varar init
created varar.config.json
created varar-examples/deep-thought.md
created varar-examples/steps/deep-thought.steps.ts
$ npx varar run
varar: Unknown file extension ".ts" for .../varar-examples/steps/deep-thought.steps.ts

With NODE_OPTIONS=--experimental-strip-types:

varar: Cannot use import statement outside a module

It only runs after also adding "type": "module" to package.json. Two things scaffolded by varar init itself (a .ts step file, ESM import) are what break it, and the tutorial's very next block shows a passing run. Options: have init set "type": "module", have the CLI register a TS loader (or detect and print the fix), and state the Node baseline in the prerequisites.

6. run-with-vitest omits things you need in a non-trivial config

• The plugin's config() hook overwrites test.include and test.exclude from varar.config.json. Fine and desirable, but surprising — worth saying, especially for the test.projects layout where you scope the plugin to one project so it doesn't clobber your unit-test globs. (It does work correctly there; I verified a projects: [unit, features] config.)
• No mention of scannerPlugins (§3) or drift (§4).
• "Varar has no lifecycle hooks of its own. Use vitest's native beforeEach / afterEach" — but afterEach has no access to the example's state, and the state factory is per-step-file, so there's no documented way to tear down something the example put in state (a spawned workflow, a created record). Migrating a Cucumber After(world => ...) hook means re-deriving everything from the outside. Worth calling out explicitly, since it's a real constraint rather than an oversight.

7. var vs varar naming is inconsistent across docs, API and code

The rename looks incomplete, and it's confusing when the CLI, the config file and the package are the three things a newcomer types first:

Where	Says
Cucumber page	"a file is a spec iff it matches the globs in var.config.ts" — stale; it's varar.config.json everywhere else
Most pages	"the var CLI", "the JVM ports have no var CLI yet" — the binary is varar
Python sample	from **var** import steps (package is pytest-varar)
Rust sample	use **var**::{Handler, Steps, Value} (crate is varar-cargotest)
vitest how-to	import **varPlugin** from '@varar/vitest', **Var**ResultsReporter
Runtime	**VAR_UPDATE** env var
Page slugs	/explanation/**var**-overview/, /explanation/**var**-for-cucumber-users/, /tutorials/try-**var**/

Relatedly, @varar/core/dist/config.js still ships a dead loadVarConfig that looks for var.config.ts / var.config.js / var.config.mjs with a vars: key — superseded by @varar/config, and the likely source of the stale doc reference above.

8. Small stuff

• Examples: "matches those sentences against your registered stimuli and sensos" → sensors.
• first-spec: "Steps come in roles ... (There's also context, for setting up the starting state — you'll meet it in bigger specs.)" — there is no context role; the API and the Cucumber page both say there are exactly two. If this refers to the steps(factory) state factory, it shouldn't be called a role.
• Published @varar/core and @varar/vitest both export VERSION = '0.0.0' at 0.6.1 — the version stamp isn't being replaced at build time.
• varar lint is documented only in --help, not on the site. Its behaviour ("a sentence that matches nothing is prose, so nothing is reported") is correct by design but is the single biggest expectation gap for a Cucumber user, who expects an "undefined step" list with snippets. Worth an explicit note.

---

Happy to send PRs for any of these — say which you'd like split out into separate issues.

[aslakhellesoy]

Follow-up: I went ahead and actually ported one feature file's step definitions (4 steps, real D1, real workflows, running under @cloudflare/vitest-pool-workers). It works, and the cell-anchored failure output is a clear improvement over the hand-rolled table differ it replaced. Two further findings, both sharper than what I wrote above.

9. Bug: DeepReadonly<C> recurses into class instances, but deepFreeze doesn't

deepFreeze is explicit and correct about this:

Class instances — any value whose prototype is not Object.prototype, null, or an array — are left live and NOT recursed into, so a stateful collaborator held in state (a Library, page object, DB client, the system under test, Date, Map, Set, …) keeps working through its mutating methods.

DeepReadonly has no matching escape hatch:

type DeepReadonly<T> = T extends (...args: never[]) => unknown ? T
  : T extends ReadonlyArray<infer U> ? ReadonlyArray<DeepReadonly<U>>
  : T extends object ? { readonly [K in keyof T]: DeepReadonly<T[K]> }
  : T

So the type and the runtime disagree about exactly the case the runtime went out of its way to support. Holding a Kysely instance in state — the DB-client case the deepFreeze comment names — mapped-types it into a structurally different object with its #private brands stripped:

Argument of type '{ readonly schema: ...; ... }' is not assignable to parameter of type 'Kysely<DB>'.
  ... is missing the following properties from type 'Kysely<DB>': #private, #private

Five errors, all of that shape. At runtime everything is fine; it's purely that the state a stimulus receives is untypeable once it contains a class instance.

Workaround, in case it's useful for the docs: DeepReadonly passes functions through untouched, so holding the collaborator behind a thunk restores the type with no cast.

type World = {
  readonly adapters: () => Adapters   // not `readonly adapters: Adapters`
  readonly basketId: number | undefined
}

But the real fix is presumably to mirror deepFreeze's rule in the type — stop recursing when the prototype isn't Object.prototype or an array. That's awkward to express in TypeScript, though T extends { constructor: Function } ? T : … gets close for most cases.

10. Drift detection is unreachable when the vitest adapter is the only way to run your specs

These interlock into a dead end:

• VarResultsReporter writes per-run results to .var/<spec>.json — it never writes varar.lock.json.
• varar.lock.json is written only by @varar/cli's runRun under --update / VAR_UPDATE.
• The runtime's drift gate no-ops when there's no baseline (if (ports.baseline)).

So the baseline can only be created by the CLI, which executes specs in Node. My steps need workerd:

$ npx varar run
varar: Only URLs with a scheme in: file, data, and node are supported by the default ESM loader. Received protocol 'cloudflare:'

There is therefore no way to record a baseline for these specs, and drift detection is permanently inert — silently, since a missing baseline is indistinguishable from a clean one. That's the exact failure mode the drift-detection docs argue is the worst one a test system can have.

This applies to anyone whose steps need a runtime the CLI can't host: Cloudflare Workers, but equally a browser/jsdom environment, or any vitest environment other than node.

Two things would close it: have VarResultsReporter (or a --update-equivalent under vitest) write varar.lock.json from the .var/*.json it already produces, and have the runtime warn rather than silently skip when a spec has no baseline entry.

---

For context on §8's VERSION = '0.0.0' — worth checking, since a version stamp that never updates would also make any future "your lockfile was written by a different Varar" check unreliable.

[aslakhellesoy]

Thanks for the exceptionally thorough report — nearly all of it is now on main. Both behavioural bugs are fixed across all seven language ports (not just TypeScript), and the docs have been reworked.

Behavioural bugs

• §1 — sensor with ≥1 slot returning undefined silently passes. Now a ReturnShapeError in every port (a sensor with N slot(s) must return one value per slot, got nothing), pinned by tests in each port. The same silent-pass hole on the header-bound row path is closed too. 80860828
• §2 — StimulusFn typed Partial<C>. Now C | void, so a partial return fails to type-check instead of silently shrinking state. first-spec's "returns a patch" wording is corrected. a140d599

Docs & tooling

• §3 — scannerPlugins / .feature files undocumented. New varar.config.json reference and a new how-to, Run your existing .feature files, including the Background: and example-name caveats. The "TODO: finish the adapter" passage is gone — it runs today. 894423d3
• §4 — drift-acceptance workflow undocumented. varar.lock.json is now named and its commit guidance given; varar run --update and the env var are documented, including that the vitest plugin only reads the baseline. 309a74f7
• §5 — varar init unrunnable out of the box. init now sets "type": "module" when absent, and varar run prints the exact fix for both Node loader failures. 991c2e73, 94e5d2eb — remaining doc half (Node baseline in the get-started prerequisites) split to docs: state the Node baseline in the get-started prerequisites #62.
• §6 — run-with-vitest omissions. Now documents the config() include/exclude replacement, the test.projects scoping pattern, drift, and the afterEach-can't-see-state teardown constraint. 309a74f7
• §7 — var vs varar naming. VAR_UPDATE → VARAR_UPDATE across all eight adapters (plus --varar-update / -Dvarar.update); @varar/vitest exports Varar-prefixed (vararPlugin, VararResultsReporter); var.config.ts → varar.config.json; from var import / use var:: corrected; the three var-* page slugs renamed to varar-* with redirects. The dead loadVarConfig for var.config.ts no longer exists in source — the live one is keyed on varar.config.json. ed04b556, 2e433630, 5570b236
• §8 — small stuff. "sensos" typo fixed; the non-existent "context" role removed from first-spec; VERSION is now stamped at release time instead of shipping 0.0.0 (2e394e69). The varar lint expectation-gap note is now in the .feature how-to and the Cucumber page — a proper reference for the command itself is split to docs: give varar lint a reference on the site #63.

While in here we also adopted "cell" as the shared vocabulary for the value a sensor compares (a slot holds either a cell or a table/row of cells), which tidied up the nine competing words §1/§8 brushed against.

Remaining

Two small doc items are left, each with its own issue:

• docs: state the Node baseline in the get-started prerequisites #62 — state the Node baseline in the get-started prerequisites (§5)
• docs: give varar lint a reference on the site #63 — give varar lint a reference on the site (§8)

Closing this as substantially resolved.