Skip to content

CLI

Jump to bottom
Pixnop edited this page Jul 16, 2026 · 4 revisions

CLI

The atlas dotnet tool (NuGet package Pixnop.Atlas.Cli) runs the Atlas scenarios of a compiled test assembly without VSTest, through the same in-process xunit runner dotnet test would use. It builds nothing: point it at the compiled test assembly. Four commands: run executes scenarios, fixture authors a prebuilt world save from a builder scenario, diff compares two TRX reports for differential testing, and stage explicitly pre-stages the engine assembly ahead of a --no-build run.

dotnet tool install -g Pixnop.Atlas.Cli

atlas run bin/Debug/net10.0/MyMod.Scenarios.dll              # run everything, sequentially
atlas run bin/Debug/net10.0/MyMod.Scenarios.dll --filter Chest
atlas run bin/Debug/net10.0/MyMod.Scenarios.dll --list       # discover only, no server boot
atlas run bin/Debug/net10.0/MyMod.Scenarios.dll --parallel   # multi-process, one class per worker
atlas fixture bin/Debug/net10.0/MyMod.Scenarios.dll \
      --scenario BuildsCastleWorld --out fixtures/castle.vcdbs   # author a world fixture
atlas diff baseline.trx candidate.trx                       # compare two runs, no server boot
atlas diff baseline.trx candidate.trx --json-tests           # + per-test outcome/duration/stdout
atlas stage bin/Debug/net10.0                                # pre-stage the engine assembly explicitly
atlas --version                                              # print the tool version

run, fixture and stage need VINTAGE_STORY set to the Vintage Story install directory, same as dotnet test (see Getting Started). The variable is validated up front, so a missing install fails fast at the CLI boundary instead of deep inside the first scenario's fixture. diff needs neither VINTAGE_STORY nor a scenario assembly: it reads only the two TRX files.

atlas run

Boots the embedded server in-process and executes the assembly's [AtlasScenario] methods sequentially, exactly like dotnet test would: same server, same world lifecycle, one live server per process. Output is a per-scenario PASS/FAIL line with duration, plus a summary; a scenario's non-empty test output (e.g. a rollback-degrade note, see Writing Scenarios) prints indented beneath its PASS/FAIL line.

Exit codes (same contract in every mode):

Code Meaning
0 Every scenario passed and at least one ran
1 At least one failure or runner error, or nothing ran (an empty run counts as a failure, so a typo'd filter cannot go green in CI)
2 Environment or usage error (VINTAGE_STORY missing, bad arguments)

--filter <substring>

Runs only the scenarios whose display name contains the substring (ordinal, case-insensitive). Display names are the same fully qualified names dotnet test reports. Combines with every mode, including --list and --parallel.

--list

Prints the discovered scenarios and exits without booting anything; VINTAGE_STORY is not required. Useful to sanity-check a --filter before paying for a server boot.

Worker mode: --worker

atlas run <dll> --worker runs exactly like plain run (one process, sequential, same exit codes) but reports exclusively as line-delimited JSON events on stdout: run-start, class-start, test-pass/test-fail/test-skip, class-end, error, run-end, every line versioned with v: 1. All human and engine chatter (the embedded server logs to the console) is rerouted to stderr, and a fail-safe guarantees the stream always ends with a well-formed run-end even when the run crashes.

  • --classes <A,B> (worker mode only; a usage error without --worker) restricts the run to the given scenario classes, comma-separated fully qualified names, exact match.
  • atlas run <dll> --list --worker performs discovery only: one discovered event per scenario, no server boot, VINTAGE_STORY not required.

Worker mode is the seam the --parallel orchestrator drives, and any tool that wants machine-readable Atlas results can consume it too. The full protocol contract (transport, versioning rules, every event's fields, an example transcript) lives in the repo: docs/specs/2026-07-06-worker-protocol.md.

Parallel execution: --parallel [N]

atlas run <dll> --parallel orchestrates the assembly's scenario classes over N worker subprocesses. The orchestrator discovers the classes without booting anything, then drains a greedy per-class queue: each worker is one atlas run <dll> --worker --classes <class> subprocess (one live server per worker, one class per dispatch), and workers pull the next class as they free up. Results stream back over the worker protocol and print live, per test; the final summary adds per-class wall clocks and the measured speedup versus the sum of class times (what running the classes back to back would have cost).

Default N. Without an explicit count, N is min(cores / 2, class count), always at least 1. Half the cores because each embedded server wants roughly two cores before workers start slowing each other down; capped at the class count because a worker without a class to run is pure overhead.

Crash translation. A worker that dies without a well-formed run-end, exits nonzero without a failing scenario explaining it, or outlives its per-class timeout is translated into a synthesized failed class carrying a stderr tail for forensics, and the queue keeps draining. A crashed worker can fail its class, never shorten the test list.

  • --worker-timeout <seconds> (parallel mode only): kills a worker stuck on one class for more than the given time (the whole worker process tree) and reports the class as failed. Default 600 per class: a generous outer defense above the in-process per-scenario watchdog (TimeoutMs), for the day a worker wedges outside any scenario.
  • --trx <path> (parallel mode only): writes one aggregated VSTest-style TRX report covering every class, so CI artifact upload and TRX tooling keep working without dotnet test. See CI Recipes.

Flag combination rules: --parallel is incompatible with --worker (workers are what it spawns) and with --list (listing never spawns workers); --worker-timeout and --trx require --parallel; --classes requires --worker.

atlas fixture

atlas fixture authors the prebuilt world save (.vcdbs) that [AtlasWorld(SaveFile = "fixtures/castle.vcdbs")] boots against, turning what used to be folklore (run a builder scenario, then harvest the save its graceful teardown wrote from the host's scratch data path) into a first-class command:

atlas fixture bin/Debug/net10.0/MyMod.Scenarios.dll \
      --scenario BuildsCastleWorld --out fixtures/castle.vcdbs

The builder-scenario contract. The builder is an ordinary [AtlasScenario] on a class deriving from AtlasScenarioBase, whose side effect is building the world: place blocks, run commands, seed data. Nothing marks it as special; any scenario can be a builder.

public class FixtureBuilders : AtlasScenarioBase
{
    [AtlasScenario]
    public async Task BuildsCastleWorld()
    {
        World.PlaceSchematic("fixtures/castle.json", World.Spawn.Offset(20, 0, 20));
        await World.ExecuteCommand("/time set day");
        await World.Ticks(5);
    }
}

How the command behaves:

  • --scenario <substring> (required) selects the builder by display-name substring (ordinal, case-insensitive) and must match exactly ONE scenario: zero or several matches is a usage error listing the candidates (exit 2).
  • The run uses the same in-process mechanics as atlas run --filter. After the scenario passes and the host tears down gracefully (the graceful shutdown is what makes the engine persist the save), the persisted save is copied to --out.
  • --out <path> (required) is where the fixture is written. Parent directories are created as needed; an existing file is only overwritten with --force (refusing without it is a usage error, exit 2).
  • A failing builder writes nothing and exits 1, so a broken builder can never silently produce a half-built fixture.

Exit codes: 0 with the fixture written; 1 when the builder failed or left no save (no fixture is written then); 2 on usage errors.

atlas diff

atlas diff <baseline.trx> <candidate.trx> compares two TRX reports and buckets what changed between them. It makes differential testing first class: the pattern it grew out of is running the same suite against a baseline and against a candidate (StratumParity runs it against vanilla Vintage Story and against the Stratum fork on every push) and gating on the outcome comparison, which used to be hand-rolled scripts.

atlas diff vanilla.trx fork.trx
atlas diff vanilla.trx fork.trx --json > diff.json

It reads only the two files: no server, no scenario assembly, no VINTAGE_STORY. It works on the TRX atlas run --parallel --trx writes and tolerates any spec-conforming TRX, including the one plain dotnet test --logger trx produces.

How tests are matched. Comparison is keyed by test name exactly as the TRX reports it. Theory rows carry their arguments in the name, so every [AtlasTheory]/[Theory] row diffs on its own. Duplicate names (one per rerun attempt) merge worst-outcome-first, so a flaky test that failed once and passed on retry compares as a failure.

Categories. Every change is bucketed into one of:

Category Meaning
New failures Failed in the candidate; passed, skipped, or absent in the baseline.
Fixed Failed in the baseline, passed in the candidate.
Vanished Present in the baseline, absent from the candidate.
New tests Absent from the baseline, present in the candidate.
Still failing Failed in both.
Duration shifts Between two runs that both passed: at least 2x AND at least 500 ms apart, both directions reported, informational only (conservative on purpose, never gates).

The console report is a summary line plus compact per-category listings; an empty category prints nothing. --json replaces the console report with a stable machine shape, versioned like the worker protocol (v: 1 first, additive evolution only, every category key always present so consumers never branch on absence).

--json-tests: per-test outcome, duration and stdout. An opt-in flag (implies --json) that adds a tests array to the JSON document, one entry per merged test identity: {test, baseline: {outcome, durationMs} | null, candidate: {outcome, durationMs} | null, stdout?}. It exists so a differential pipeline can build its own markdown job summary or history dashboard straight from the diff's JSON, without a hand-rolled TRX parser sitting next to it just to read outcome, duration and per-test stdout. Duplicate names within a report merge worst-outcome-first exactly like the category diff, and the kept attempt's stdout survives that merge outright (no falling back to the losing attempt's stdout, unlike the failure message). v stays 1: this is purely additive, and tests is omitted entirely (not emitted as an empty array) unless --json-tests is given, so the default --json payload is unchanged.

Exit codes gate differential CI directly:

Code Meaning
0 No regressions.
1 At least one regression: a new failure or a vanished test, and nothing else counts (a duration shift or a still-failing test does not gate).
2 Usage error or unreadable input.

So atlas diff vanilla.trx fork.trx IS the parity gate: a green baseline run and a candidate run, then one diff whose exit code fails the build the moment the candidate regresses relative to the baseline. The full contract (matching rules, the exact JSON shape, the duration-shift thresholds) lives in the repo: docs/specs/2026-07-14-diff-command.md.

atlas stage

atlas stage <path/to/test-output-or-assembly.dll> is an explicit pre-stage entry point for the engine-assembly auto-staging preflight (see Compatibility): it runs the identical decision the module initializers run automatically at boot, but as its own process, before dotnet test ever starts.

atlas stage bin/Debug/net10.0/MyMod.Scenarios.dll
atlas stage bin/Debug/net10.0

What it stages. Point it at a test-output directory or at any dll in it. VINTAGE_STORY must be set to the target install, same as run and fixture. It rewrites that directory's VintagestoryAPI.dll and .pdb as a unit from the target install (through same-directory temp files and atomic renames, exactly like the module initializers do), and handles the game-provided Newtonsoft.Json.dll direction-aware: a same-or-newer build is left alone, an older build the newer engine cannot run is refused. It prints one line per file (pair): staged, already identical, or nothing to stage.

Why it exists. Auto-staging normally does the right thing on a repointed VINTAGE_STORY: the module initializers re-stage the test-output copy on disk before anything can bind it. But when engine types were already JITted before any Atlas code ran in that process, THAT run still fails fast; a rerun in a fresh process goes green, because the copy is now already staged. A script that runs each install exactly once cannot absorb that fail-then-rerun cycle, and used to fall back to a per-install rebuild instead. atlas stage closes that gap: run it as its own process ahead of time, and by the time dotnet test starts, the copy is already staged, so its own module initializer finds a no-op and nothing ever fails. The one-shot cross-install pattern is:

atlas stage out/ && VINTAGE_STORY=... dotnet test out/ --no-build

which boots green on the first try, no rebuild and no fail-then-rerun, on every install the script points at in turn.

Exit codes:

Code Meaning
0 Staged, or already identical (nothing to do)
2 Usage or setup error: an unwritable output, an install without its pdb, a diverged copy already bound elsewhere, the Newtonsoft direction refusal

--version

atlas --version (or atlas version) prints the package version (the informational version without the +sha build metadata) and exits 0. It needs no scenario assembly and no VINTAGE_STORY.

Option summary

Option Modes Meaning
--filter <substring> run: all Only scenarios whose display name contains the substring (ordinal, case-insensitive).
--list run: plain, worker Print the discovered scenarios and exit without booting anything.
--worker run: worker Report exclusively as JSONL v1 events on stdout; chatter goes to stderr.
--classes <A,B> run: worker Run only these scenario classes (fully qualified names, exact match).
--parallel [N] run: parallel Run the classes on N worker subprocesses; N defaults to min(cores / 2, class count).
--worker-timeout <s> run: parallel Kill a worker stuck on one class for more than <s> seconds (default 600) and fail the class.
--trx <path> run: parallel Write one aggregated VSTest-style TRX report covering every class.
--scenario <substring> fixture Required: display-name substring selecting the builder scenario; must match exactly one.
--out <path> fixture Required: where to write the .vcdbs fixture; parent directories created as needed.
--force fixture Overwrite an existing --out file.
--json diff Emit the comparison as a versioned JSON document instead of the console report.
--json-tests diff Implies --json; adds a per-test tests array (outcome, durationMs, stdout) to the JSON document.
<path> stage Required: a test-output directory or a dll in it, to pre-stage against VINTAGE_STORY.
--version all Print the atlas version and exit (no assembly, no VINTAGE_STORY).
-h, --help all Show usage.

MIT licensed - GitHub - NuGet - Mod DB

Atlas

Clone this wiki locally