Repository navigation
Running tests with Claude
Operational notes for driving the GeoDMS regression suite from an automation agent (Claude Code) or any non-interactive harness. These are the gotchas that are not obvious and have each cost real time to learn.
Related: Running Linux tests on Windows · Test references and report generation
A GeoDMS run started as a child of the agent/terminal session is killed when that session is cleaned up (observed after a few hours, but it can be any session-teardown). The entire python + wsl bridge + GeoDmsRun process tree dies at once, with no error, and the GeoDMS /L log is cut off mid-line. It looks exactly like a crash but it is not — it is the session reaper.
Always launch anything longer than ~2 minutes detached, so it survives session teardown:
# single version
powershell -ExecutionPolicy Bypass -File batch\run_detached.ps1 -Version 20.1.0.l -Tests t100_network
# multi-version chain (overnight)
powershell -ExecutionPolicy Bypass -File batch\run_chain.ps1 # default 20.1.0.m then 19.0.0run_detached.ps1 uses Start-Process (detached), writes the PID to …\run_<tag>.pid and stdout/stderr to …\run_<tag>_<stamp>.out/.err. To pass extra flags (e.g. -linux-gui) that run_detached.ps1 doesn't forward, replicate its Start-Process python -ArgumentList full.py,… -WindowStyle Hidden -PassThru call directly.
A background Wait-Process -Id <pid>; <report> is a good way to be notified when a detached run finishes — but the watchdog itself is also a session-scoped background task and can be reaped. Don't assume it will fire. Always be able to re-derive state from disk:
-
…\run_<tag>.pid+Get-Process -Id <pid>→ is it still alive? -
<results>\<ver>\log\<test>.txt→ GeoDMS log (look for@@@@@ Logging started/Logging ended/ errors) -
<results>\<ver>\<test>.txt→ result indicator (<result>OK</result>etc.) -
…\run_<tag>_<stamp>.out→ harness stdout (Running experiment,finished (...))
-tests t200 also matches t2000 (the ~1h Hestia test). -tests t100 is fine, but to be safe always use the longest unique stem:
| Want | Use |
|---|---|
| t100 only | t100_network |
| t200 grid (not t2000) | t200_grid |
| t2000 Hestia | t2000_hestia |
| all t405 | t405 |
The filter wipes the matching .bin caches so the targeted tests actually re-run; unselected tests keep their cached results in the report.
| Flag | Effect |
|---|---|
-version <v>[.m|.l] |
which build (e.g. 20.1.0.m, 20.1.0.l, 19.0.0) |
-MT1 / -MT2 / -MT3
|
multithreading settings handed to GeoDmsRun (S1/C1, S2/C2, S3/C3) |
-tests a,b,c |
comma-separated substrings to run |
-report-only |
regenerate the HTML report from existing result folders, run nothing |
-resume |
continue an interrupted round instead of restarting it: keep this round's intermediates rather than wiping them at round start. Same version only — a fresh round must wipe, or it silently measures the previous round's leftovers |
-linux-gui |
also run GeoDmsGuiQt tests on .l (off by default; see Running Linux tests on Windows) |
-no-gui |
skip the GUI tests (t1630/t1640/t1642) on any flavour — for older builds whose GUI binary crashes on the current test scripts |
-all-versions |
show every historical result folder as a column |
-sp |
run GeoDmsRun with /SP (PerformanceLogging, GeoDMS 20.16.0 and later). Off by default — see below |
-report-only is the fast way to refresh the report after editing the renderer or after a run was interrupted before its report step.
python full.py -h prints all of these with the reasoning behind each default, and is the authoritative list; this table is the short form.
/SP (PerformanceLogging) exists since GeoDMS 20.16.0, and the suite leaves it off; pass -sp to turn it on.
Turned on, GeoDMS measures and reports the cost and footprint of each operation under the performance message category — every operator run, scheduled calculation and storage read — plus the memory diagnostics on the same gate: the huge-allocation lines, the periodic live-memory census and the end-of-run allocation histogram. All of it goes to the log, and nothing in this harness reads it.
Keep launcher/watchdog commands simple. A parse error in an inline expression means the whole script fails and Start-Process never runs — you think a run started but nothing did.
- No
&&/||chaining, no ternary?:, no??. - Complex inline
if(){...Substring...}inside aForEach-Objectis a frequent parse-error source — split it out or do the reporting in Python instead. - Native-exe stderr redirection (
2>&1) wraps lines and flips$?— stderr is already captured for you.
Heavy tests cannot run concurrently:
| Test | Peak RAM (observed) |
|---|---|
| t2000 Hestia | ~85 GB |
| t405 NetworkModel | ~33 GB |
Run one heavy test at a time. Lighter tests can overlap, but when in doubt, sequence and gate each on its watchdog/PID.
The point of this suite is to catch behavioural changes. When a test fails and "the reference looks stale", verify it is actually stale before rebaselining:
- Run the same test on the previous version and compare. If all tested versions produce the same output and only the reference is old → safe to regenerate the ref.
- If versions diverge, it is a real regression — file an issue, do not rebaseline (rebaselining would mask the bug, the same hollow-OK trap the harness was hardened against).
- Worked example: t810 looked like a stale-ref failure but 20.1.0 diverged from the LUISA reference by 348k cells where 19.0.0 differed by only ~6k → a real 20.1.0 land-use regression, filed as ObjectVision/GeoDMS#1136. See Test references and report generation.
| What | Path |
|---|---|
| Results + HTML report |
C:\LocalData\GeoDMS-Test\Regression\ (reports\<new>___<old>.html) |
| Per-run logs | …\Regression\<ver>\log\<test>.txt |
| Result indicators | …\Regression\<ver>\<test>.txt |
| Reference data (read-only) | D:\SourceData\TestReferenceFiles\ |
| Source/input data | D:\SourceData\ |
| Per-round intermediates |
C:\LocalData\runs\<run_id>\ (wiped at round start unless -resume) |
| Detached-run pid/out/err | …\Regression\run_<tag>.* |
| Machine paths config |
batch\local_settings.json (gitignored) |
gh may not be installed and there may be no GH_TOKEN. The git credential store often still has a usable PAT:
# retrieve token without printing it, then POST to the API
import subprocess, json, urllib.request
p = subprocess.run(["git","credential","fill"], input="protocol=https\nhost=github.com\n\n",
capture_output=True, text=True)
tok = next(l.split("=",1)[1] for l in p.stdout.splitlines() if l.startswith("password="))
# POST https://api.github.com/repos/<owner>/<repo>/issues with Authorization: Bearer <tok>Engine regressions belong in ObjectVision/GeoDMS; test/harness issues in ObjectVision/GeoDMS-Test.