Skip to content

Test references and report generation

Jump to bottom
Maarten Hilferink edited this page Aug 23, 2026 · 4 revisions

Test references and report generation

How the regression suite stores references, computes results, decides pass/fail, and renders the HTML report — and the integrity rules that keep "green" honest.

Related: Running tests with Claude · Running Linux tests on Windows

Data architecture (keep these separate)

Role Location Notes
Reference (expected output) D:\SourceData\TestReferenceFiles\ read-only (StorageReadOnly="True")
Input / source data D:\SourceData\ read-only
Fresh compute C:\LocalData\runs\<run_id>\regression\ recomputed each round; the round's intermediates land here
Results + report C:\LocalData\GeoDMS-Test\Regression\ indicators, logs, reports\*.html

At runtime the separation holds: references are read read-only, the result is computed on-the-fly (e.g. t102 deliberately has no StorageName so it recomputes instead of caching), and the comparison reads the run's own output, never a shared intermediate.

Caveat: the references in TestReferenceFiles are themselves build-generated snapshots, not version-independent ground truth. That is the root of most cross-version failures (see below).

The cardinal rule: cross-version check before rebaselining

A test failing because "the reference is stale" might actually be a real version regression. Before regenerating any reference:

  1. Run the test on the previous version(s) too.
  2. If all tested versions produce the same output and only the saved reference is old → the ref is genuinely stale → safe to regenerate.
  3. If versions diverge → real regression → file an issue, do not rebaseline. Rebaselining from the regressed output masks the bug.

t810 worked example. Looked like a stale LUISA reference. Drilling the per-cell diff: 19.0.0 differed from the ref by 5,983 land-use cells (0.04%), 20.1.0 by 348,587 (2.6%) — same config/input/ref, only the binary differs. That is a real 20.1.0 land-use-allocation regression, not a stale ref. Filed GeoDMS#1136; the reference was left untouched.

Version-specific references

When an operator change legitimately alters output across versions, keep per-era reference files selected by a version switch in the filename. Pattern (from Regression/cfg/stam.dms):

parameter<string> PC6_connect_ok_filename :=
    '%TestRefDir%/t100/pc6_connect'
    + (GeoDMSVersion() < 7.198d ? '_before7198'
       : GeoDMSVersion() < 20d  ? '_tm_v19'
       : '') + '.fss';

So pre-7.198 builds read _before7198, pre-20 builds read _tm_v19, ≥20 read the current ref. The _tm_v19 set for t100/t102 was generated from 19.0.0's own connect/impedance output (the connect/impedance_matrix row count changed at the 19→20 boundary).

Generating a reference safely (the Make_/ReadOnly pair)

The idiom for regenerating a reference is a writable Make_X unit (writes this build's output to the ref path) plus a read-only X unit (reads it back), e.g. t301's Make_BAG_type_woonpand_rel / BAG_type_woonpand_rel.

Guard the writable one so it can't silently overwrite the live reference (which would make the test pass against itself):

unit<uint32> Make_BAG_type_woonpand_rel: StorageName = "%TestRefDir%/t301/type_woonpand_rel_ok.fss"
{
    parameter<bool> RegenerateReference := False;   // set True only to deliberately rebuild
    unit<uint32> domain               := ..., IntegrityCheck = "RegenerateReference";
    attribute<string> identificatie   := ..., IntegrityCheck = "RegenerateReference";
    attribute<uint8>  type_woonpand_rel := ..., IntegrityCheck = "RegenerateReference";
}

Gate the domain too, not just the attributes — targeting an attribute first materialises the domain, which would otherwise restamp domain.dmsdata.

For file_comparison tests (the ref is a plain file, e.g. t2000's statistics HTML), regenerating is just a file copy — but the cross-version rule still applies.

Integrity: no hollow "OK", no false pass

Two traps the harness now closes:

  • Hollow-OK (#12). A declared result indicator that is missing must not fall back to "OK". The cfg writes result_html into the harness result folder (via TempDir/results_folder.txt); the report reads the declared indicator_results_file, and a missing one shows "niet gevalideerd" (amber), not green.
  • False pass (crash). A GeoDmsRun/GeoDmsGuiQt that crashes at startup can still exit 0 (its real code is swallowed by the wsl → run_with_sampler.sh → timeout chain). The profiler now treats return 0 + no /L log written (the log always starts with @@@@@ Logging started) as a crash → status 134, rendered gecrasht (geen log) (red). This is what makes the broken .l GUI tests report honestly.

Report generation

full.py runs the experiments, then collect_and_generate_test_results builds reports/<newest>___<oldest>.html. To rebuild without running: python full.py -version <v> -report-only.

Things worth knowing about the report:

  • Columns = result folders whose version is ≤ the run version (get_valid_result_folders filters <= target). So a -version 19.0.0 report shows 19.0.0 and older only; the newest-version report is the comprehensive one. (By design — change the filter if you want all columns regardless.)
  • Row order is numeric by test code (t200 before t1742, t405_2 before t405_3), not alphabetical.
  • Status pills / colours: ok=green, test faalt / output verschilt (exit 99) / gecrasht (134) / numeric error=red (fail), niet gevalideerd (declared indicator missing)=amber (warn), niet gedraaid / GUI-skipped=grey (skip). "output verschilt" is a real failure → red, distinct from the amber "niet gevalideerd".
  • "gewijzigd" marker = a test's indicator value changed vs the previous version. The comparison is flavour-aware: a column is diffed against the newest strictly-older result of the same flavour, falling back to the newest non-flavour version (e.g. 20.1.0.m → 19.0.0 when there is no older .m).
  • Total time in a column header = the sum of all per-test durations in that column (serial sum, not wall-clock).
  • Sub-second tests legitimately show 0:00:00 and a degenerate 0.01 GB / 9 thr metric — the sampler can't catch a process that exits instantly. The validation is still real.

Warm vs. cold runs (why timings vary)

A timing reflects what was already on disk. By default a round computes into its own <LocalDataDir>/runs/<run_id> root and wipes it at round start, so the round is cold. -resume keeps that root and -report-only never touches it, so a resumed round reuses whatever its predecessor wrote — the expensive prepared data (RSopen base data, NetworkModel PBL prep) above all — and its timings are warm. Tests whose .bin is already stored are skipped and keep their earlier timing in the report.

Version-switch pattern for old builds

Several configs branch on GeoDmsVersion() so an old binary parses/computes the supported form. Use =-evaluation so only the chosen branch is parsed (the other name may not exist in that version):

Feature / boundary Where
pow/^ first-class operator (< 20.01d) Operator.dms
min empty-partition escape \xFF (< 20d) Operator.dms
metric m² superscript vs m^2 (< 20.0004d) Operator.dms
bp_polygon_* vs polygon_* (<= 19.0d) Operator.dms
t405 short fileNameSuffix (pre-#916, < 20d) PublicTransport.dms
connect/impedance ref _tm_v19 (< 20d) stam.dms

#10 also adds a relative numeric tolerance (1e-4) fallback to file_comparison, so outputs that differ only in last-digit rounding still pass.

Clone this wiki locally