GeoDms 20.16.0

Pre-release. GeoDms 20.16.0 is the first release since 20.12.0, and it is a large one. An external model — an R or python script — can now be driven from a configuration as an ordinary supplier, with its output in the event log and its place in the calculation order guaranteed (#659). An IntegrityCheck on a container or unit finally guards the items underneath it (#1180), which is the change most likely to be noticed in an existing configuration. MMD storages now record and verify what they hand back, so a store read into the wrong domain fails instead of returning a plausible wrong answer (#1154, #1187). The GUI no longer goes "(Not Responding)" during a long computation (#1156), starts with an empty project, and got a round of long-standing usability fixes. Grid mapping / mapping_count gained a separable fast path (#298), and a sweep over the registered operator list turned up several names that silently produced wrong or empty results.

Flavour	Asset
.m — Windows, MSBuild	GeoDms20.16.0.m-Setup-x64.exe
.c — Windows, CMake	GeoDms20.16.0.c-Setup-x64.exe
.l — Linux, Ubuntu 24.04	GeoDms20.16.0.l-linux-x64.deb, .tar.gz (+ .sha256, .sha256.p7s)

Read this first: what can start failing after the upgrade

Several changes turn silence into a message. That is the point of them, but it means a configuration that ran clean on 20.12.0 can stop on this build:

• An IntegrityCheck on an ancestor is now evaluated when an item below it is read. A check that was never actually enforced — because nobody requested the item carrying it — starts firing, and it fails the read.
• An MMD store whose length does not match the domain it is read into is now refused, as is a store whose dictionary restrictions no longer hold.
• An IntegrityCheck configured on an item below a storage holder is refused at write time, with a message pointing at the holder.
• A point range spelled {row, col} still parses, but now emits a depreciation warning naming its xy(x; y) replacement.
• CalcRule = "" (or the deprecated Expr = "") warns where it used to be accepted in silence.

Driving an external model from a configuration (#659)

Running Rscript or python from a configuration over exec_ec plus files already worked, but two things made it unusable in practice, and both are fixed.

The child process inherited our standard handles. Under GeoDmsGuiQt, which has no console, a failing script therefore left nothing behind but an exit code — no message, no traceback. The child now gets one inheritable pipe carrying both stdout and stderr, and whatever arrives is reassembled into lines and reported as exec: <line> under the commands message category. Reporting is capped at 1 MB, after which the pipe is still drained — not draining it would block the child once the buffer fills — with a single warning saying so. A redirection inside the command itself still wins, so an existing dir *.shp > dirinfo.str behaves as before. Implemented on both platforms: CreatePipe with PeekNamedPipe-gated reads on Windows, pipe() plus posix_spawn file actions with a poll()-driven non-blocking read on Linux.

The wait was a single infinite WaitForSingleObject, which left no room to drain that pipe and made a multi-minute child indistinguishable from a hang. It is now a 500 ms tick that drains the pipe and reports exec: still waiting for <program> after <n> seconds every ten seconds.

The ordering half. A stored item with no calculation rule collected only its calculation suppliers and its domain/values-unit futures as prerequisites of its read. Its configured ExplicitSuppliers were honoured by actor-level visitation but never entered that future list, so a read could race ahead of the supplier still producing the very file to read — measured at two seconds ahead, with a gdal.vect read of a parquet file starting before the exec_ec that runs the python writing it. PrepareDataRead now makes each ExplicitSupplier's future a prerequisite of the read.

That makes a full write → exec_ec → read round trip orderable as ordinary suppliers within one run, with the whole chain under interest up front. On a 13-step model coupling, per-step time went flat at 13–16 s instead of growing from 79 s to 188 s, total 6 minutes instead of 33, and peak commit 27.6 GB instead of 67.6 GB. At full scale the same chain runs at a constant ~120 s per step where the meta-scripted equivalent needed hours and 327 GB.

IntegrityChecks now guard the items below them

An ancestor's check applies to its descendants (#1180). Using a nested item did not validate its ancestors, so an IntegrityCheck on a container or unit only ever fired when that item was itself requested. Reading anything underneath it succeeded and delivered exactly the data the check was written to gate — silently, which is the worst of the three possible behaviours. Visiting a check and enforcing it are separate concerns, so this took two halves: VisitSuppliers now walks self and ancestors, making an ancestor's checker part of the item's dependency graph — scheduled, and with cycles detected rather than hidden — and the validate phase evaluates the item's own check while the ancestors' verdicts arrive through the tree structure, the parent being validated first.

A check that reaches into its own subtree closes a cycle and is reported as a circular dependency. It always was one whenever the item carrying it was requested, so such a check could never be relied upon.

Making that affordable (#1182). Folding every ancestor check into the items below it means a single check on a root container multiplies over the whole configuration: one test configuration applied the integrity_check operator 3393 times. The set of conditions an expression is certain to evaluate is now memoized per DataController and derived once per node, as an explicit-stack post-order walk that stops at nodes already carrying a set — exact at any depth, where the first attempt was depth-bounded and lost sight of the guard exactly where configurations are deepest. A 25-link reference chain under one ancestor check now yields one integrity_check DataController instead of 25. Conditions are compared per conjunct rather than as whole expressions, so a && b normalises the same way however it is nested.

Debug builds run these configurations again (#1181). Four sites in the lookahead estimation and reporting instrumentation called accessors whose Debug preconditions — meta thread, being the range item, holding interest — an estimator or reporter cannot guarantee, so every Debug run whose operator produced a data item died on an assert. Each now uses an accessor whose contract it can honour. Two of them were also wrong in Release: the tiling was read from a delegating unit's own empty slot, so the estimator missed the actual tiling.

MMD storages: say what is stored, and verify it

The dictionary records restrictions on external units (#1154). An .mmd dictionary references a unit declared outside the storage by name only, which re-resolves against whatever the reading configuration declares under that name. When that declaration changed since the write — an fpoint that became a dpoint — the reader silently bound the new element type to the old bytes. The dictionary root now carries a synthesized IntegrityCheck restricting every unit that is referenced by a stored attribute but declared outside the dictionary: its ValueType always, and for an external domain its extent as well when the range is known at emission. Merged onto the read holder, those restrictions guard every sub-item read through it via the ancestor fold above.

A stored array must match its domain (#1187). An MMD attribute is a keyless positional array; its element count comes from the domain the reader binds to, never from the file. The mapping call only noticed a file that was too short, and then as an access-denied error naming neither the domain nor the sizes. A file that was too long was accepted silently: the requested capacity was clamped down to the file, the surplus dropped, and the first N elements handed out as if they belonged to this domain — values attached to the wrong elements while every total stayed plausible. That is how a production run produced a wrong but credible answer, reading 11,008,492 cached elements into a domain of 11,007,909. The read path now compares the file length against the very function the write path sizes the file with, and throws an error naming the storage, both sizes, the element count and the tile count. It applies to stores whose dictionary carries no restriction at all, i.e. to everything written by earlier versions.

Restrictions on MMD content belong on the common ancestor (#1179). An IntegrityCheck on a stored sub-item derailed the write session: the data file was produced, but the dictionary was never written and the whole storage read back empty — silently, with exit code 0. The rule is now explicit and enforced on both sides. On write, a check on any item below the storage holder is refused loudly before any file is produced, with a message directing to the holder; a check on the holder itself works and writes the dictionary normally. On read, a read-only holder that declares sub-items or carries a reader-declared check is refused, because the sub-items and restrictions of a read holder come from the dictionary — previously a dictionary-supplied property silently replaced the reader's.

Points state their coordinate order (#1155, #1165)

A textual point was rendered as a bare pair whose order the reader had to guess, and the two halves of the product guessed differently: the range property and the Range entry of a dictionary wrote {row, col}, while the Range row of the detail pages applied a shuffle and wrote {col, row}. The same rectangle therefore appeared in opposite coordinate order on one and the same detail page.

Points now carry the xy() / yx() tag that data-block point literals have used since 18.1.0. Output is always xy(x; y); input also accepts yx(y; x) and the legacy untagged {row, col}, so existing configurations and dictionaries keep working. , and ; are interchangeable, both inside a point and between the two bounds of a range, and surrounding whitespace is tolerated. Reading a range in the legacy brace spelling now emits a depreciation warning naming the replacement; detection sits in the range parser itself, so the high-volume point readers are untouched.

Numbers rendered along the way: the shortest round-tripping form turns 300000.0 into 3e+05, and the thousand-separator pass then mistook the whole rendering for an integer part and produced a corrupted 3e,+05. Plain notation is now preferred, and a rendering that does stay scientific no longer gets separators.

GUI

No more "(Not Responding)" during long computations (#1156). A multi-minute computation on the Qt main thread stopped retrieving messages, so Windows ghosted the window — and once the ghost window exists, all further input is rerouted to it, starving the very check that would make the computation yield. Clicking the window to interrupt therefore did nothing. Three measures, none of which dispatch a message mid-computation: the waiting-messages probe looks at QS_ALLINPUT, so a pending incoming cross-thread SendMessage also makes the engine yield; ghosting is disabled at start of main(), so input keeps landing in the real queue and the suspend trigger answers the first click within its one-second tick; and where the main thread parks while joining work it blocks in a message-aware wait, which Windows counts as waiting for input and therefore exempts from the not-responding verdict.

Startup is an empty project (#1162). The "Reopen last configuration?" prompt was raised from the MainWindow constructor, before the window is shown — a modal box whose parent top-level is still hidden gets no taskbar button and can end up behind other windows — and it cost a keystroke on every launch. The dialog is gone, the default startup state is an empty project, and Settings > GUI options > Startup > Reopen last configuration at startup restores the auto-load. File > Reopen (Alt+R) opens the last configuration when none is loaded, and says which of the two it will do. The startup sequence now resolves what to load before constructing the window, so with a configuration to load the main window is shown first and the load queued after it: an error dialog from a broken configuration now appears together with its window rather than a second later, behind everything else. Measured time to a visible window with a configuration on the command line: 41 ms, and no splash. Only an idle start still shows the splash.

Keyboard focus stays on the view that was clicked (#1184). Zoom keys stopped working in a MapView as soon as the user had clicked an item in the TreeView in between, while the mouse wheel kept zooming — the asymmetry is the routing rule, not the key mapping: Windows delivers the wheel to the window under the cursor, while +/- follow keyboard focus. A click into a view called an unconditional activateWindow(), which ends in SetFocus on the main window even when it is already active, upon which Qt restores its own last focus widget — the tree view, which silently eats +/- as expand/collapse. Every key was dead in that state; +/- is simply what gets noticed, and what remote-desktop users depend on since RDP does not forward the wheel.

The recent-configurations list (#1034, #1193). The File menu showed the same configuration several times: the stored list held real duplicates differing only in path separators and case, which the menu rendered identically and nothing could ever collapse. Exactly one canonical form is stored now, and each entry is canonicalised before the duplicate test in a function that already runs at startup and on every File-menu open — so existing lists heal themselves. Configurations can now be pinned to their own numbered block at the top of the menu; the context-menu item existed but was permanently disabled with nothing behind it. Opening a configuration moves it to the front of its own block, so a pinned one stays inside the pinned block and an unpinned one never jumps above it. Both context-menu actions now keep the menu open — maintaining a long list cost one visit to the File menu per entry — and removal no longer asks for confirmation, since it takes a path out of a registry list and does not touch the configuration.

Value info says why a row has no value (#612). The page traced a null to a dead end: invert(org_rel) showed null and an empty supplier list, with nothing to say why. Operators that can explain their result now record a reason next to the value: invert / invertAll report the argument rows that map onto the explained row — which also reveals a value dropped as a duplicate — and state when there are none; rlookup states that a requested value does not occur. lookup needs no operator support, since the generic mechanism already resolves the explained row into each argument and follows the resulting index into the domain it points at.

Two long-standing appearance asks. The palette colour picker is now a QColorDialog with an alpha channel (#859), which is why "set to transparent" no longer needs to be a separate menu entry beside it, and which also implements the picker on non-Windows builds for the first time. Layer controls get a three-pixel 3D border and a darker grey when selected (#828); table header controls keep their two pixels.

Diagnostics

• Progress lines name a config item, always (#795). An intermediate result is an anonymous cache item with no back reference, so progress lines from an operator calculating one named no item at all and could not be clicked, while the same operator bound to a config item did. An origin item now travels down the nesting of argument calculations and is stored on each participating DataController before its operator is scheduled. The reference is weak: a name for a message must not keep an item alive.
• An unresolvable name is reported on the item that holds the reference (#1188). The bare symbol was handed back into the substituted expression after failing its holder, and resolving it later from the config root produced a "cannot find item" message carrying no item — therefore not clickable, and logged once per dependent DataController. One reported configuration filled the event log with dozens of identical such lines.
• CalcAndWrite writes a message to the event log when its argument is finished (#846): parameter<uint32> total := CalcAndWrite(sum(src/val), 'total is ready');. The result is the first argument, so nothing is copied and nothing extra is scheduled. Getting this line previously meant wrapping the value in a container and phasing it, which bought a fence nobody asked for.

Grids: separable mapping / mapping_count (#298)

A W × H tile mapped between two grids contains only W distinct x's and H distinct y's, so a coordinate-separable transformation needs W + H transformations rather than W × H. Separability is decided by a structural CRS gate and then verified numerically on a probe lattice; anything unproven falls back to the generic loop. The conversion state is built once per invocation instead of once per tile — on a 12M-cell EPSG:4326 → EPSG:3857 mapping, ~220 ms of ~230 ms went into rebuilding the transformer per tile under a global lock — and one functor is handed out per thread, which speeds up the non-separable path too. The same-CRS case, by far the most common, is separable by construction:

	before	after
mapping_count, 4M cells, EPSG:28992 10 m → 100 m	23.4 ms	0.2 ms
mapping, 49M cells, EPSG:28992 10 m → 100 m	23.0 ms	11.0 ms

This fixed a live correctness bug on the way: in cross-CRS mapping(D,V) the source index restarted at 0 for every 1024-point block while the output iterator advanced correctly, so any tile larger than 1024 cells repeated the first 1024 cells' coordinates for the whole tile.

Operators that did not do what their name promised

• modus_count_uint16 was registered as modus_count_uint17 (#1173). Operator lookup is by exact name, so the name every other part of the system spells did not resolve at all — and the table view generates precisely that name for domains whose cardinality fits UInt16, so the GUI's modus-count aggregation was broken for that whole cardinality band. Renamed without a transition alias.
• potentialPacked / potentialRawPacked convolved with a bit-punned kernel (#1174). The pre-computed kernel FFT read the Float32 reversed-kernel buffer as an array of doubles, so each pair of adjacent taps decoded into one garbage double and the read ran exactly 2× past the allocation. Small grids underflowed to zero, larger ones picked up NaNs from out-of-bounds heap — and NaN is the Float32 null, hence the all-null results reported.
• ordered_union_data now verifies its own promise (#1168). The operator is how a modeller states that the unioned values are non-decreasing; consumers read that flag and take a sorted-merge path instead of building a sort index, so a false statement silently produced wrong results. The order is now checked as the values pass into the result, carried across tile and argument boundaries; a descent fails the result naming both values involved.
• ipf_alloc removed (#1177). Its entire algorithm was commented out, so it built its result items and returned an all-zero landuse grid with an empty status and no warning. A silent wrong answer is worse than an error; the name is now unknown to the parser.
• bp_buffer_multi_polygon unregistered (#1177) — boost::polygon has no multi-polygon buffer here, so the group only ever produced "no implemented operator". Use bg_buffer_multi_polygon or geos_buffer_multi_polygon, which also accept float coordinates.
• UrlEncode, HtmlEncode, HtmlDecode implemented (#1177), alongside the existing UrlDecode. They were registered names with no operator behind them while the wiki documented them as working. HtmlEncode escapes only the five predefined entities so UTF-8 stays UTF-8, and HtmlDecode is its inverse plus   and numeric character references.
• The obsolete stubs that error by design (claim_*, subset, dijkstra_s / _m / _m64, PartNr) stay for v20, but their promised v21 removal was only a runtime throw inside a static initializer — at v21 that would escape through DllMain and every executable would fail to start with no message at all. Each site now also carries a static_assert, which fails the build at the exact line to delete.

Geometry

• cgal_union_polygon lost area (#1178). The CGAL reader subtracted every inner ring from the whole accumulated polygon set instead of from the polygon that ring belongs to, so any polygon nested inside another polygon's hole — an annex on the courtyard of a merged building block — was erased. On 123,773 building footprints in inner-city Amsterdam this cost 0.29% of the area when uniting partition unions; the idempotence invariant now holds to 2.9e-9. Elementary unions of single polygons were never affected.
• bg_union_polygon could kill the process (#1176). Geometry that MakeValid could not repair to Boost.Geometry's satisfaction was passed to the union anyway — undefined behaviour, which ended the process with no error line and the log cut off mid-line. The repair result is now cleaned of degenerate rings, the required winding order is restored, and a geometry that still cannot be repaired raises a regular error. bg_union_polygon remains not recommended: on such data it now reports a clean failure rather than completing the union.
• The named value-type casts preserve their argument's ValueComposition (#1038 follow-up). fpolygon, dpolygon, ipolygon, spolygon, upolygon and wpolygon built their result with the arc composition for any sequence result type, so casting a declared-(poly) storage attribute yielded an arc result: bp_union_polygon(ipolygon(geom), part_rel) warned about arc geometry on correct configurations, and the declared (poly) on an intermediate attribute was overwritten by the computed arc, so it could not be worked around either. The composition is now inherited when both the result type and the argument carry a sequence-like one. convert() and the bracket cast were already fixed; the named casts were not.

Performance

join_equal_values sized its six counting arrays on the range of the join key's values unit rather than on the data (#1175). A key typed by its value type instead of by a domain unit — a plain uint32 attribute — therefore reserved 2^32−2 slots per array: 60 GB and two minutes to join three rows against three rows. When the range is large the index is now sparse, holding only the distinct values that actually occur in the first argument.

The FFTW3 port is complete. float32 has always been a valid element type for convolution data, but the FFTW3 migration only ported the double-precision API, leaving the single-precision backends running a double transform whose result was merely stored in float32 buffers — half a port, and the half that produced #1174. They now use FFTW's single-precision API throughout, and the single-precision backends are available as potential32 / potentialRaw32.

Build, packaging and internals

• The CMake (.c) setup now ships the MSVC C/C++ runtime (#1186). It shipped none at all, so the install borrowed msvcp140* / vcruntime140* from C:\Windows\System32 — from whatever redistributable happened to be on the target machine. Anything older than 14.40 lacks a symbol that arrow.dll imports through GDAL, so the GUI died at startup with a missing-entry-point dialog naming that DLL. Build servers have a recent redistributable, which is why this never showed there. The six DLLs the shipped binaries actually import are packed as hard entries, so a missing one fails packaging instead of surfacing on a user's PC.
• A release test for the shipped content (#1031), running against the copies the installer packages rather than the source tree beside them: the shipped testcases battery exactly as a user runs it from the install folder, plus an example over a real source dataset with its download forced, since that path is user-facing and silent when it breaks. This exposed that the two Windows setups did not ship the same library: a template and its only caller existed solely in the CMake copy, and that copy predated a bug fix the MSBuild setup had been shipping for years. The duplicate trees are gone.
• A pass over the DLL export surface, driven by a cross-match of what each binary exports against what the others import: roughly 1150 entirely-dead exported symbols removed, headers and translation units relocated toward their sole consumers, and several linkage-hygiene defects fixed on the way — including a decorated definition in a header, which gave the GUI its own private copy of a drawing-size threshold so that option never reached the drawing code.
• Header hygiene and translation-unit reorganisation: normalized prologs and include guards, dead includes removed from the hot headers, precompiled-header contents enriched per module, and small translation units merged while a few oversized ones were split. The Unit<V> hierarchy and the DataArray adapter layers were each collapsed into a single class.
• Release hygiene: the huge-allocation, periodic census and end-of-run histogram log lines are emitted only under performance logging (/SP), and operation-context tracing follows the Debug flag again instead of costing Release a critical section per operation-context lifecycle.
• The VS Code language extension's operator list is back in sync with the registered operator groups — 31 names were never listed — and its grammar generator is reproducible again, its ordering having depended on randomized string hashing.
• Linux: a functional-style cast that only MSVC accepts, an order-dependent registry-access section that aborted the Debug build at startup, and a round of GNU C++ warning mitigation.

Test report