Skip to content

UpdateAll and cache internals

MaartenHilferink edited this page Oct 28, 2026 · 23 revisions

This page is for developers of the model. It describes how batch\UpdateAll.cmd finds its engine and runs the output step, which files it leaves in batch\log, and how the model writes and checks the signature of a cache. How to run the model with UpdateAll.cmd is on How to run the model.

Engine

UpdateAll.cmd takes the GeoDmsRun.exe it uses from, in this order: the environment variable GEODMS_EXE, for example another engine or a test build; the local build C:\dev\GeoDMS_2026\bin\Release\x64\GeoDmsRun.exe; and the installed GeoDMS of its GEODMS_VERSION line (20.26.0), in the folder that the installer writes to the registry. It stops when it finds none. Any engine must meet GeoDMS software. For runs of many hours a copy of the build folder (without the .pdb files) is safer than the build folder itself: a rebuild during the run then neither fails on files in use nor replaces DLLs under the running process.

The output step

The output section runs in five steps:

  1. The model writes the pre- and post-transport caches of the base sets without one, as the legs section does (with all that section has done it already, and -nolegs skips it), so that the portions read them instead of each computing them again.
  2. The model writes the output folder to batch\log\output_dir.txt (NetworkSetup/GenerateOutput/OutputDir_ForBatch) and checks the longest path (NetworkSetup/GenerateOutput/Output_Sig/PathGate).
  3. The script deletes the files that the model writes in that folder: tt_* and signature.* in the folder, the files in PerBlock, Mediaan and Tellingen, and od_tellingen.* and unieke_od_tellingen.*. It does so only when the folder lies under a folder Output, or Output_<O>_<D> for an OD pair.
  4. batch\WriteOutputBlocks.ps1 asks the model for the list of blocks (Generate_Output/OutputBlocks_List, written to batch\log\output_blocks.txt), which also writes or checks the signature of the output folder. It then runs one GeoDmsRun per portion of Advanced/Components/BlocksPerPortion blocks (see Model parameters), so that memory is released between the portions. With MergeOutput each portion merges the components per block (ComponentMerge/PerBlock/<block>/Fence/Generate).
  5. batch\OdTellingen.py adds up the counts per block (see Counts next to the output).

With the environment variable OUTPUT_SINGLE, steps 4 and 5 are one request for all blocks, NetworkSetup/GenerateOutput/Generate_Output. While the file batch\pause_before_output.flag exists, the output section waits before it starts, without a computing process.

Logs and helper files

The portions of the chain and output steps write their console output to UpdateAll_chains_b<nnn>.txt and UpdateAll_output_b<nnn>.txt. batch\ReportUpdateAll.py [step ...] summarises the last run of each step: start, end, duration, number of result rows, the peak CommitCharge and PeakLiveLarge from the memory summary at the end of the log, the size of the cache and the error lines. The UpdateAll_* files are not in git. Some steps first ask the model for a value, which it writes to a text file in batch\log: car_moments.txt (the moments of the run, for cars), chain_blocks_todo.txt (the stop blocks without a cache, for chains), and output_dir.txt, output_blocks.txt and output_blocks_per_portion.txt (for output). They are not in git either.

How the signature of a cache is checked

The hash (Templates/StringHash_T, in Templates/Signature_T.dms) is a polynomial hash modulo 2^31 - 1 written in plain integer arithmetic in the configuration, because GeoDMS has no string hash; it is stable across versions and platforms, as a cache key must be. So caches for different settings live side by side. The signature of an MMD cache is in its own folder (signature.txt and signature.xml, next to the block caches or next to 0Dictionary.dms), so that IntermediateResults lists only the caches. A pre- and post-transport cache is a single parquet file and keeps its signature next to it, in <name>.signature.txt. Templates/Signature_T handles the signature file:

  • a writer (ChainGeneration_PerBlock/<block>/Write, ChainGeneration_Single/Write, PT_Chains_Blocks/ToDo_List, Write_Car, PrePostTransport/<base set>/Write, the output per block) has <signature>/ForWriting among its ExplicitSuppliers. It writes the signature when the file does not exist yet, stops when the file holds another signature, and stops when the longest file path would exceed ModelParameters/Advanced/MaxPathLength (259), before anything is computed;
  • a reader (ChainGeneration_Single/Read, the block readers PT_Chains_Blocks/PerBlock/<block>/Read, Read_Car, PrePostTransport/<base set>/Read, the readers and merges of the output, FindMedian) builds its storage name from <signature>/ForReading and stops when the signature is missing or differs.

A hash collision therefore gives an error instead of a wrong cache. For the output, another set of parameters under the same OutputLabel gives that error too; choose another label, or let batch\UpdateAll.cmd output empty the folder.

A change in the code is not in the signature by itself. For that the signatures carry revisions: PT_Chains_Revision, CarNetwork_Revision, the revision of the pre- and post-transport caches (_parquet_r2 in their signature) and, for the OSM network cache, OSM/CacheRevision. Raise the revision when a code change alters the content of a cache. The chain cache is read only for the departure moments of PT_DepartureHours and PT_DepartureMinutes, because its time window is in its name.

Clone this wiki locally