Repository navigation
UpdateAll and cache internals
This page is for developers of the model. It describes how batch\UpdateAll.cmd finds its engine and runs the output step, which files it leaves in batch\log, and how the model writes and checks the signature of a cache. How to run the model with UpdateAll.cmd is on How to run the model.
UpdateAll.cmd takes the GeoDmsRun.exe it uses from, in this order: the environment variable GEODMS_EXE, for example another engine or a test build; the local build C:\dev\GeoDMS_2026\bin\Release\x64\GeoDmsRun.exe; and the installed GeoDMS of its GEODMS_VERSION line (20.26.0), in the folder that the installer writes to the registry. It stops when it finds none. Any engine must meet GeoDMS software. For runs of many hours a copy of the build folder (without the .pdb files) is safer than the build folder itself: a rebuild during the run then neither fails on files in use nor replaces DLLs under the running process.
The output section runs in five steps:
- The model writes the pre- and post-transport caches of the base sets without one, as the
legssection does (withallthat section has done it already, and-nolegsskips it), so that the portions read them instead of each computing them again. - The model writes the output folder to
batch\log\output_dir.txt(NetworkSetup/GenerateOutput/OutputDir_ForBatch) and checks the longest path (NetworkSetup/GenerateOutput/Output_Sig/PathGate). - The script deletes the files that the model writes in that folder:
tt_*andsignature.*in the folder, the files inPerBlock,MediaanandTellingen, andod_tellingen.*andunieke_od_tellingen.*. It does so only when the folder lies under a folderOutput, orOutput_<O>_<D>for an OD pair. -
batch\WriteOutputBlocks.ps1asks the model for the list of blocks (Generate_Output/OutputBlocks_List, written tobatch\log\output_blocks.txt), which also writes or checks the signature of the output folder. It then runs oneGeoDmsRunper portion ofAdvanced/Components/BlocksPerPortionblocks (see Model parameters), so that memory is released between the portions. WithMergeOutputeach portion merges the components per block (ComponentMerge/PerBlock/<block>/Fence/Generate). -
batch\OdTellingen.pyadds up the counts per block (see Counts next to the output).
With the environment variable OUTPUT_SINGLE, steps 4 and 5 are one request for all blocks, NetworkSetup/GenerateOutput/Generate_Output. While the file batch\pause_before_output.flag exists, the output section waits before it starts, without a computing process.
The portions of the chain and output steps write their console output to UpdateAll_chains_b<nnn>.txt and UpdateAll_output_b<nnn>.txt. batch\ReportUpdateAll.py [step ...] summarises the last run of each step: start, end, duration, number of result rows, the peak CommitCharge and PeakLiveLarge from the memory summary at the end of the log, the size of the cache and the error lines. The UpdateAll_* files are not in git. Some steps first ask the model for a value, which it writes to a text file in batch\log: car_moments.txt (the moments of the run, for cars), chain_blocks_todo.txt (the stop blocks without a cache, for chains), and output_dir.txt, output_blocks.txt and output_blocks_per_portion.txt (for output). They are not in git either.
The hash (Templates/StringHash_T, in Templates/Signature_T.dms) is a polynomial hash modulo 2^31 - 1 written in plain integer arithmetic in the configuration, because GeoDMS has no string hash; it is stable across versions and platforms, as a cache key must be. So caches for different settings live side by side. The signature of an MMD cache is in its own folder (signature.txt and signature.xml, next to the block caches or next to 0Dictionary.dms), so that IntermediateResults lists only the caches. A pre- and post-transport cache is a single parquet file and keeps its signature next to it, in <name>.signature.txt. Templates/Signature_T handles the signature file:
- a writer (
ChainGeneration_PerBlock/<block>/Write,ChainGeneration_Single/Write,PT_Chains_Blocks/ToDo_List,Write_Car,PrePostTransport/<base set>/Write, the output per block) has<signature>/ForWritingamong itsExplicitSuppliers. It writes the signature when the file does not exist yet, stops when the file holds another signature, and stops when the longest file path would exceedModelParameters/Advanced/MaxPathLength(259), before anything is computed; - a reader (
ChainGeneration_Single/Read, the block readersPT_Chains_Blocks/PerBlock/<block>/Read,Read_Car,PrePostTransport/<base set>/Read, the readers and merges of the output,FindMedian) builds its storage name from<signature>/ForReadingand stops when the signature is missing or differs.
A hash collision therefore gives an error instead of a wrong cache. For the output, another set of parameters under the same OutputLabel gives that error too; choose another label, or let batch\UpdateAll.cmd output empty the folder.
A change in the code is not in the signature by itself. For that the signatures carry revisions: PT_Chains_Revision, CarNetwork_Revision, the revision of the pre- and post-transport caches (_parquet_r2 in their signature) and, for the OSM network cache, OSM/CacheRevision. Raise the revision when a code change alters the content of a cache. The chain cache is read only for the departure moments of PT_DepartureHours and PT_DepartureMinutes, because its time window is in its name.
NetworkModel PBL documentation