Repository navigation
UpdateAll and cache internals
This page is for developers of the model. It describes how batch\UpdateAll.cmd finds its engine, which items its sections ask for, how it runs the output step and which files it leaves in batch\log. It also lists the caches with their names, and how the model writes and checks the signature of a cache. How to run the model with UpdateAll.cmd is on How to run the model.
UpdateAll.cmd takes the GeoDmsRun.exe it uses from, in this order: the environment variable GEODMS_EXE, for example another engine or a test build; the local build C:\dev\GeoDMS_2026\bin\Release\x64\GeoDmsRun.exe; and the installed GeoDMS of its GEODMS_VERSION line (20.27.0), in the folder that the installer writes to the registry. It stops when it finds none. Any engine must meet GeoDMS software. For runs of many hours a copy of the build folder (without the .pdb files) is safer than the build folder itself: a rebuild during the run then neither fails on files in use nor replaces DLLs under the running process.
UpdateAll.cmd asks the model for these items with GeoDmsRun. When to run a section, and when to run it again, is under The sections on How to run the model.
-
sources:SourceData/Infrastructure/OSM/Make_Final_NetworkandGTFS/LoadFeeds/Write. See Cache names for where the caches are. -
legs:PublicTransport_Prep/Write_PrePostTransport, in oneGeoDmsRun: the parquet caches of the base sets of legs (PrePostTransport/<base set>/Write),Org2Stops_Walking,Org2Stops_Cycling,Stops2Dest_Cyclingand, when origins and destinations differ,Stops2Dest_Walking. It writes only the base sets without a cache with the current signature (BaseSets_ToDo), and nothing withAdvanced/PrePostTransportDecoupledFALSE. -
chains: the checkChecksBeforeRunning/PT_CheckBlockswithOVprijzen/PriceCoverage/Table_OK_DOVA(the step stops when the price coverage is not complete), then the stop blocks without a cache, in portions of 20 blocks (batch\WriteChainBlocks.ps1). With the environment variableCHAINCACHE_SINGLEit writes one cache instead,PublicTransport_Prep/ChainGeneration_Single/Write(withAdvanced/PT_Chains_PerBlockFALSE). -
cars:PublicTransport_Prep/Direct/CarPerMoment/<moment>/Write_Carfor the named moments. Without moments, the moments the run reads (MakeUnlinkedData/Step4_DirectCar/Moments): the departure moments (TD07h00m...) for the time-dependent car trip on TomTom, andMaxSpeedon OSM; see step 5. -
output: the output per origin block and departure moment, in portions of blocks, each in its ownGeoDmsRun; see The output step.
The output section runs in five steps:
- The model writes the pre- and post-transport caches of the base sets without one, as the
legssection does (withallthat section has done it already, and-nolegsskips it), so that the portions read them instead of each computing them again. - The model writes the output folder to
batch\log\output_dir.txt(NetworkSetup/GenerateOutput/OutputDir_ForBatch) and checks the longest path (NetworkSetup/GenerateOutput/Output_Sig/PathGate). - The script deletes the files that the model writes in that folder:
tt_*andsignature.*in the folder, the files inPerBlock,MediaanandTellingen, andod_tellingen.*andunieke_od_tellingen.*. It does so only when the folder lies under a folderOutput, orOutput_<O>_<D>for an OD pair. -
batch\WriteOutputBlocks.ps1asks the model for the list of blocks (Generate_Output/OutputBlocks_List, written tobatch\log\output_blocks.txt), which also writes or checks the signature of the output folder. It then runs oneGeoDmsRunper portion ofClassifications/Components/BlocksPerPortionblocks (see Model parameters), so that memory is released between the portions. For the merge (the componentMerged) each portion merges the components per block (ComponentMerge/PerBlock/<block>/Fence/Generate). -
batch\OdTellingen.pyadds up the counts per block (see Counts next to the output).
With the environment variable OUTPUT_SINGLE, steps 4 and 5 are one request for all blocks, NetworkSetup/GenerateOutput/Generate_Output. While the file batch\pause_before_output.flag exists, the output section waits before it starts, without a computing process.
Every unit of work of a section has a test for "already there", and that test decides what an aborted run has to do again. It is not the same test everywhere. What this means for a user is on If a run is aborted.
-
sources: the OSM cache and the GTFS cache are each one MMD cache, written in oneGeoDmsRun. GeoDMS writes0Dictionary.dmsin the folder of the cache after the data, so an interrupted cache has no dictionary, is not read, and is made again from the start. -
legs:BaseSets_ToDoholds the base sets whoseIsWrittenis false, andIsWrittenisSig/IsChecked: the signature file exists and equals the current signature.Signature_T/ForWritingwrites that file before the writer writes the parquet, in a measured run two minutes before it, so a base set whose write was aborted in between counts as written while its parquet is missing or incomplete, and the reader fails. DeleteLegs_<name>_<hash>.*and letlegsmake it again. -
chains:PT_Chains_Blocks/PerBlock/<block>/IsWrittenis not the signature but anExistingFiletest on<n>of<N>.done. The block writerChainGeneration_PerBlock/<block>/Writewrites that file after the parquet of the block, when GDAL has closed it and written its footer, so a half-written block is made again.ToDoselects the blocks without one andWriteChainBlocks.ps1computes those, which is what makes the chain step safe to interrupt. -
cars: the cache of a moment is an MMD cache withsignature.txtnext to0Dictionary.dms.UpdateAll.cmd carsasks forWrite_Carof every moment of the run; a moment with a complete cache costs nothing, and an interrupted moment is computed again in full. -
output: nothing is kept. Step 3 above empties the folder andWriteOutputBlocks.ps1runs every portion of the list again, so the time of an aborted output step is lost.
The portions of the chain and output steps write their console output to UpdateAll_chains_<component>_b<nnn>.txt and UpdateAll_output_<component>_b<nnn>.txt. batch\ReportUpdateAll.py [step ...] summarises the last run of each step: start, end, duration, number of result rows, the peak CommitCharge and PeakLiveLarge from the memory summary at the end of the log, the size of the cache and the error lines. The UpdateAll_* files are not in git. Some steps first ask the model for a value, which it writes to a text file in batch\log: car_moments.txt (the moments of the run, for cars), chain_blocks_todo.txt (the stop blocks without a cache, for chains), and output_dir.txt, output_blocks.txt and output_blocks_per_portion.txt (for output). They are not in git either.
A cache that exists is read, so its name is its cache key: a setting that changes the content must change the name, or the model silently reads the old cache. The caches of a run (the pre- and post-transport, chain and car caches) and each output folder therefore have a signature: the text with all settings that set its content. The source caches are named after their edition instead: the feed date and the OSM extract date with its revision. The name keeps a readable part and, for the caches, a hash of 8 characters over the signature (#118). This keeps the paths within the 259 characters that Windows allows; GeoDMS does not ask for long paths. How the model writes and checks the signatures is on UpdateAll and cache internals. The names below are relative to %LocalDataProjDir%.
- Name:
IntermediateResults/PT_Chains_<date>_<time window>_<hash>: a folder with a parquet file per stop block (<n>of<N>.parquet, with<n>of<N>.doneonce it is complete), or one.mmdwithAdvanced/PT_Chains_PerBlockFALSE. - Signature:
PublicTransport_Prep/PT_Chains_Signature, insignature.txtin the cache folder.
- Name:
IntermediateResults/DirectCar_<network>_<moment>_<origin set>_<destination set><revision>_<hash>.mmd, for exampleDirectCar_TT2025-12_TD07h00m_b2025_b2025_r4_4ff0c20f.mmd. The name starts with the road network (NetTag:TTand the TomTom edition,TT2025-12foryr2025_12, orOSMand the extract date): a car cache depends on the edition, not on the reference date, soY2025andY2026share one. The sets are short (Impl/OrgSet_PathTag,DestSet_PathTag:b2025forbuurt2025); the signature keeps the full names. - Signature:
PublicTransport_Prep/Direct/CarPerMoment/<moment>/Signature, insignature.txtin the cache folder: network edition, day selection,MaxCarTime, origin and destination sets, speed cap, Pareto tag, the window, start time and search of a time-dependent cache, and revision.
- Name:
IntermediateResults/Legs_<Set>toStops_Walking_<hash>.parquet,Legs_<Set>toStops_Cycling_<hash>.parquetandLegs_Stops2<Set>_Cycling_<hash>.parquet, with<Set>fromImpl/OrgSet_NameTagorDestSet_NameTag(for exampleB2025). - Signature:
PublicTransport_Prep/PrePostTransport/<base set>/Signature, in<name>.signature.txtnext to the cache: type, network and speed, maximum time, origin and destination sets, reference date and feed, numbers of places and stops, format and revision.
- Name:
GTFS/<GTFS_file_date>.mmd, made from the csv files in%NetworkModel_Dir%/Infrastructuur/GTFS/<GTFS_file_date>/. - Signature: none; the feed date is the name.
- Name:
OSM/Final_Network_<extract date><revision>.mmd, made from the Geofabrik extract in%NetworkModel_Dir%/Infrastructuur/OSM. - Signature: none; the extract date and
OSM/CacheRevisionare the name.
- Name:
Output/<date>_<OutputLabel>_<component>[_fb]/, or<date>_<OutputLabel>_Merged<MergeRunNr>[_fb]/for a merge. It holdssignature.txt,PerBlock/tt_<moment>_<n>of<N>.parquetor.csv,Tellingen/tellingen_<n>of<N>.csv,od_tellingen.csvandunieke_od_tellingen.csv; withAdvanced/MakeMedianalsoMediaan/tt_<date>_mediaan_<n>of<N>; and from the merge itemstt_<date>_<moment>.csvandtt_<date>.csv. - Signature:
NetworkSetup/GenerateOutput/Output_Signature, insignature.txtin the folder: origin set,PT_Output_ParamTag, exported columns and format, and the signatures of the chain cache and the car cache. For a merge it is the signature ofNetworkSetup/ComponentMerge_Def, with the signatures of the merged components.
The model can also compute a single origin-destination pair and keep the route of every option on the front, so that the options can be shown on a map, read as a table and plotted as price against travel time; see Routes of an OD pair. It does so when OD_Origin and OD_Destination are set in ModelParameters.dms, and then puts its caches in IntermediateResults_<O>_<D> and its output in Output_<O>_<D>, for example IntermediateResults_BU04020501_BU0363KP03. The caches and output of the whole model stay as they are. Without such a run these folders do not exist.
The hash (Templates/StringHash_T, in Templates/Signature_T.dms) is a polynomial hash modulo 2^31 - 1 written in plain integer arithmetic in the configuration, because GeoDMS has no string hash; it is stable across versions and platforms, as a cache key must be. So caches for different settings live side by side. The signature of an MMD cache is in its own folder (signature.txt and signature.xml, next to the block caches or next to 0Dictionary.dms), so that IntermediateResults lists only the caches. A pre- and post-transport cache is a single parquet file and keeps its signature next to it, in <name>.signature.txt. Templates/Signature_T handles the signature file:
- a writer (
ChainGeneration_PerBlock/<block>/Write,ChainGeneration_Single/Write,PT_Chains_Blocks/ToDo_List,Write_Car,PrePostTransport/<base set>/Write, the output per block) has<signature>/ForWritingamong itsExplicitSuppliers. It writes the signature when the file does not exist yet, stops when the file holds another signature, and stops when the longest file path would exceedModelParameters/Advanced/MaxPathLength(259), before anything is computed; - a reader (
ChainGeneration_Single/Read, the block readersPT_Chains_Blocks/PerBlock/<block>/Read,Read_Car,PrePostTransport/<base set>/Read, the readers and merges of the output,FindMedian) builds its storage name from<signature>/ForReadingand stops when the signature is missing or differs.
A hash collision therefore gives an error instead of a wrong cache. For the output, another set of parameters under the same OutputLabel gives that error too; choose another label, or let batch\UpdateAll.cmd output empty the folder.
A change in the code is not in the signature by itself. For that the signatures carry revisions: PT_Chains_Revision, CarNetwork_Revision, the revision of the pre- and post-transport caches (_parquet_r2 in their signature) and, for the OSM network cache, OSM/CacheRevision. Raise the revision when a code change alters the content of a cache. The chain cache is read only for the departure moments of PT_DepartureHours and PT_DepartureMinutes, because its time window is in its name.
NetworkModel PBL documentation