UpdateAll and cache internals

This page is for developers of the model. It describes how batch\UpdateAll.cmd finds its engine and which items its sections ask for. It also describes how the script runs the output step, which files it leaves in batch\log, and how to run the same steps in GeoDmsGui. Finally it lists the caches with their names, and explains how the model writes and checks the signature of a cache. How to run the model with UpdateAll.cmd is described on How to run the model.

Engine

UpdateAll.cmd takes the GeoDmsRun.exe from the environment variable GEODMS_EXE, for example another engine or a test build. Without it, the script uses the installed GeoDMS of the version in its GEODMS_VERSION line (20.28.0), in the folder that the installer writes to the registry. It stops when it finds no engine. Any engine must meet the requirements under GeoDMS software. For runs of many hours with a test build, point GEODMS_EXE to a copy of the build folder (without the .pdb files), not to the build folder itself. A rebuild during the run then does not fail on files in use, and does not replace DLLs under the running process.

The items of the sections

UpdateAll.cmd asks the model for these items with GeoDmsRun. When to run a section, and when to run it again, is described under The sections on How to run the model.

• checks: ChecksBeforeRunning/Report_Settings, which every section requests first, then ChecksBeforeRunning/Report and ChecksBeforeRunning/Components. All three run as the component Merged, whose window of departure moments is the widest. The step stops at a line that begins with FAIL; a WARN, such as a combination of mode and agency without a row in the fare table (Price checks), does not stop it.
• sources: first SourceData/Infrastructure/OSM/ForBatch/CacheState and GTFS/LoadFeeds/ForBatch/CacheState in one GeoDmsRun. They write written or missing to batch\log\osm_cache.txt and gtfs_cache.txt. Then, for each cache that is missing, OSM/Make_Final_Network or GTFS/LoadFeeds/Write. See Cache names for where the caches are.
• legs: PublicTransport_Prep/Write_PrePostTransport, in one GeoDmsRun. It writes the parquet caches of the base sets of connections (PrePostTransport/<base set>/Write): Org2Stops_Walking, Org2Stops_Cycling, Stops2Dest_Cycling and, when origins and destinations differ, Stops2Dest_Walking. It writes only the base sets without a cache with the current signature (BaseSets_ToDo), and nothing with Advanced/PrePostTransportDecoupled FALSE.
• chains: first the check ChecksBeforeRunning/PT_CheckBlocks with OVprijzen/PriceCoverage/Table_OK_DOVA; the step stops when the DOVA fare table, which also prices the rides without a row in the detailed table, has empty prices or prices above the limits. The step checks has already checked the detailed table (Check_PriceCoverage). Then the stop blocks without a cache, in portions of 20 blocks (batch\WriteChainBlocks.ps1). For an OD pair, the request for the list of blocks (PT_Chains_Blocks/ToDo_List) first writes the caches of the pre- and post-transport (Write_PrePostTransport), and every block of the pair then runs in a portion of its own. With Advanced/PT_Chains_PerBlock FALSE, the list PT_Chains_Blocks/ToDo_List holds the writer of the one cache instead, PublicTransport_Prep/ChainGeneration_Single/Write, as long as that cache is missing (Single_IsWritten).
• cars: for each named moment, first PublicTransport_Prep/Direct/CarPerMoment/<moment>/ForBatch/CacheState, then Write_Car when its cache is missing. Without named moments, the script takes the moments that the run reads (MakeUnlinkedData/Step4_DirectCar/Moments): the departure moments (TD07h00m ...) for the time-dependent car trip on TomTom, and MaxSpeed on OSM. See The cache per congestion moment.
• all: per component, first its output state: NetworkSetup/GenerateOutput/OutputDir_ForBatch, Output_Sig/PathGate and ForBatch/IsComplete. For a component whose output is not complete, then the items of cars or legs that it needs, and then its output per origin block and departure moment. The output runs in portions of blocks, each in its own GeoDmsRun; see The output step. The chain step also requests the output state of the public transport components, to find the first one whose output is not complete.

The output step

In all, the output step of a component comes after its cache steps. Its portions then read the pre- and post-transport caches that the legs step wrote, instead of each computing them again. The output step has four steps:

1. The model writes the output folder to batch\log\output_dir.txt (NetworkSetup/GenerateOutput/OutputDir_ForBatch) and checks the longest path (NetworkSetup/GenerateOutput/Output_Sig/PathGate). It also writes whether the output is complete (ForBatch/IsComplete, to batch\log\output_complete.txt). The script skips a complete output. The component step has already requested these items.
2. The script deletes the files that the model writes in that folder: tt_* in the folder and in PerBlock and Mediaan, Tellingen/tellingen_*, signature.*, od_tellingen.* and unieke_od_tellingen.*. Other files stay, such as the map of an OD pair (PerBlock/od_<moment>.gpkg). It does so only when the folder is known and lies under a folder Output, or Output_<O>_<D> for an OD pair.
3. batch\WriteOutputBlocks.ps1 asks the model for the list of blocks (Generate_Output/OutputBlocks_List, written to batch\log\output_blocks.txt). This request also writes or checks the signature of the output folder. The script then runs one GeoDmsRun per portion of Impl/BlocksPerPortion blocks: Classifications/Components/BlocksPerPortion of the component, or 10 for the merge (see Model parameters). Memory is thus freed between the portions. For the merge (the component Merged), each portion merges the components per block (ComponentMerge/PerBlock/<block>/Fence/Generate).
4. One more GeoDmsRun adds up the counts per block (NetworkSetup/GenerateOutput/OD_Counts_FromFiles; see Counts next to the output).

The environment variable OUTPUT_BATCHSIZE sets the number of blocks per portion for one run. With OUTPUT_MAXBLOCKS, the step computes at most that many blocks, for a test. GeoDmsGui computes all blocks in one request with NetworkSetup/GenerateOutput/Generate_Output. That needs much more memory, because nothing is freed between the blocks.

After an abort: what is redone

Every unit of work of a section has a test for "already there", and that test decides what an aborted run has to do again. It is not the same test everywhere. What this means for a user is described under If a run is aborted.

• sources: the OSM cache and the GTFS cache are each one MMD cache, each written in a GeoDmsRun of its own. GeoDMS writes 0Dictionary.dms in the folder of the cache after the data. An interrupted cache therefore has no dictionary: the model does not read it, and makes it again from the start. The sources step reads IsWritten of both caches through ForBatch/CacheState. IsWritten tests that dictionary, and for the OSM cache also its signature.
• legs: BaseSets_ToDo holds the base sets whose IsWritten is false. IsWritten needs the signature file with the current signature and the file <name>.done next to the parquet. The writer PrePostTransport/<base set>/Write writes the parquet first and <name>.done after it. A base set whose write was aborted therefore has no .done file, and legs makes it again (#155).
• chains: PT_Chains_Blocks/PerBlock/<block>/IsWritten does not test the signature. It is an ExistingFile test on <n>of<N>.done, and for an OD pair also on the dictionary of <n>of<N>_onderdelen.mmd, the parts of the chains, which OD_Chains/PerBlock/<block>/Write_ChainParts writes in the same request as the block. The block writer ChainGeneration_PerBlock/<block>/Write writes <n>of<N>.done after the parquet of the block, when GDAL has closed the parquet and written its footer. A half-written block is therefore made again. ToDo selects the blocks that do not count as written, and WriteChainBlocks.ps1 computes those. This makes the chain step safe to interrupt.
• cars: the cache of a moment is an MMD cache with signature.txt next to 0Dictionary.dms. Write_Car computes on every request. UpdateAll.cmd cars therefore first asks for CarPerMoment/<moment>/ForBatch/CacheState, which writes written or missing to batch\log\car_cache_<moment>.txt, and skips a moment that is written. IsWritten needs the signature of the current settings and the dictionary of the cache, and in the OD pair mode also the links. The signature alone would not do: Write_Car writes it before it computes, while GeoDMS writes the dictionary only when it writes the cache. So an interrupted moment is computed again in full. To make a complete cache again, remove its folder in IntermediateResults. Without the state file, the script stops with a message instead of computing.
• The output: nothing is kept. Step 2 of the output step empties the folder, and WriteOutputBlocks.ps1 runs every portion of the list again. The time of an aborted output step is therefore lost.

Logs and helper files

The portions of the chain step write their console output to UpdateAll_chains_b<nnn>.txt, without a component, because the public transport components share the chain cache. The portions of the output step write to UpdateAll_output_<component>_b<nnn>.txt. batch\ReportUpdateAll.py [--since <moment>] [step ...] summarises the steps of the last run in UpdateAll_status.txt: the steps whose log it wrote, and the components and car moments that it skipped because their output or cache was already there. For a run that was stopped and started again, --since takes every step whose log was written since that moment. Per step it gives the start, end and duration, the number of result rows, the peak CommitCharge and PeakLiveLarge from the memory summary at the end of the log, the size of the cache and the error lines. For an output step it gives these over all its portions together, with the duration as their sum. The UpdateAll_* files are not in git. Some steps first ask the model for a value, which it writes to a text file in batch\log: checks_settings.txt, checks.txt, components.txt and variants.txt (the checks, the components of the variant, and the names of the merge variants for -variant all), osm_cache.txt and gtfs_cache.txt (the state of the source caches, for sources), car_moments.txt and car_cache_<moment>.txt (the moments of the run and the state of their caches, for cars), chain_blocks_todo.txt (the stop blocks without a cache, for chains), and output_dir.txt, output_complete.txt, output_blocks.txt and output_blocks_per_portion.txt (for the output step). They are not in git either.

Running the steps in GeoDmsGui

GeoDmsGui computes the same steps for one component, GuiComponent in ModelParameters.dms. It does so in one process, so nothing is freed between the steps or the blocks. Use it to test a step or one block, not for a complete run. The steps are under MakeUnlinkedData: right-click an item and choose Update TreeItem (see TreeView). Reload the configuration with Alt+R after you change a parameter, and after a step that writes a cache, so that the items that read the cache are computed again.

Step	Item in GeoDmsGui	Section of UpdateAll.cmd
source caches	Step1_Sources/OSM_Network and Step1_Sources/GTFS_Feed	sources
pre- and post-transport caches	Step2_PrePostTransport/Generate_All for all base sets without a cache, or Org2Stops_Walking, Org2Stops_Cycling or Stops2Dest_Cycling for one	legs
chain cache	Step3_PT_Chains/Check_Blocks, which should say PASS, and Check_PriceCoverage, which should not say FAIL; then Generate_ToDo for all stop blocks without a cache, or PerBlock/<block>/Write for one; BlocksWritten counts the blocks with a cache	chains
car caches	Step4_DirectCar/Generate_All for the moments of the run (Step4_DirectCar/Moments), or NetworkSetup/PublicTransport_Prep/Direct/CarPerMoment/<moment>/Write_Car for one	cars
output	NetworkSetup/GenerateOutput/Generate_Output for all blocks, or NetworkSetup/ConfigurationPerBlock/Block_<n>of<N>/PublicTransport/Generate_Output/Traveltime_ForEachDepTime_Fence/Generate for one; with GuiComponent 'Merged', Generate_Output merges the components (NetworkSetup/ComponentMerge/Generate)	the output step of all

Because GeoDmsGui computes for GuiComponent only, three things differ from a batch run:

• The chain cache needs a public transport component as GuiComponent. WW, CC and Car without Impl/CarTimeDependent compute one departure moment (Impl/SingleDepartureMoment), so their window of departure moments is shorter. With one of them, Generate_ToDo would write a chain cache under another name, which no public transport component reads.
• The car caches need GuiComponent 'Car', or 'Merged' with a direct car trip in the variant: Step4_DirectCar/Generate_All writes them only when the component has a direct car trip (Impl/AllowDirectCar).
• For a component, Generate_Output also adds up the counts of all blocks into od_tellingen.csv (ConfigurationPerBlock/Generate_Output/OD_Counts). For 'Merged' it writes only the counts per block, so request NetworkSetup/GenerateOutput/OD_Counts_FromFiles/Generate after it (see Counts next to the output). Without od_tellingen.csv the output counts as not complete, and UpdateAll.cmd all computes it again.

Cache names

The model reads a cache that exists, so the name of a cache is its cache key. A setting that changes the content must change the name, or the model silently reads the old cache. The caches of a run (the pre- and post-transport, chain and car caches) and each output folder therefore have a signature: the text with all settings that set its content. The OSM network cache has one too, because its stored network also depends on a few settings. The GTFS cache is named after its feed date only, because it holds the tables of the feed as they are. The name keeps a readable part and a hash of 8 characters over the signature (#118). This keeps the paths within the 259 characters that Windows allows; GeoDMS does not ask for long paths. How the model writes and checks the signatures is described below, under How the signature of a cache is checked. The names below are relative to %LocalDataProjDir%.

Chain cache

• Name: IntermediateResults/PT_Chains_<date>_<time window>_<hash>: a folder with a parquet file per stop block (<n>of<N>.parquet, with <n>of<N>.done once it is complete), or one .mmd with Advanced/PT_Chains_PerBlock FALSE.
• Signature: PublicTransport_Prep/PT_Chains_Signature, in signature.txt in the cache folder.

Car cache per moment

• Name: IntermediateResults/DirectCar_<network>_<moment>_<origin set>_<destination set><revision>_<hash>.mmd, for example DirectCar_TT2025-12_TD07h00m_b2025_b2025_r4_4ff0c20f.mmd. The name starts with the road network (NetTag): TT and the TomTom edition (TT2025-12 for yr2025_12), or OSM and the extract date. A car cache depends on the edition, not on the analysis date, so Y2025 and Y2026 share one. The sets are short in the name (Impl/OrgSet_PathTag, DestSet_PathTag: b2025 for buurt2025); the signature keeps the full names.
• Signature: PublicTransport_Prep/Direct/CarPerMoment/<moment>/Signature, in signature.txt in the cache folder: network edition, day selection, MaxCarTime, origin and destination sets with the settings that place them (Impl/OrgDest_Tag), speed cap, the settings of the car network (Impl/CarNetwork_Tag: the street types of TomTom, or the OSM network cache with the maximum speed and the speed limit, two-way roads and the junction penalties), Pareto tag, the window, start time and search of a time-dependent cache, and revision.

Pre- and post-transport cache per base set

• Name: IntermediateResults/Legs_<Set>toStops_Walking_<hash>.parquet, Legs_<Set>toStops_Cycling_<hash>.parquet, Legs_Stops2<Set>_Cycling_<hash>.parquet and, when the origins and destinations differ, Legs_Stops2<Set>_Walking_<hash>.parquet, with <Set> from Impl/OrgSet_NameTag or DestSet_NameTag (for example B2025). Next to each parquet is <name>.done once it is complete.
• Signature: PublicTransport_Prep/PrePostTransport/<base set>/Signature, in <name>.signature.txt next to the cache: type, network and speed (Impl/WalkingNetwork_Tag, CyclingNetwork_Tag: the OSM network cache by its hash, or the network of the Fietsersbond by its delivery date, its ferry settings and Impl/Fietsersbond_Revision; the junction penalties and the clean-up of the network), maximum time, origin and destination sets with the settings that place them (Impl/OrgDest_Tag), analysis date and feed, numbers of places and stops, format and revision.

GTFS cache per feed

• Name: GTFS/<GTFS_file_date>.mmd, made from the csv files in %NetworkModel_Dir%/Infrastructuur/GTFS/<GTFS_file_date>/.
• Signature: none; the feed date is the name. After a new delivery of the feed under the same date, remove the folder of the cache, because a cache that exists is read.

OSM network cache

• Name: OSM/Final_Network_<extract date><revision>_<hash>.mmd, for example Final_Network_20250101_r3_509c5dee.mmd, made from the Geofabrik extract in %NetworkModel_Dir%/Infrastructuur/OSM.
• Signature: SourceData/Infrastructure/OSM/Signature, in signature.txt in the cache folder: the extract and whether it is read per province, the car speeds of OSM roads without a speed of their own (Advanced/OSM_MaxSpeedQuantile, CarDefaultSpeed, CarDefaultSpeed_low), Advanced/Ferry_Speed, the snap distance of the extra connections (Advanced/ExtraLink_MaxSnapDistance), the fixed text _fietstel8, which keeps the names of the existing caches (#143), and OSM/CacheRevision. An example is OSM-20250101-prov_q0.5_car50-30_ferry25_snap250_fietstel8_r3. The reader takes the name and does not check the signature. The legs, the car caches and the output name the network cache by its extract date, revision and hash (Impl/OSMNetwork_Tag), so the model makes them again when the network changes.

Output

• Name: Output/<date>_<PartOfDay>_<OutputLabel>_<component>[_fb]_<hash>/, for WW and CC <date>_<origin set>_<component>[_fb]_<hash>/ (Impl/OrgSet_PathTag), or <date>_<PartOfDay>_<OutputLabel>_Merged<MergeRunNr>[_fb]_<hash>/ for a merge, with the hash of the signature. It holds signature.txt, PerBlock/tt_<moment>_<n>of<N>.parquet or .csv (for WW and CC tt_alle_<n>of<N>), Tellingen/tellingen_<n>of<N>.csv, od_tellingen.csv and unieke_od_tellingen.csv; with Advanced/MakeMedian also Mediaan/tt_<date>_mediaan_<n>of<N>; and from the items that combine the files per block, tt_<date>_<moment>.csv and tt_<date>.csv.
• Signature: NetworkSetup/GenerateOutput/Output_Signature, in signature.txt in the folder. For a component it is CompOutput_T/Signature, with the origin set, the destination set, the type, the criteria, the tags and the settings of the component (SettingsTag), with csv the exported columns, the format, and the signatures of the chain cache and the car cache. Every run computes it in the same way for every component, and it holds only settings that change the output of the component. For a merge it is the signature of NetworkSetup/ComponentMerge_Def, with the costs of WW and CC whose price the merge computes, and the hashes of the signatures of the merged components.

One OD pair

The model can also compute a single OD pair and keep the route of every option in the Pareto front. The options can then be shown on a map, read as a table and plotted as price against travel time; see Routes of an OD pair. The model does so when OD_Origin and OD_Destination are set in ModelParameters.dms. It then puts its caches in IntermediateResults_<O>_<D> and its output in Output_<O>_<D>, for example IntermediateResults_BU04020501_BU0363KP03. The caches and output of the whole model stay as they are. Without such a run, these folders do not exist.

How the signature of a cache is checked

The hash (Templates/StringHash_T, in Templates/Signature_T.dms) is a polynomial hash modulo 2^31 - 1. It is written in plain integer arithmetic in the configuration, because GeoDMS has no string hash. It is stable across versions and platforms, as a cache key must be. So caches for different settings live side by side. The signature of an MMD cache is in its own folder (signature.txt and signature.xml, next to the block caches or next to 0Dictionary.dms), so that IntermediateResults lists only the caches. A pre- and post-transport cache is a single parquet file, and keeps its signature next to it, in <name>.signature.txt. Templates/Signature_T handles the signature file:

• a writer (OSM/Make_Final_Network, ChainGeneration_PerBlock/<block>/Write, ChainGeneration_Single/Write, PT_Chains_Blocks/ToDo_List, Write_Car, PrePostTransport/<base set>/Write, the output per block) has <signature>/ForWriting among its ExplicitSuppliers. It writes the signature when the file does not exist yet. It stops when the file holds another signature, or when the longest file path would exceed ModelParameters/Advanced/MaxPathLength (259). All this happens before anything is computed;
• a reader (ChainGeneration_Single/Read, the block readers PT_Chains_Blocks/PerBlock/<block>/Read, Read_Car, PrePostTransport/<base set>/Read, the readers and merges of the output, FindMedian) builds its storage name from <signature>/ForReading and stops when the signature is missing or differs.

A hash collision therefore gives an error instead of a wrong cache. The output folder carries the hash of its signature too, so other parameters under the same OutputLabel get another folder. The error only comes with a collision, or with a signature.txt that was edited by hand; then empty the folder, or let the output step of batch\UpdateAll.cmd empty it.

A change in the code is not in the signature by itself. For code changes, the signatures carry revisions: PT_Chains_Revision, CarNetwork_Revision, the revision of the pre- and post-transport caches (_parquet_r3 in their signature) and, for the OSM network cache, OSM/CacheRevision. Raise the revision when a code change alters the content of a cache. The chain cache is read only for the departure moments of PT_DepartureHours and PT_DepartureMinutes, because its time window is in its name.