Repository navigation
UpdateAll and cache internals
This page is for developers of the model. It describes how batch\UpdateAll.cmd finds its engine, which items its sections ask for, how it runs the output step and which files it leaves in batch\log, and how to run the same steps in GeoDmsGui. It also lists the caches with their names, and how the model writes and checks the signature of a cache. How to run the model with UpdateAll.cmd is on How to run the model.
UpdateAll.cmd takes the GeoDmsRun.exe it uses from the environment variable GEODMS_EXE, for example another engine or a test build, and otherwise from the installed GeoDMS of its GEODMS_VERSION line (20.28.0), in the folder that the installer writes to the registry. It stops when it finds none. Any engine must meet GeoDMS software. For runs of many hours with a test build, point GEODMS_EXE to a copy of the build folder (without the .pdb files) rather than the build folder itself: a rebuild during the run then neither fails on files in use nor replaces DLLs under the running process.
UpdateAll.cmd asks the model for these items with GeoDmsRun. When to run a section, and when to run it again, is under The sections on How to run the model.
-
checks:ChecksBeforeRunning/Report_Settings, which every section requests first, thenChecksBeforeRunning/ReportandChecksBeforeRunning/Components, all as the componentMerged, whose window of departure moments is the widest. -
sources:SourceData/Infrastructure/OSM/ForBatch/CacheStateandGTFS/LoadFeeds/ForBatch/CacheStatein oneGeoDmsRun, which writewrittenormissingtobatch\log\osm_cache.txtandgtfs_cache.txt, and then for a cache that is missingOSM/Make_Final_NetworkorGTFS/LoadFeeds/Write. See Cache names for where the caches are. -
legs:PublicTransport_Prep/Write_PrePostTransport, in oneGeoDmsRun: the parquet caches of the base sets of legs (PrePostTransport/<base set>/Write),Org2Stops_Walking,Org2Stops_Cycling,Stops2Dest_Cyclingand, when origins and destinations differ,Stops2Dest_Walking. It writes only the base sets without a cache with the current signature (BaseSets_ToDo), and nothing withAdvanced/PrePostTransportDecoupledFALSE. -
chains: the checkChecksBeforeRunning/PT_CheckBlockswithOVprijzen/PriceCoverage/Table_OK_DOVA(the step stops when the price coverage is not complete), then the stop blocks without a cache, in portions of 20 blocks (batch\WriteChainBlocks.ps1). WithAdvanced/PT_Chains_PerBlockFALSEthe list ofPT_Chains_Blocks/ToDo_Listholds instead the writer of the one cache,PublicTransport_Prep/ChainGeneration_Single/Write, as long as that cache is missing (Single_IsWritten). -
cars: per moment firstPublicTransport_Prep/Direct/CarPerMoment/<moment>/ForBatch/CacheState, thenWrite_Carwhen its cache is missing, for the named moments. Without moments, the moments the run reads (MakeUnlinkedData/Step4_DirectCar/Moments): the departure moments (TD07h00m...) for the time-dependent car trip on TomTom, andMaxSpeedon OSM; see The cache per congestion moment. -
all: per component first its output state,NetworkSetup/GenerateOutput/OutputDir_ForBatch,Output_Sig/PathGateandForBatch/IsComplete, and for a component whose output is not complete the items ofcarsorlegsthat it needs and then its output, per origin block and departure moment, in portions of blocks, each in its ownGeoDmsRun; see The output step. The chain step requests the output state of the public transport components too, to find the first one whose output is not complete.
The output step of a component comes after its cache steps in all, so that its portions read the pre- and post-transport caches that the legs step wrote instead of each computing them again. It runs in four steps:
- The model writes the output folder to
batch\log\output_dir.txt(NetworkSetup/GenerateOutput/OutputDir_ForBatch), checks the longest path (NetworkSetup/GenerateOutput/Output_Sig/PathGate) and writes whether the output is complete (ForBatch/IsComplete, tobatch\log\output_complete.txt); a complete output is skipped. The component step has requested this already. - The script deletes the files that the model writes in that folder:
tt_*andsignature.*in the folder, the files inPerBlock,MediaanandTellingen, andod_tellingen.*andunieke_od_tellingen.*. It does so only when the folder is known and lies under a folderOutput, orOutput_<O>_<D>for an OD pair. -
batch\WriteOutputBlocks.ps1asks the model for the list of blocks (Generate_Output/OutputBlocks_List, written tobatch\log\output_blocks.txt), which also writes or checks the signature of the output folder. It then runs oneGeoDmsRunper portion ofClassifications/Components/BlocksPerPortionblocks (see Model parameters), so that memory is released between the portions. For the merge (the componentMerged) each portion merges the components per block (ComponentMerge/PerBlock/<block>/Fence/Generate). -
batch\OdTellingen.pyadds up the counts per block (see Counts next to the output).
The environment variable OUTPUT_BATCHSIZE sets the number of blocks per portion for one run, and OUTPUT_MAXBLOCKS computes at most that many blocks, for a test. GeoDmsGui computes all blocks in one request with NetworkSetup/GenerateOutput/Generate_Output, which needs much more memory because nothing is freed between the blocks.
Every unit of work of a section has a test for "already there", and that test decides what an aborted run has to do again. It is not the same test everywhere. What this means for a user is on If a run is aborted.
-
sources: the OSM cache and the GTFS cache are each one MMD cache, written in oneGeoDmsRun. GeoDMS writes0Dictionary.dmsin the folder of the cache after the data, so an interrupted cache has no dictionary, is not read, and is made again from the start.IsWrittenof both caches, which thesourcesstep reads throughForBatch/CacheState, tests that dictionary, and for the OSM cache also its signature. -
legs:BaseSets_ToDoholds the base sets whoseIsWrittenis false, andIsWrittenisSig/IsChecked: the signature file exists and equals the current signature.Signature_T/ForWritingwrites that file before the writer writes the parquet, in a measured run two minutes before it, so a base set whose write was aborted in between counts as written while its parquet is missing or incomplete, and the reader fails. DeleteLegs_<name>_<hash>.*and letlegsmake it again. -
chains:PT_Chains_Blocks/PerBlock/<block>/IsWrittenis not the signature but anExistingFiletest on<n>of<N>.done. The block writerChainGeneration_PerBlock/<block>/Writewrites that file after the parquet of the block, when GDAL has closed it and written its footer, so a half-written block is made again.ToDoselects the blocks without one andWriteChainBlocks.ps1computes those, which is what makes the chain step safe to interrupt. -
cars: the cache of a moment is an MMD cache withsignature.txtnext to0Dictionary.dms.Write_Carcomputes on every request, soUpdateAll.cmd carsfirst asks forCarPerMoment/<moment>/ForBatch/CacheState, which writeswrittenormissingtobatch\log\car_cache_<moment>.txt, and skips a moment that is written.IsWrittenneeds the signature of the current settings and the dictionary of the cache, and in the OD pair mode also the links. The signature alone would not do:Write_Carwrites it before it computes, while GeoDMS writes the dictionary only when it writes the cache. So an interrupted moment is computed again in full. To make a complete cache again, remove its folder inIntermediateResults. Without the state file the script stops with a message instead of computing. - The output: nothing is kept. Step 2 of the output step empties the folder and
WriteOutputBlocks.ps1runs every portion of the list again, so the time of an aborted output step is lost.
The portions of the chain and output steps write their console output to UpdateAll_chains_b<nnn>.txt, without a component because the public transport components share the chain cache, and UpdateAll_output_<component>_b<nnn>.txt. batch\ReportUpdateAll.py [--since <moment>] [step ...] summarises the steps of the last run in UpdateAll_status.txt: the steps whose log it wrote, and the components and car moments it skipped because their output or cache was there. For a run that was stopped and started again, --since takes every step whose log was written since that moment. Per step it gives start, end, duration, number of result rows, the peak CommitCharge and PeakLiveLarge from the memory summary at the end of the log, the size of the cache and the error lines; for an output step over all its portions together, with the duration as their sum. The UpdateAll_* files are not in git. Some steps first ask the model for a value, which it writes to a text file in batch\log: checks_settings.txt, checks.txt and components.txt (the checks and the components of the variant), osm_cache.txt and gtfs_cache.txt (the state of the source caches, for sources), car_moments.txt and car_cache_<moment>.txt (the moments of the run and the state of their caches, for cars), chain_blocks_todo.txt (the stop blocks without a cache, for chains), and output_dir.txt, output_complete.txt, output_blocks.txt and output_blocks_per_portion.txt (for the output step). They are not in git either.
GeoDmsGui computes the same steps for one component, GuiComponent in ModelParameters.dms, in one process, so that nothing is freed between the steps or the blocks. Use it to test a step or one block, not for a complete run. The steps are under MakeUnlinkedData: right-click an item and choose Update TreeItem (see TreeView). Reload the configuration with Alt+R after a change of a parameter and after a step that writes a cache, so that the items that read the cache are computed again.
| Step | Item in GeoDmsGui | Section of UpdateAll.cmd
|
|---|---|---|
| source caches |
Step1_Sources/OSM_Network and Step1_Sources/GTFS_Feed
|
sources |
| pre- and post-transport caches |
Step2_PrePostTransport/Generate_All for all base sets without a cache, or Org2Stops_Walking, Org2Stops_Cycling or Stops2Dest_Cycling for one |
legs |
| chain cache |
Step3_PT_Chains/Check_Blocks and Check_PriceCoverage, which should both say PASS; then Generate_ToDo for all stop blocks without a cache, or PerBlock/<block>/Write for one; BlocksWritten counts the blocks with a cache |
chains |
| car caches |
Step4_DirectCar/Generate_All for the moments of the run (Step4_DirectCar/Moments), or NetworkSetup/PublicTransport_Prep/Direct/CarPerMoment/<moment>/Write_Car for one |
cars |
| output |
NetworkSetup/GenerateOutput/Generate_Output for all blocks, or NetworkSetup/ConfigurationPerBlock/Block_<n>of<N>/PublicTransport/Generate_Output/Traveltime_ForEachDepTime_Fence/Generate for one; with GuiComponent 'Merged', Generate_Output merges the components (NetworkSetup/ComponentMerge/Generate) |
the output step of all
|
Because GeoDmsGui computes for GuiComponent only, three things differ from a batch run:
- The chain cache needs a public transport component as
GuiComponent.WW,CCandCarwithoutImpl/CarTimeDependentcompute one departure moment (Impl/SingleDepartureMoment), so their window of departure moments is shorter, andGenerate_ToDowould write a chain cache under another name, which no public transport component reads. - The car caches need
GuiComponent'Car', or'Merged'with a direct car trip in the variant:Step4_DirectCar/Generate_Allwrites them only when the component has a direct car trip (Impl/AllowDirectCar). - For a component,
Generate_Outputalso adds up the counts of all blocks intood_tellingen.csv(ConfigurationPerBlock/Generate_Output/OD_Counts). For'Merged'it writes only the counts per block; runpython batch\OdTellingen.py <output folder>after it (see Counts next to the output). Withoutod_tellingen.csvthe output counts as not complete, andUpdateAll.cmd allcomputes it again.
A cache that exists is read, so its name is its cache key: a setting that changes the content must change the name, or the model silently reads the old cache. The caches of a run (the pre- and post-transport, chain and car caches) and each output folder therefore have a signature: the text with all settings that set its content. So does the OSM network cache, whose stored network depends on a few settings as well. The GTFS cache is named after its feed date only, because it holds the tables of the feed as they are. The name keeps a readable part and, for the caches, a hash of 8 characters over the signature (#118). This keeps the paths within the 259 characters that Windows allows; GeoDMS does not ask for long paths. How the model writes and checks the signatures is on UpdateAll and cache internals. The names below are relative to %LocalDataProjDir%.
- Name:
IntermediateResults/PT_Chains_<date>_<time window>_<hash>: a folder with a parquet file per stop block (<n>of<N>.parquet, with<n>of<N>.doneonce it is complete), or one.mmdwithAdvanced/PT_Chains_PerBlockFALSE. - Signature:
PublicTransport_Prep/PT_Chains_Signature, insignature.txtin the cache folder.
- Name:
IntermediateResults/DirectCar_<network>_<moment>_<origin set>_<destination set><revision>_<hash>.mmd, for exampleDirectCar_TT2025-12_TD07h00m_b2025_b2025_r4_4ff0c20f.mmd. The name starts with the road network (NetTag:TTand the TomTom edition,TT2025-12foryr2025_12, orOSMand the extract date): a car cache depends on the edition, not on the reference date, soY2025andY2026share one. The sets are short (Impl/OrgSet_PathTag,DestSet_PathTag:b2025forbuurt2025); the signature keeps the full names. - Signature:
PublicTransport_Prep/Direct/CarPerMoment/<moment>/Signature, insignature.txtin the cache folder: network edition, day selection,MaxCarTime, origin and destination sets, speed cap, Pareto tag, the window, start time and search of a time-dependent cache, and revision.
- Name:
IntermediateResults/Legs_<Set>toStops_Walking_<hash>.parquet,Legs_<Set>toStops_Cycling_<hash>.parquetandLegs_Stops2<Set>_Cycling_<hash>.parquet, with<Set>fromImpl/OrgSet_NameTagorDestSet_NameTag(for exampleB2025). - Signature:
PublicTransport_Prep/PrePostTransport/<base set>/Signature, in<name>.signature.txtnext to the cache: type, network and speed, maximum time, origin and destination sets, reference date and feed, numbers of places and stops, format and revision.
- Name:
GTFS/<GTFS_file_date>.mmd, made from the csv files in%NetworkModel_Dir%/Infrastructuur/GTFS/<GTFS_file_date>/. - Signature: none; the feed date is the name. After a new delivery of the feed under the same date, remove the folder of the cache, because a cache that exists is read.
- Name:
OSM/Final_Network_<extract date><revision>_<hash>.mmd, for exampleFinal_Network_20250101_r3_509c5dee.mmd, made from the Geofabrik extract in%NetworkModel_Dir%/Infrastructuur/OSM. - Signature:
SourceData/Infrastructure/OSM/Signature, insignature.txtin the cache folder: the extract and whether it is read per province, the car speeds of OSM roads without a speed of their own (Advanced/OSM_MaxSpeedQuantile,CarDefaultSpeed,CarDefaultSpeed_low),Advanced/Ferry_Speed, the snap distance of the extra connections (Advanced/ExtraLink_MaxSnapDistance), the link distance to the Fietstelweek for the cycling speeds (Advanced/Fietstel_OSM_link_dist) andOSM/CacheRevision, for exampleOSM-20250101-prov_q0.5_car50-30_ferry25_snap250_fietstel8_r3. The reader takes the name and does not check the signature. The other caches name the network only by its extract date, so they do not change with the hash.
- Name:
Output/<date>_<OutputLabel>_<component>[_fb]/, or<date>_<OutputLabel>_Merged<MergeRunNr>[_fb]/for a merge. It holdssignature.txt,PerBlock/tt_<moment>_<n>of<N>.parquetor.csv,Tellingen/tellingen_<n>of<N>.csv,od_tellingen.csvandunieke_od_tellingen.csv; withAdvanced/MakeMedianalsoMediaan/tt_<date>_mediaan_<n>of<N>; and from the merge itemstt_<date>_<moment>.csvandtt_<date>.csv. - Signature:
NetworkSetup/GenerateOutput/Output_Signature, insignature.txtin the folder: origin set,PT_Output_ParamTag, exported columns and format, and the signatures of the chain cache and the car cache. For a merge it is the signature ofNetworkSetup/ComponentMerge_Def, with the signatures of the merged components.
The model can also compute a single origin-destination pair and keep the route of every option on the front, so that the options can be shown on a map, read as a table and plotted as price against travel time; see Routes of an OD pair. It does so when OD_Origin and OD_Destination are set in ModelParameters.dms, and then puts its caches in IntermediateResults_<O>_<D> and its output in Output_<O>_<D>, for example IntermediateResults_BU04020501_BU0363KP03. The caches and output of the whole model stay as they are. Without such a run these folders do not exist.
The hash (Templates/StringHash_T, in Templates/Signature_T.dms) is a polynomial hash modulo 2^31 - 1 written in plain integer arithmetic in the configuration, because GeoDMS has no string hash; it is stable across versions and platforms, as a cache key must be. So caches for different settings live side by side. The signature of an MMD cache is in its own folder (signature.txt and signature.xml, next to the block caches or next to 0Dictionary.dms), so that IntermediateResults lists only the caches. A pre- and post-transport cache is a single parquet file and keeps its signature next to it, in <name>.signature.txt. Templates/Signature_T handles the signature file:
- a writer (
OSM/Make_Final_Network,ChainGeneration_PerBlock/<block>/Write,ChainGeneration_Single/Write,PT_Chains_Blocks/ToDo_List,Write_Car,PrePostTransport/<base set>/Write, the output per block) has<signature>/ForWritingamong itsExplicitSuppliers. It writes the signature when the file does not exist yet, stops when the file holds another signature, and stops when the longest file path would exceedModelParameters/Advanced/MaxPathLength(259), before anything is computed; - a reader (
ChainGeneration_Single/Read, the block readersPT_Chains_Blocks/PerBlock/<block>/Read,Read_Car,PrePostTransport/<base set>/Read, the readers and merges of the output,FindMedian) builds its storage name from<signature>/ForReadingand stops when the signature is missing or differs.
A hash collision therefore gives an error instead of a wrong cache. For the output, another set of parameters under the same OutputLabel gives that error too; choose another label, or let the output step of batch\UpdateAll.cmd empty the folder.
A change in the code is not in the signature by itself. For that the signatures carry revisions: PT_Chains_Revision, CarNetwork_Revision, the revision of the pre- and post-transport caches (_parquet_r2 in their signature) and, for the OSM network cache, OSM/CacheRevision. Raise the revision when a code change alters the content of a cache. The chain cache is read only for the departure moments of PT_DepartureHours and PT_DepartureMinutes, because its time window is in its name.
NetworkModel PBL documentation