Repository navigation
Components and merging
A run of the model computes components. A component is a direct trip from origin to destination (walking, cycling or by car), or public transport with one type of pre-transport and one type of post-transport. Each component has its own output folder. The merge combines the components of a variant into one Pareto front per origin-destination pair. Which components a run computes is set with the switches of the variant; see The variant on Model parameters.
- The components
- The direct trips in the output
- Car and bicycle ownership as criteria
- Merging the components
Classifications/Components lists every component (#117, #133):
- the direct trips
WW(walking),CC(cycling) andCar(AAinModeUsed, with start and parking time); - the public transport components
<pre-transport code>_PT_<post-transport code>: every combination of the seven pre-transport types (W,C,Crm,Crmt,Cric,Cr,Cn) and the five post-transport types (W,C,rmC,rmtC,ovfC), 35 in all, such asW_PT_W,C_PT_ovfCorCrm_PT_W. The types are described on Walking and cycling legs.
A direct trip is a component of its own and never part of a public transport component. It needs no chains, the car keeps several routes per pair (Advanced/ParetoLegs_Car), and walking and cycling do not depend on the departure moment.
One GeoDmsRun computes one component, Impl/ComponentName. It follows from ConfigSettings/Overridable/RunComponent, which batch\UpdateAll.cmd sets per step in the environment variable GEODMS_Overridable_RunComponent; in GeoDmsGui, ModelParameters/GuiComponent takes its place. Everything else follows from the name: the pre- and post-transport type, the direct trips, the output folder and its signature.
What a component needs:
- A public transport component reads the pre- and post-transport caches and the chain cache (
Classifications/Components/WithPT). The output step joins them as described on From chains to journeys. -
Carreads the car caches (DirectCar). -
WWandCCcompute their trips in the output step itself.
UpdateAll.cmd therefore skips the chain cache for the direct components, and the car caches for every component but Car. The public transport components of the variant share the chain cache: its last boarding moment, and the _v in its signature, follow the longest pre-transport leg over their pre-transport types (Impl/PT_Chains_Max_V_Time), not that of one component. Because the signature of the output of a public transport component holds that of the chain cache, a change of that longest leg, for example by a pre-transport type with a longer maximum time, also gives the other public transport components a new output folder.
The output folder of a component is <Analysis_date>_<OutputLabel>_<component>_<hash> (NetworkSetup/GenerateOutput/CompOutput_T/Dir), with _fb after the component when it cycles on the network of the Fietsersbond (Impl/CyclingNetwork_DirTag), and at the end a hash of eight characters of the signature of the output, as the caches have it (Templates/StringHash_T). For WW and CC the origin set (Impl/OrgSet_PathTag, for example b2025) takes the place of OutputLabel: their output does not depend on the part of the day, so every part of the day and every variant label reads the same folder. Output with other settings therefore gets a folder of its own beside the earlier one, and switching back to the earlier settings finds the earlier output again. Its subfolder PerBlock holds tt_<moment>_<n>of<N>.parquet (or .csv); the date and the component are in the folder name, the settings in signature.txt, which keeps the paths short.
WW and CC do not depend on the departure moment (Classifications/Components/TimeInvariant). They compute one moment per block, and their file applies to every departure moment of every part of the day: tt_alle_<n>of<N>.parquet (Classifications/Components/FileMoment). Car without Impl/CarTimeDependent also computes one moment per block, the congestion moment of the part of the day (Classifications/Components/SingleDepartureMoment, 07:30 in the morning rush), and its file carries that moment, because the TomTom speeds of the car depend on it.
The signature of the output folder mentions the chain cache and the public transport settings only for a public transport component, and the car settings only for Car. It holds every setting that changes the output of the component and is in no cache it reads, with its value, also the default, and no setting that does not change it, so that runs that differ only in such a setting share the output:
- for
WWandCC: the maximum time of the trip, the network with its speed, the OSM network cache, the junction penalties and the clean-up of the network, and forCCthe start time. Their costs per minute and whether they count inPriceare in it only with csv, which writes the price; with parquet the merge computes the price ofWWandCC, and these costs are in the signature of the merge (ComponentMerge_Def/DirectPriceTag).WWandCChave one route per pair, so the criteria and the epsilon of a front are not in their signature either; - for a public transport component:
MaxTravelTime, the wait at home (Advanced/MaxWaitingTimeAtOrigin,IncludeWaitingAtHomeInTravelTime), whether the costs of walking and cycling count inPrice, and per pre- and post-transport type its network, the maximum time of its base set and its own, the start, parking and penalty times of cycling, and its price per minute or the rent of the OV-fiets with the search distance to its stops (Classifications/TimeInvariantTypes/OutputSignatureTag); - for
Car:Car_StartTimeandCar_ParkingTime, which are added to the travel time when the car cache is read; - with csv the exported columns (
Export_TravelledDistance,Export_PriceInformation); parquet always has all columns of its kind; - the departure moment of
CarwithoutImpl/CarTimeDependent,Advanced/MakeMedian, and the settings that place the origins and destinations (Impl/OrgDest_Tag: the weight of the centroids, the CBS year of a region set without a year of its own, the BAG date, the public transport nodes as destinations).
Every run computes the signature of every component the same (CompOutput_T), so the merge, Output/FrontCheck and Output/OD_PairRoutes find the folder of each component they read. For that the signatures of the chain cache and the car cache follow from the departure moments of the part of the day (Impl/PartOfDay_Moments) and the criteria from MinimiseCriteria (Classifications/ComponentParetoAttr), not from the component of the run.
The direct components go through the same output step as the public transport components (PerDepartureMoment_T), without pre-transport, chains and post-transport (Impl/WithPT FALSE): their options (AllOptions) are the direct trips of the component.
- The trips come from
PublicTransport_Prep/Direct:WalkingandCycling(see step 5 of Walking and cycling legs) and the car per congestion moment (see Private transport). -
PublicTransport_Prep/Direct_ODholds the direct trips of the component that do not depend on the departure moment, with duration, distance, price, augmented price and mode.NetworkSetup/impl/Direct_ODadds, once for all blocks, the places of origin and destination, the ownership the trip needs (NeedsCar,NeedsBike) and the origin block and part of each row; each part selects its rows. - With
Impl/CarTimeDependent(default with the TomTom network) the car trip depends on the departure moment and is not inDirect_OD. Each departure moment reads the rows of its origins from the car cache of that moment (CarRows) and makes the front on travel time and distance there (DirectCar, templateCarFront_T). - A direct trip leaves at the departure moment and arrives after its duration. Its time, distance and price are in the pre-transport columns (
PretransTime,TravelDist_V,PretransPrice); public transport and post-transport are zero.ModeUsedisWW,CCorAA. - The two condensations of step 7 of From chains to journeys keep the front per pair. Walking and cycling have one route per pair. The car keeps the routes of its front on travel time and distance, whose price follows from the distance.
The price of a direct trip:
| Trip | Price |
Price_augm |
|---|---|---|
| walking |
Direct_Walking_Time_Costs per minute with Advanced/IncludeWalkingCostsInPrice, else 0 |
Direct_Walking_Time_Costs per minute |
| cycling |
Direct_Cycling_Time_Costs per minute with Advanced/IncludeCyclingCostsInPrice, else 0 |
Direct_Cycling_Time_Costs per minute |
| car | the distance times Car_CostPerKm plus Car_FixedCosts_PerTrip, with Advanced/IncludeCarCostsInPrice
|
equal to Price
|
MinimiseCriteria, and in the merge MergeCriteria, can also name Car and Bike (#117). Each adds a criterion that is true or false:
-
NeedsCar: the direct car trip (AA). There is no park-and-ride pre-transport, so no public transport journey needs a car. -
NeedsBike: the direct cycling trip (CC), and a journey with a pre- or post-transport leg on one's own bicycle. The OV-fiets is rented, soovfCdoes not need one's own bicycle (Classifications/TimeInvariantTypes/NeedsOwnBike).
The criteria are minimised, false before true. An option that needs neither a car nor a bicycle is therefore never dominated by one that does: a public transport journey that is dearer and slower than the car trip stays next to it, whereas a car trip that is dearer and slower than a public transport journey still drops out. One front then serves every group of travellers:
- with a car and a bicycle: all rows;
- without a car: the rows with
NeedsCar0; - without a car and a bicycle: the rows with
NeedsCarandNeedsBikeboth 0.
Without these criteria such journeys drop out, and an analysis of travellers without a car finds no option on those pairs, or a slower one than exists. Output/PostAnalysis/FindMedian, for example, leaves out the car trips (ExcludedModes = 'AA') to describe accessibility without a car.
Rules:
- Within one component each criterion is the same for every option, so in the run of a component it does not change the front. The criteria matter in the merge; the default of
MinimiseCriteria('Price,Time') does not name them and that ofMergeCriteria('Price,Time,Car,Bike') does. - A criterion that cannot differ between the options is left out (
Classifications/OwnershipParetoAttr_Candidates/IsRelevant), also from the signature:NeedsCarcounts only with the direct car trip,NeedsBikeonly with the direct cycling trip or a pre- or post-transport type on one's own bicycle, and in the merge only when a merged component needs one's own bicycle (Classifications/Components/NeedsBike). - The chain cache does not depend on them: the chain joiner computes public transport only.
- The signature of the output carries the extra criteria in its
MinCritpart, as the letters ofModeUsed:MinCrit-Price_Time_A_C. - Both columns,
NeedsCarandNeedsBike(0 or 1), are in the csv output of a component and in the merged output, whatever the criteria; in the parquet output of a component they are constant and left out. - The criteria need the true-or-false criteria of
pareto_optimal_eps(GeoDMS #1287), which every engine that the model accepts has.
How many options of each component the criteria keep in a merge is shown under MergeCriteria.
Merging is optional and computes no trips itself. With ModelParameters/Merge on, batch\UpdateAll.cmd ends with the merge, as the component Merged, also when the variant has one component. It merges, per origin block and departure moment, the output of the components of the variant into one Pareto front per origin-destination pair (NetworkSetup/ComponentMerge). With one component it still fills in the columns that the parquet of the component leaves out and applies MergeCriteria, so that every variant has its final table in a Merged folder, with the same columns (#148); with 'Time', for example, only the fastest of the car routes of Car stays.
- The merge reads the parquet files of the components (
<Analysis_date>_<OutputLabel>_<component>_<hash>/PerBlock, see Signature), with a reader per kind of component (Classifications/Components/ExportClass:Direct,CarorPT). The components must therefore be written withOutputFormat'parquet', with the times in whole seconds and the columns of The parquet files. That format is in the signature of a component (_parquet-s3-), and the merge reads only a folder whose signature equals the one it computes; so it never reads an older output, which has the times in minutes, which GDAL would read as seconds without a warning, or lacks the distances and prices of pre- and post-transport or the walking and waiting at transfers. A component with one departure moment (WW,CC, andCarwithoutCarTimeDependent) serves every departure moment of the merge. - It fills in the columns that are constant in a component or follow from its columns: the price of
WWandCCfrom the travel time and that ofCarfrom the distance, as inPublicTransport_Prep/Direct;ModeUsedWW,CCorAA;NeedsCarandNeedsBikefrom the component; zeros for the missing times and distances. A direct trip is its own pre-transport, as in the csv of a component: its time, distance and price are alsoPretransTime,TravelDist_VandPretransPrice. Every row gets the columnComponent, the component it comes from. - It splits time, price and distance per part and per mode, as the csv of a component does (see The csv files):
PTTimeisTraveltimeminusPretransTimeandPosttransTime, and minusWaitingTimewhenAdvanced/IncludeWaitingAtHomeInTravelTimeis on;InVehicleTimeisPTTimeminusTransferWalkTimeandPTWaitTime;PTPriceisPriceminusPretransPriceandPosttransPrice, in cents; walking, cycling and car get the time, price and distance of the pre- and post-transport with that mode. The mode of the pre- and post-transport follows from the component (Classifications/Components/PreNetworkTypeandPostNetworkType); the OV-fiets counts as cycling. - It keeps per origin-destination pair the rows that are not dominated, with
pareto_optimal_epsonClassifications/OutputParetoAttr, the same operator and epsilons as in a run, with price and time in cents and seconds. The rows are in the order of the components inClassifications/Components, direct trips first; of rows that are equal on every criterion the first stays. - It writes the result in
OutputFormat, parquet or csv, to<Analysis_date>_<OutputLabel>_Merged<MergeRunNr>_<hash>, one block per phase, with the counts per block inTellingen, whichNetworkSetup/GenerateOutput/OD_Counts_FromFilesadds up. Both formats have the same columns in the same order, whatever the components of the variant; a column that does not apply to a row is 0. The times are in minutes, in the parquet as float32, while the parquet of a component has them in whole seconds (#149). The columns of the split come afterComponent, so that a reader that reads the csv by position keeps the earlier ones.
The merge has its own criteria, MergeCriteria (default 'Price,Time,Car,Bike'), apart from MinimiseCriteria of the components: any subset of Price (or Price_Augm), Time, Car and Bike, with at least Price or Time (an integrity check). With 'Time' or 'Price' alone each pair and departure moment keeps exactly one row, the fastest or the cheapest. Car and Bike count only as far as a merged component can need them.
With IncludeWalkingCostsInPrice FALSE, the default, walking is free. The direct walking trip then stays in the merged front of every pair within walking range, and an option that is not more than 1% faster drops out against it.
The folder of a merge is <Analysis_date>_<OutputLabel>_Merged<MergeRunNr>_<hash>, with _fb before the hash when a component cycles on the network of the Fietsersbond (ComponentMerge_Def/Dir). Its signature lists the components, each with the hash of the signature of its output ([W_PT_W:<hash>]), the criteria, the format (<OutputFormat>-m6) and the costs of WW and CC, whose price it computes, so that it changes when a component is computed with other settings, and a merge with other criteria or other components gets a folder of its own; MergeRunNr only labels the merge. The full signatures of the components are in their own folders; in the signature of the merge they would make it too long for the hash.
The merge computes the folder of every component from the current settings (ComponentMerge_Def/PerComp, with NetworkSetup/GenerateOutput/CompOutput_T), and reads a component only when the signature.txt in that folder equals the signature it computes (ComponentMerge_Def/CompDir). Otherwise it stops, with an error that names the component: that component has not been computed with the current settings.
Because the 1% rule of the front is not transitive, merging can keep or drop a few options that lie within about 2% of a cheaper one, compared with one run over the same options. Example: CC, C_PT_W and C_PT_ovfC merged, against one run over the same options (Dronten, 5 blocks, morning rush of 2023), differed in 275 of 359,127 lines; every pair kept an option, and every differing line was within 2.5% of the travel time of an equal or cheaper option in both.
NetworkModel PBL documentation