Output files

This page describes the files of the output step. These are the files per block with the options per origin-destination (OD) pair, the counts and the median beside them, and the items that combine the files per block into larger files. Where the output of a component goes is described on How to run the model.

The files per block

Every origin block and departure moment gives one file, PerBlock/tt_<moment>_<n>of<N>.parquet, or .csv with OutputFormat 'csv'. WW and CC do not depend on the departure moment. They have one file per block, tt_alle_<n>of<N>.parquet, that applies to every departure moment. Car without Impl/CarTimeDependent has one file per block, with the congestion moment in its name. A file has one row per option: per OD pair, the journeys of its Pareto front. How a public transport component computes them is described on From chains to journeys; the direct trips and the merge are on Components and merging.

The parquet files

A parquet file (zstd) of a component holds only the columns that vary within the component and cannot be derived from its other columns. Its kind follows from the component (Impl/ExportClass):

Kind	Components	Columns after OrgName and DestName
Direct	WW, CC	Traveltime, TravelDist_V
Car	Car	Traveltime, TravelDist_Car
PT	the public transport components	Traveltime, Price, Price_Augm, ModeUsed, WaitingTime, PretransTime, PosttransTime, TravelDist_Bus, TravelDist_Metro, TravelDist_Tram, TravelDist_Rail, TravelDist_Ferry, TravelDist_V, TravelDist_N, PretransPrice, PosttransPrice, TransferWalkTime, PTWaitTime
merged	Merged	Traveltime, Price, Price_Augm, ModeUsed, NeedsCar, NeedsBike, WaitingTime, PretransTime, PosttransTime, the five public transport distances, TravelDist_Car, Component, and the split per part and per mode of the csv files: TravelDist_V, TravelDist_N, PTTime, PretransPrice, PTPrice, PosttransPrice, TravelDist_PT, TravelDist_Walking, TravelDist_Cycling, WalkingTime, CyclingTime, CarTime, WalkingPrice, CyclingPrice, CarPrice, InVehicleTime, TransferWalkTime, PTWaitTime

In the parquet of a component the times are in whole seconds (uint16, which GDAL writes as int16); in that of the merge they are in minutes (float32), as in the csv files. Prices are in euro (float32) and distances in metres (uint32); an unknown value is null. The signature of the output folder names this format: _parquet-s3- for a component, _parquet-m6 for the merge. The departure moment is in the file name, not in the column names.

The merge reads the parquet files of its components, and with OutputFormat 'csv' the components write only csv. So a run with a merge (Merge, or a merge variant) needs 'parquet': otherwise ChecksBeforeRunning/Check_OutputFormat gives a FAIL, and batch\UpdateAll.cmd stops before it makes any cache or output.

The distance of a walking or cycling trip or leg is in the file, because it does not follow from its time. The time holds the junction penalties (Advanced/Junction_Penalties), for cycling the start and parking times, and on the Fietsersbond network the wait at a ferry. The prices of the pre- and post-transport leg are in the file too, so that the parts add up to Price exactly in cents. Prices recomputed from the times, which are rounded to whole seconds, can differ by a cent. The other columns are constant within a component (ModeUsed of a direct trip, NeedsCar, NeedsBike, WaitingTime 0 of a direct trip), or they follow from these columns. PTTime is Traveltime minus the pre- and post-transport time (and the wait at home when it counts). InVehicleTime is PTTime minus TransferWalkTime and PTWaitTime. PTPrice is Price minus the price of the pre- and post-transport leg. The columns per mode follow from the mode of the pre- and post-transport of the component. The merge fills them in (see Merging the components).

The csv files

A csv file is semicolon-delimited, with the departure moment after each column name, or alle for WW and CC. CreateExports/Columns in PerDepartureMoment_T sets which columns it has and in which order: one row per column, with its name and its kind. A column of kind dist is written only with ModelParameters/Export_TravelledDistance, and a column of kind price only with ModelParameters/Export_PriceInformation; both settings apply to the csv files only. The header and the lines are built from that table. A new column goes at the end, so that a reader that reads by position keeps the existing columns.

The columns after OrgName and DestName:

1. Traveltime, Price, Price_Augm, ModeUsed;
2. NeedsCar and NeedsBike, 0 or 1, whatever MinimiseCriteria says (see Car and bicycle ownership as criteria); a reader that takes the columns by position must skip them, and the readers of the model read by name;
3. the distances of the pre-transport leg, of the public transport per mode and of the post-transport leg: TravelDist_V, TravelDist_Bus, TravelDist_Metro, TravelDist_Tram, TravelDist_Rail, TravelDist_Ferry, TravelDist_N;
4. the times of the waiting at home, the pre-transport leg, the public transport and the post-transport leg: WaitingTime, PretransTime, PTTime, PosttransTime;
5. the prices of the pre-transport leg, the public transport and the post-transport leg: PretransPrice, PTPrice, PosttransPrice;
6. distance, time and price per mode, and TravelDist_Car (table below);
7. the public transport time split into the time in the vehicle, walking at transfers and waiting: InVehicleTime, TransferWalkTime, PTWaitTime, which add up to PTTime.

These three prices split Price like the travel time, and add up to Price exactly, in cents:

• PretransPrice: the pre-transport leg, as far as it counts in Price: a cycling leg with Advanced/IncludeCyclingCostsInPrice, a walking leg with Advanced/IncludeWalkingCostsInPrice (default FALSE, so 0). For a direct trip (WW, CC, AA) this column holds the price of the whole trip, as PretransTime and TravelDist_V hold its time and distance.
• PTPrice: the public transport chain, fares and surcharges; 0 for a direct trip.
• PosttransPrice: the post-transport leg, under the same rules as the pre-transport leg; the rent of the OV-fiets always counts.

Price_Augm is not split. Distance, time and price are also given per mode:

Category	Distance	Time	Price
pre-transport	TravelDist_V	PretransTime	PretransPrice
public transport	TravelDist_PT (and per mode TravelDist_Bus to TravelDist_Ferry)	PTTime	PTPrice
post-transport	TravelDist_N	PosttransTime	PosttransPrice
walking	TravelDist_Walking	WalkingTime	WalkingPrice
cycling	TravelDist_Cycling	CyclingTime	CyclingPrice
car	TravelDist_Car	CarTime	CarPrice

Public transport as a mode is the chain, including walking and waiting at transfers. Its time splits into three parts: the time in the vehicles (InVehicleTime), walking at transfers (TransferWalkTime, with Advanced/GradeSeparatedTransferPenalty), and waiting (PTWaitTime). The waiting is the waiting at transfers and the extra time per boarding of a ride on a flex line (Advanced/FlexExtraWaitingTime) or on a train with a trip surcharge (OVprijzen/NS/TripSurchargeTrains/ExtraTime, 0 by default). The chain joiner adds them up per chain, and the chain cache keeps them (see Public transport, Travel time). Walking, cycling and car are the pre- and post-transport legs and the direct trip of that mode; the OV-fiets counts as cycling. The wait at home (WaitingTime) belongs to none of the modes. For this split the final result carries the mode of the pre- and post-transport leg (V_mode_rel, N_mode_rel); a direct trip has its mode in V_mode_rel and none in N_mode_rel.

Counts next to the output

The output step also writes the counts of each block, Tellingen/tellingen_<n>of<N>.csv in the output folder. It does so in the same request as the files per block (PublicTransport/Generate_Output/Traveltime_ForEachDepTime/Counts). The counts come from the final result of each departure moment, behind the fence of that moment (PerDepartureMoment_T, CreateExports/LongFormat/Results_Traveltime/Counts); that result is in memory anyway for the file per block. After the last portion, batch\WriteOutputBlocks.ps1 requests NetworkSetup/GenerateOutput/OD_Counts_FromFiles/Generate in one more GeoDmsRun. It adds up the files of the blocks, each with its own origins, into two tables in the output folder:

File	Rows	Columns
od_tellingen.csv	the rows of the output per kind of option (Classifications/OD_Counts_Kind): alle, OV-keten (a chain with pre- and post-transport), and the direct trips WW, CC and AA	soort, one column per departure moment (07h00m ...), totaal
unieke_od_tellingen.csv	the unique OD pairs with at least one option of that kind, or for that group of travellers (Classifications/OD_Counts_Group): alle, OV-keten, WW, CC, AA, zonder auto (NeedsCar 0) and zonder auto en fiets (NeedsCar and NeedsBike 0)	groep, one column per departure moment, samen (the union over the departure moments)

In one request for all blocks (NetworkSetup/GenerateOutput/Generate_Output in GeoDmsGui), NetworkSetup/ConfigurationPerBlock/Generate_Output/OD_Counts writes the two tables itself.

OD_Counts_FromFiles/Generate can also be requested on its own, in GeoDmsGui or with GeoDmsRun, for the output folder of the current settings. It writes nothing when the file of a block is missing or has other departure moments than the run (OD_Counts_FromFiles/IsConsistent). The GeoDMS Academy compares a Python and a GeoDMS implementation of these counts, with measured run times.

Median travel time per pair

With Advanced/MakeMedian, the output step also writes the median per OD pair over the departure moments. It writes one file per block, Mediaan/tt_<Analysis_date>_mediaan_<n>of<N>.csv (.parquet with OutputFormat 'parquet'), in the same request as the files per block (Generate_Output/Traveltime_ForEachDepTime/Median). The parquet has the times in whole seconds, the csv in minutes. MakeMedian is FALSE by default, because the medians are not part of the delivery.

Per departure moment it takes the cheapest option of each pair: the lowest Price, at equal price the lowest Traveltime. Over the departure moments with an option it writes:

• Traveltime_median, the median travel time (rth_element; for an even number the mean of the two middle values), and Traveltime_min and Traveltime_max, the fastest and the slowest;
• TravelDist_median, the median distance (TravelDist_V + TravelDist_PT + TravelDist_N, in m);
• Price_median, the median price in euro;
• NrMoments, the number of departure moments with an option; 1 for a component with one departure moment.

It takes all options of the component. For Car that is the direct car trip, whose travel time includes the start and parking time (Car_StartTime + Car_ParkingTime, 5 min) on top of the limit MaxCarTime (90 min) on the driving time.

Combining the output files

Three items combine the files per block of one component into larger files, each building on the one before. They work on the output folder of the component of the current settings (GuiComponent in GeoDmsGui). They read the csv files per block, with the moment in the column names, so they need output written with OutputFormat 'csv'; a run with such output cannot merge (see The parquet files).

Per departure moment

NetworkSetup/ConfigurationPerBlock/Merge_Output/MergeRegions/OUTPUT_Merge_PublicTransport_Regions_fullOD_long_CSVFiles reads the files per block of each departure moment and writes one file per departure moment, <output folder>/tt_<Analysis_date>_<moment>.csv. Its columns are, in this order (Templates/Merge_Regions_OV_T):

• OrgName, DestName, Traveltime, ModeUsed, NeedsCar, NeedsBike and Price;
• the times of the waiting at home, the pre-transport leg, the public transport and the post-transport leg: WaitingTime, PretransTime, PTTime and PosttransTime;
• the prices of the pre-transport leg, the public transport and the post-transport leg: PretransPrice, PTPrice and PosttransPrice;
• the distances of these three parts: TravelDist_V, TravelDist_PT and TravelDist_N;
• distance, time and price per mode: TravelDist_Walking, TravelDist_Cycling, TravelDist_Car, WalkingTime, CyclingTime, CarTime, WalkingPrice, CyclingPrice and CarPrice.

Price_Augm, the distances per public transport mode and the split of PTTime are not in these files. A column is empty when the file per block does not have it, such as the prices without Export_PriceInformation; see The csv files.

Over the departure moments

NetworkSetup/ConfigurationPerBlock/Merge_Output/MergeTimes/OUTPUT_Merge_PublicTransport_Times_fullOD_long_CSVFiles reads the files per departure moment and writes <output folder>/tt_<Analysis_date>.csv: all departure moments below each other, with the moment in the column DepTime after DestName, and then the same columns as the files per departure moment.

The median over the departure moments

Output/PostAnalysis/FindMedian reads the file over all departure moments. Per origin and destination it takes the shortest travel time of each departure moment, and then the median of those over the departure moments with a connection (NrMoments). FindMedian/Result holds the pairs with a median up to FindMedian/SearchLimit (30 minutes), with their MedianTT and NrMoments; FindMedian writes no file. Rows whose ModeUsed is in FindMedian/ExcludedModes (default AA, the direct car trip) do not count, so that the median describes accessibility without a car.

Folder, checks and size

<output folder> is the output folder of the component with the current settings, %LocalDataProjDir%/Output/<Analysis_date>_<PartOfDay>_<OutputLabel>_<component>_<hash> (see Output folder and departure moments). The readers check its signature first and stop when it is missing or differs from the current parameters (NetworkSetup/GenerateOutput/Output_Sig/CheckGate). The readers of the files per block take the moment in the file and column names as the writer per block gives it (Impl/MeasureMoments/FileTag): the departure moment, or alle for WW and CC, whose one file per block applies to every departure moment. The readers fill a column that the file does not have with empty values, and they read and write through GDAL. The combined files are comma-delimited with quoted text. The numbers are written as numbers (float64, and uint8 for NeedsCar and NeedsBike), unquoted, with an empty field for an empty value. A departure moment over all blocks has tens of millions of rows, and a number takes 8 bytes or less per row instead of a text, which matters most for the file over the departure moments. And float64 writes the text of the files per block back unchanged.