Output files

This page describes the files of the output step: the files per block with the options per origin-destination pair, the counts and the median beside them, and the items that merge the files per block into larger files. Where the output of a component goes is on How to run the model.

The files per block

Every origin block and departure moment gives one file, PerBlock/tt_<moment>_<n>of<N>.parquet, or .csv with OutputFormat 'csv'. It has one row per option: per origin-destination pair, the journeys of its Pareto front. How a public transport component computes them is on From chains to journeys; the direct trips and the merge are on Components and merging.

The parquet files

A parquet file (zstd) of a component holds only the columns that vary within the component and cannot be derived from its other columns. Its kind follows from the component (Impl/ExportClass):

Kind	Components	Columns after OrgName and DestName
Direct	WW, CC	Traveltime, TravelDist_V
Car	Car	Traveltime, TravelDist_Car
PT	the public transport components	Traveltime, Price, Price_Augm, ModeUsed, WaitingTime, PretransTime, PosttransTime, TravelDist_Bus, TravelDist_Metro, TravelDist_Tram, TravelDist_Rail, TravelDist_Ferry, TravelDist_V, TravelDist_N, PretransPrice, PosttransPrice, TransferWalkTime, PTWaitTime
merged	Merged	Traveltime, Price, Price_Augm, ModeUsed, NeedsCar, NeedsBike, WaitingTime, PretransTime, PosttransTime, the five public transport distances, TravelDist_Car, Component, and the split per part and per mode of the csv files: TravelDist_V, TravelDist_N, PTTime, PretransPrice, PTPrice, PosttransPrice, TravelDist_PT, TravelDist_Walking, TravelDist_Cycling, WalkingTime, CyclingTime, CarTime, WalkingPrice, CyclingPrice, CarPrice, InVehicleTime, TransferWalkTime, PTWaitTime

Times are in whole seconds (uint16, which GDAL writes as int16), prices in euro (float32) and distances in metres (uint32); an unknown value is null. The signature of the output folder names _parquet-s3- for this format. The departure moment is in the file name, not in the column names.

The distance of a walking or cycling trip or leg is in the file, because it does not follow from its time: the time holds the junction penalties (Advanced/Junction_Penalties), for cycling the start and parking times, and on the Fietsersbond network the wait at a ferry. The prices of the pre- and post-transport leg are in the file too, so that the parts add up to Price exactly in cents; recomputed from the times, which are rounded to whole seconds, they can differ by a cent. The other columns are constant within a component (ModeUsed of a direct trip, NeedsCar, NeedsBike, WaitingTime 0 of a direct trip) or follow from these columns: PTTime is Traveltime minus the pre- and post-transport time (and the wait at home when it counts), InVehicleTime is PTTime minus TransferWalkTime and PTWaitTime, PTPrice is Price minus the price of the pre- and post-transport leg, and the columns per mode follow from the mode of the pre- and post-transport of the component. The merge fills them in (see Merging the components).

The csv files

A csv file is semicolon-delimited, with the departure moment after each column name. CreateExports/Columns in PerDepartureMoment_T sets which columns it has and in which order: one row per column, with its name and its kind. A column of kind dist is written only with ModelParameters/Export_TravelledDistance, a column of kind price only with ModelParameters/Export_PriceInformation (both apply to the csv files only). The header and the lines are built from that table, and a new column goes at the end, so that a reader by position keeps the existing ones.

The columns after OrgName and DestName:

1. Traveltime, Price, Price_Augm, ModeUsed;
2. NeedsCar and NeedsBike, 0 or 1, whatever MinimiseCriteria says (see Car and bicycle ownership as criteria); a reader that takes the columns by position must skip them, and the readers of the model read by name;
3. the distances of the pre-transport leg, of the public transport per mode and of the post-transport leg: TravelDist_V, TravelDist_Bus, TravelDist_Metro, TravelDist_Tram, TravelDist_Rail, TravelDist_Ferry, TravelDist_N;
4. the times of the waiting at home, the pre-transport leg, the public transport and the post-transport leg: WaitingTime, PretransTime, PTTime, PosttransTime;
5. the prices of the pre-transport leg, the public transport and the post-transport leg: PretransPrice, PTPrice, PosttransPrice;
6. distance, time and price per mode, and TravelDist_Car (table below);
7. the public transport time split into the time in the vehicle, walking at transfers and waiting: InVehicleTime, TransferWalkTime, PTWaitTime, which add up to PTTime.

These three prices split Price like the travel time, and add up to Price exactly, in cents:

• PretransPrice: the pre-transport leg, as far as it counts in Price: a cycling leg with Advanced/IncludeCyclingCostsInPrice, a walking leg with Advanced/IncludeWalkingCostsInPrice (default FALSE, so 0). For a direct trip (WW, CC, AA) this column holds the price of the whole trip, as PretransTime and TravelDist_V hold its time and distance.
• PTPrice: the public transport chain, fares and surcharges; 0 for a direct trip.
• PosttransPrice: the post-transport leg, under the same rules as the pre-transport leg; the OV-fiets counts with its rental price.

Price_Augm is not split. Distance, time and price are also given per mode:

Category	Distance	Time	Price
pre-transport	TravelDist_V	PretransTime	PretransPrice
public transport	TravelDist_PT (and per mode TravelDist_Bus to TravelDist_Ferry)	PTTime	PTPrice
post-transport	TravelDist_N	PosttransTime	PosttransPrice
walking	TravelDist_Walking	WalkingTime	WalkingPrice
cycling	TravelDist_Cycling	CyclingTime	CyclingPrice
car	TravelDist_Car	CarTime	CarPrice

Public transport as a mode is the chain, including walking and waiting at transfers. Its time splits into the time in the vehicles (InVehicleTime), walking at transfers (TransferWalkTime, with Advanced/GradeSeparatedTransferPenalty) and waiting (PTWaitTime): at transfers, and the extra waiting time of a ride on a flex line (Advanced/FlexExtraWaitingTime). The chain joiner adds them up per chain, and the chain cache keeps them (see Public transport, Travel time). Walking, cycling and car are the pre- and post-transport legs and the direct trip of that mode; the OV-fiets counts as cycling. The wait at home (WaitingTime) belongs to none of the modes. For this split the final result carries the mode of the pre- and post-transport leg (V_mode_rel, N_mode_rel); a direct trip has its mode in V_mode_rel and none in N_mode_rel.

Counts next to the output

The output step also writes the counts of each block, Tellingen/tellingen_<n>of<N>.csv in the output folder, in the same request as the files per block (PublicTransport/Generate_Output/Traveltime_ForEachDepTime/Counts). They come from the final result of each departure moment, which is in memory anyway for the file per block, behind the fence of that moment (PerDepartureMoment_T, CreateExports/LongFormat/Results_Traveltime/Counts). After the last portion, batch\WriteOutputBlocks.ps1 runs batch\OdTellingen.py, which adds up the blocks, each with its own origins, into two tables in the output folder:

File	Rows	Columns
od_tellingen.csv	the rows of the output per kind of option (Classifications/OD_Counts_Kind): alle, OV-keten (a chain with pre- and post-transport), and the direct trips WW, CC and AA	soort, one column per departure moment (07h00m ...), totaal
unieke_od_tellingen.csv	the unique origin-destination pairs with at least one option of that kind, or for that group of travellers (Classifications/OD_Counts_Group): alle, OV-keten, WW, CC, AA, zonder auto (NeedsCar 0) and zonder auto en fiets (NeedsCar and NeedsBike 0)	groep, one column per departure moment, samen (the union over the departure moments)

In one request for all blocks (NetworkSetup/GenerateOutput/Generate_Output in GeoDmsGui), NetworkSetup/ConfigurationPerBlock/Generate_Output/OD_Counts writes the two tables itself.

python batch\OdTellingen.py <output folder> [--force] can also be run on its own. It adds up the files in Tellingen, or, when there are none, counts the per-block csv files itself. It stops when blocks or moments are missing or when the two tables already exist (--force overwrites). The GeoDMS Academy compares a Python and a GeoDMS implementation of these counts, with measured run times.

Median travel time per pair

With Advanced/MakeMedian the output step also writes the median per origin-destination pair over the departure moments, one file per block, Mediaan/tt_<Analysis_date>_mediaan_<n>of<N>.csv (.parquet with OutputFormat 'parquet'), in the same request as the files per block (Generate_Output/Traveltime_ForEachDepTime/Median). The parquet has the times in whole seconds, the csv in minutes. It is FALSE by default, because the medians are not part of the delivery.

Per departure moment it takes the cheapest option of each pair: the lowest Price, at equal price the lowest Traveltime. Over the departure moments with an option it writes:

• Traveltime_median, the median travel time (rth_element; for an even number the mean of the two middle values), and Traveltime_min and Traveltime_max, the fastest and the slowest;
• TravelDist_median, the median distance (TravelDist_V + TravelDist_PT + TravelDist_N, in m);
• Price_median, the median price in euro;
• NrMoments, the number of departure moments with an option; 1 for a component with one departure moment.

It takes all options of the component. For Car that is the direct car trip, whose travel time includes the start and parking time (Car_StartTime + Car_ParkingTime, 5 min) on top of the limit MaxCarTime (90 min) on the driving time.

Merging the output files

Three items combine the files per block of a run into larger files, each building on the one before. They read the csv files per block, with the departure moment in the column names, so they need a run with OutputFormat 'csv'.

Per departure moment

NetworkSetup/ConfigurationPerBlock/Merge_Output/MergeRegions/OUTPUT_Merge_PublicTransport_Regions_fullOD_long_CSVFiles reads the per-block files of each departure moment and writes one file per departure moment, <output folder>/tt_<Analysis_date>_<moment>.csv. Its columns are OrgName, DestName, Traveltime, ModeUsed, NeedsCar, NeedsBike and Price, and distance, time and price per category:

• for the pre-transport leg, the public transport and the post-transport leg: TravelDist_V, TravelDist_PT, TravelDist_N, WaitingTime, PretransTime, PTTime, PosttransTime, PretransPrice, PTPrice and PosttransPrice;
• per mode: TravelDist_Walking, TravelDist_Cycling, WalkingTime, CyclingTime, CarTime, WalkingPrice, CyclingPrice and CarPrice.

A column is empty when the per-block file does not have it, such as the prices without Export_PriceInformation; see The csv files.

Over the departure moments

NetworkSetup/ConfigurationPerBlock/Merge_Output/MergeTimes/OUTPUT_Merge_PublicTransport_Times_fullOD_long_CSVFiles reads the files per departure moment and writes <output folder>/tt_<Analysis_date>.csv: all departure moments below each other, with a column for the moment and the same columns as the files per departure moment.

The median over the departure moments

Output/PostAnalysis/FindMedian/Result/MedianTT reads the file over all departure moments. It writes per origin and destination the fastest front row of each departure moment, and the median of those over the departure moments with a connection (NrMoments), for the pairs with a median up to FindMedian/SearchLimit (30 minutes). Rows whose ModeUsed is in FindMedian/ExcludedModes (default AA, the direct car trip) do not count, so that the median describes accessibility without a car.

Folder, checks and size

<output folder> is %LocalDataProjDir%/Output/<Analysis_date>_<OutputLabel>_<component>_<hash>, the folder of the current settings. The readers check its signature first and stop when it is missing or differs from the current parameters (NetworkSetup/GenerateOutput/Output_Sig/CheckGate). They build the file names as the writer per block does, use its column names, read Price only when the file has that column, and read and write through GDAL. The merged files are comma-delimited with quoted text, and the numbers are written as numbers (float64, and uint8 for NeedsCar and NeedsBike), unquoted, with an empty field for an empty value. A number takes 8 bytes or less per row instead of a text, which matters for the merge over the departure moments, and float64 writes the text of the files per block back unchanged.

As an example of the size: one departure moment of 1 October 2024 over 450 origin blocks is 39 million rows. The merge per moment took about 35 minutes and 15 GB, the merge over the moments about 30 minutes for that one moment, and FindMedian 7 minutes.