Validating the Pareto front

/Output/FrontCheck compares the public transport output of a run with a reference output of the same component (#53). It checks the hard requirement of a front: no option of an origin-destination (OD) pair is dominated by another option of the same pair. It also shows how the reachability, the shortest travel time and the lowest price changed against the reference. The reference can be an earlier run or delivery, or a run that leaves out the price criterion.

Running the check

1. Set ModelParameters to the settings of the run that you check. Its output folder follows from them: the analysis date, OutputLabel, the component and the hash of the signature of the output (see Output folder and departure moments). The departure moments of the run and the number of origin blocks (Advanced/NumberOfOrgBlocks) must be those of both outputs.
2. Set the parameters of cfg/main/Output/FrontCheck.dms, in the file or in GeoDmsGui:

Parameter	Default	Meaning
Component	W_PT_W	The public transport component, a name from Classifications/Components.
Dir	the output folder of Component with the current settings	%LocalDataProjDir%/Output/<Analysis_date>_<PartOfDay>_<OutputLabel>_<component>_<hash>, as the run of the component names it (CompOutput).
RefDir	Dir	The output folder of the reference. When it equals Dir, only the dominated options count.
Tag	<PartOfDay>_<OutputLabel>_<Component>, for WW and CC <OutputLabel>_<Component>	The name of the check in the file name.
MinPerUnit, RefMinPerUnit	1/60	Minutes per unit of Traveltime in Dir and RefDir: 1/60 for output in seconds, 1 for output in minutes of older versions of the model.
NrBlocks	10	The origin blocks 1 to NrBlocks are compared.
TimeMargin, PriceMargin	1 minute, 0.10 euro	A change counts beyond these margins.

3. Request /Output/FrontCheck/File and /Output/FrontCheck/Result, in GeoDmsGui or with

GeoDmsRun.exe /L<log file> cfg\main.dms /Output/FrontCheck/File /Output/FrontCheck/Result

FrontCheck reads the parquet files per origin block of both outputs (OutputFormat 'parquet'). For 10 blocks and four departure moments it takes a few seconds.

4. Read %LocalDataProjDir%/Validatie/frontcheck_<Tag>.csv, with one line per origin block and departure moment. Result is PASS when no option of either output is dominated, and FAIL otherwise.

The columns

Column	Meaning
case	The origin block and the departure moment, such as B1_07h00m.
rows, pairs, rows_ref, pairs_ref	The options and the OD pairs of both outputs.
dominated, dominated_ref	Options that another option of the same pair dominates exactly on price and travel time. A front of the model holds none, so both must be 0. A dominated option points to an error in the condensation of the front or in the preselection of the output step.
pairs_only_new, pairs_only_ref	Pairs that only one of the two outputs reaches.
faster, slower	Pairs whose shortest travel time changed by more than TimeMargin.
cheaper, dearer, mean_dprice_euro	Pairs whose lowest price changed by more than PriceMargin, and the mean change of the lowest price in euro.
dearer_at_fastest	Pairs whose price at the shortest travel time rose by more than PriceMargin.

Against a run without the price criterion

With the price criterion in MinimiseCriteria, the front of a pair must still hold its fastest option, at no higher price. This holds apart from the relative epsilon ChainParetoTimeEpsilon: 1%, which stays under a minute up to a travel time of 100 minutes. Against a reference that differs only in leaving out the price, slower and dearer_at_fastest must therefore be 0. The price at the shortest travel time may be lower than in the reference: among equally fast options, a run without the price criterion keeps any one price.

The reference leaves out only the price: MinimiseCriteria 'Time' against the default 'Price,Time'. Car and bicycle ownership play no part here. All options of one component need the same: a public transport component holds only public transport journeys, which need no car, and its pre- and post-transport types decide for all its options whether they need an own bicycle (Classifications/Components/NeedsBike). Ownership counts in the merge (see Car and bicycle ownership as criteria).

Such a reference is a second run of the same commit with MinimiseCriteria 'Time'. MinimiseCriteria is in the signature of the output, so the output of the reference gets a folder of its own, with another hash under the same OutputLabel. It is in the signature of the chain cache too, so the reference makes a chain cache of its own. With MinimiseCriteria set to 'Time', /Output/FrontCheck/Dir shows the folder of the reference. Give that folder as RefDir after setting MinimiseCriteria back to 'Price,Time'.

Results

A FAIL in Result gives the number of dominated options over all compared files. frontcheck_<Tag>.csv shows the details, one line per origin block and departure moment (case). Read it as follows:

• dominated and dominated_ref must be 0 in every line; a line with a dominated option shows the block and the departure moment to look into (see The columns).
• rows divided by pairs is the mean number of options per OD pair, the size of the front; rows_ref and pairs_ref give the same for the reference.
• The other columns compare Dir with RefDir. pairs_only_new and pairs_only_ref count the pairs that only Dir or only RefDir reaches. faster and slower count, among the pairs that both reach, those whose shortest travel time is shorter or longer in Dir than in RefDir by more than TimeMargin. cheaper and dearer do the same for the lowest price, with PriceMargin, and dearer_at_fastest counts the pairs whose price at the shortest travel time is higher in Dir by more than PriceMargin. mean_dprice_euro is the mean change of the lowest price, negative when Dir is cheaper.
• Against an earlier run or delivery these columns show what a change of the model or its data did to reachability, travel time and price.
• Against a run without the price criterion, slower and dearer_at_fastest must be 0, as described above. cheaper and mean_dprice_euro then show how much cheaper the front with the price criterion is.