Stations jump over each other

[pietervdvn]

Seems like in some cases the order of stations is incorrect.
See the table below, generated with

dotnet run --ctlc stops=https://graph.irail.be/sncb/stops connections=https://graph.irail.be/sncb/connections today 1hour --validate jump --filter-trip http://irail.be/vehicle/IC723/20191001 --validate jump --dump-connections trip.csv

GlobalId	DepartureStop	DepartureStopName	ArrivalStop	ArrivalStopName	DepartureTime	DepartureDelay	ArrivalTime	ArrivalDelay	TravelTime	Mode	TripId	TripHeadSign
http://irail.be/connections/8821006/20191001/IC723	http://irail.be/stations/NMBS/008821006	Anvers-Central	http://irail.be/stations/NMBS/008821121	Anvers-Berchem	2019-10-01T19:37:00.0000000Z	0	2019-10-01T19:41:00.0000000Z	0	240	0	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8821121/20191001/IC723	http://irail.be/stations/NMBS/008821121	Anvers-Berchem	http://irail.be/stations/NMBS/008821196	Anvers-Sud	2019-10-01T19:44:00.0000000Z	60	2019-10-01T19:48:00.0000000Z	60	240	1	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8821196/20191001/IC723	http://irail.be/stations/NMBS/008821196	Anvers-Sud	http://irail.be/stations/NMBS/008894821	Zwijndrecht	2019-10-01T19:48:00.0000000Z	60	2019-10-01T19:50:00.0000000Z	0	120	3	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8894821/20191001/IC723	http://irail.be/stations/NMBS/008894821	Zwijndrecht	http://irail.be/stations/NMBS/008894755	Melsele	2019-10-01T19:50:00.0000000Z	0	2019-10-01T19:52:00.0000000Z	0	120	3	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8894755/20191001/IC723	http://irail.be/stations/NMBS/008894755	Melsele	http://irail.be/stations/NMBS/008894748	Beveren	2019-10-01T19:52:00.0000000Z	0	2019-10-01T19:53:00.0000000Z	0	60	3	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8894748/20191001/IC723	http://irail.be/stations/NMBS/008894748	Beveren	http://irail.be/stations/NMBS/008894714	Nieuwkerken-Waas	2019-10-01T19:53:00.0000000Z	0	2019-10-01T19:55:00.0000000Z	0	120	3	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8894714/20191001/IC723	http://irail.be/stations/NMBS/008894714	Nieuwkerken-Waas	http://irail.be/stations/NMBS/008894508	Saint-Nicolas	2019-10-01T19:55:00.0000000Z	0	2019-10-01T20:00:00.0000000Z	0	300	2	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8894508/20191001/IC723	http://irail.be/stations/NMBS/008894508	Saint-Nicolas	http://irail.be/stations/NMBS/008894433	Belsele	2019-10-01T20:03:00.0000000Z	60	2019-10-01T20:06:00.0000000Z	60	180	1	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8894425/20191001/IC723	http://irail.be/stations/NMBS/008894425	Sinaai	http://irail.be/stations/NMBS/008894201	Lokeren	2019-10-01T20:06:00.0000000Z	60	2019-10-01T20:10:00.0000000Z	0	240	2	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8894433/20191001/IC723	http://irail.be/stations/NMBS/008894433	Belsele	http://irail.be/stations/NMBS/008894425	Sinaai	2019-10-01T20:06:00.0000000Z	60	2019-10-01T20:08:00.0000000Z	60	120	3	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8894201/20191001/IC723	http://irail.be/stations/NMBS/008894201	Lokeren	http://irail.be/stations/NMBS/008894151	Beervelde	2019-10-01T20:12:00.0000000Z	0	2019-10-01T20:17:00.0000000Z	0	300	1	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8894151/20191001/IC723	http://irail.be/stations/NMBS/008894151	Beervelde	http://irail.be/stations/NMBS/008893120	Gand-Dampoort	2019-10-01T20:17:00.0000000Z	0	2019-10-01T20:25:00.0000000Z	60	480	2	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8893120/20191001/IC723	http://irail.be/stations/NMBS/008893120	Gand-Dampoort	http://irail.be/stations/NMBS/008893179	Gentbrugge	2019-10-01T20:27:00.0000000Z	120	2019-10-01T20:29:00.0000000Z	120	120	1	http://irail.be/vehicle/IC723/20191001
http://irail.be/connections/8893179/20191001/IC723	http://irail.be/stations/NMBS/008893179	Gentbrugge	http://irail.be/stations/NMBS/008892007	Gand-Saint-Pierre	2019-10-01T20:29:00.0000000Z	120	2019-10-01T20:35:00.0000000Z	120	360	2	http://irail.be/vehicle/IC723/20191001

[pietervdvn]

It turns out that this is a combination of factors

• If departure time == arrival time, the departure delay was applied as extra while loading the data, in order to 'fix' this faulty data. This is now removed and 'timetravelling'-trains can now happen in the dataset
• Some trains/buses have a schedule where they depart and leave at the same moment. The above (attempt to) correct things made this worse, but these should still be scheduled in order

[pietervdvn]

The core issue has been identified.

The workflow used to be:

1. Load the connections from LC, where they were in a correct order
2. Sort the timewindow in the ConnectionsDB. This sorting was not stable** and removed this ordering, causing these jumps
3. Use the connectionsDb.

I've removed the QuickSort and replaced it by a laughably* simple other algorithm - which might even perform better given that the lists often will be sorted already. (*the original algorithm might even have started as a joke)

Note however that, although it works for now, I'd like to have more robust handling of this case in the future - thus: our database should do this sorting itself.

The assumption is:

If a connection C0 has a traveltime of zero, there will be a next connection C1 with the same departure time from the arrivalstop of C1.

If such a condition is met, C0 should be ordered in the list before C1. This should be handled by 'SortWindow' as well. However, I would first like to refactor all those DB's as they are way to complicated right now for what they are doing.

[pietervdvn]

This is also a partly upstream issue, where connections are teleporting