Add support for GTFS-RT platform change, headsign change, NEW and REPLACEMENT trips

[miklcct]

Summary

This PR implements the features added in the recently accepted proposal google/transit#504 to formalize the ability to add new trips or replace trips in GTFS-RT. It also implements an earlier proposal google/transit#219 to add assigned_stop_id as well.

Issue

Closes #5230
Closes #6009

Unit tests

Unit tests are added for TripUpdates which specify both time and delay in a StopTimeEvent. This case was previously assumed to be invalid and hence a new method is added to generate items. Tests are also added for NEW trips containing skipped stops as well.

Documentation

The original // TODO: should we incorporate the delay field if present? is done by this PR.

[codecov]

Codecov Report

❌ Patch coverage is 78.79464% with 95 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.83%. Comparing base (3bae1a1) to head (f38ede7).
⚠️ Report is 133 commits behind head on dev-2.x.

Files with missing lines	Patch %	Lines
...dater/trip/gtfs/GtfsRealTimeTripUpdateAdapter.java	75.46%	30 Missing and 10 partials ⚠️
...ripplanner/updater/trip/gtfs/TripTimesUpdater.java	71.79%	23 Missing and 10 partials ⚠️
...lanner/updater/trip/gtfs/model/StopTimeUpdate.java	89.21%	4 Missing and 7 partials ⚠️
...lanner/updater/trip/gtfs/model/TripDescriptor.java	70.58%	4 Missing and 1 partial ⚠️
...ripplanner/updater/trip/gtfs/model/TripUpdate.java	83.33%	0 Missing and 3 partials ⚠️
...pentripplanner/transit/model/framework/Result.java	33.33%	1 Missing and 1 partial ⚠️
...ripplanner/updater/trip/gtfs/model/AddedRoute.java	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             dev-2.x    #6028      +/-   ##
=============================================
+ Coverage      71.76%   71.83%   +0.06%     
- Complexity     19330    19417      +87     
=============================================
  Files           2090     2094       +4     
  Lines          78771    78872     +101     
  Branches        8027     8016      -11     
=============================================
+ Hits           56529    56655     +126     
+ Misses         19384    19369      -15     
+ Partials        2858     2848      -10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[optionsome]

ADDED and REPLACEMENT updates don't have many users in the OTP community (I think REPLACEMENT is slowly being replaced by new additions to the GTFS specifications, but I'm not fully up-to-date on the status of those drafts), but we are open to these implementations being fixed and/or improved.

I doubt this will break any existing OTP deployments as I think your interpretation of how delay should be used is the only reasonable one. Similarly, I think previously someone might have just left out a stop in an ADDED trip if it was no longer used but having this possibility to mark them as SKIPPED shouldn't negatively affect anyone. If someone is opposed to these changes, let us know.

The only small question mark is that should we wait for some comments from the GTFS community on your suggestion before this is merged or not. Hopefully, the suggestion to the GTFS specification goes through so there is no longer any gray area around these usages.

[miklcct]

REPLACEMENT is officially deprecated, and the behaviour of ADDED remains unspecified. The proposal addresses DUPLICATED (which has a clearly defined behaviour) as well as ADDED.

Google uses ADDED to duplicate an existing trip to run it at another time, which is now officially migrated to DUPLICATED in the standard, while OpenTripPlanner requires specifying the whole trip and allows arbitrary pattern for ADDED trips, which fits the use case to add trips which are not based on existing patterns (which Google doesn't support).

To my understanding there isn't a way in the official GTFS-RT standard to add arbitrary new realtime trips into a route not based on an existing pattern (even Google doesn't support that). Even in the OpenTripPlanner implementation it is impossible to specify headsigns of added trips (both at trip and stop level) but this will be another issue to solve.

GTFS-TripModification proposal can be used for diverted trips but it isn't supported in OpenTripPlanner yet and it is much more complicated to just replacing the whole trip in case of diversion using the deprecated REPLACEMENT feature of OpenTripPlanner.

I am thinking that we should wait for a while to see how the GTFS community progresses on trip additions / diversions / replacement, but I will have a production deployment in the next few months using this change in order to support real time added / diverted trains, which few other commercial providers support (neither Google nor Citymapper could route me via the official diversion to Battersea Park when the line to Clapham Junction was blocked unexpectedly).

Could you please let me know if there are any existing OpenTripPlanner deployments which do handle real-time diverted trips?

[miklcct]

Possibly need further testing for REPLACEMENT trips:

[leonardehrenfried]

We talked about this in today's dev meeting (to which you are welcome). We are generally ok with this interpreation of the spec but would like to see it codified in the spec.

Are you interested in un-deprecating the ADDED schedule relationship in the GTFS spec repo? google/transit#490 is a good start but can you create an actual PR which can be discussed and critiqued line by line by the community? To me the most important change I would like to see is an actually description of what ADDED means. The OTP implementation thinks that it allows you to create completely new trips but I would like to clarify.
If you're okay with trying the spec change I would support you with it.

Since the realtime code is old and has incurred lots of technical debt, we would also like to see some clean up. Are you interested in that?

[miklcct]

I will be interested in cleaning up the code if it means a chance for me to get headsigns, shapes, completely new stops, etc., into the realtime updates but I would like to see how the GTFS-ServiceChanges v3.1 goes first because we commit to anything.

A major limitation of the product we are developing is the inability to specify the headsign for modified trips under the current specification, so I am relying on frontend tweaks to "correct" the headsign for trains terminating short.

[leonardehrenfried]

I also found a thread on Slack about ADDED: https://mobilitydata-io.slack.com/archives/C3D321CKB/p1715957220876199

I think there will be interest in specifying it.

[leonardehrenfried]

Before I review any further: are you going to convert google/transit#490 into a spec PR?

[miklcct]

I plan to do so but not in these few days.

[leonardehrenfried]

Please format the code.

[leonardehrenfried]

@optionsome will be back when I go on holiday, so you can merge without a re-approval from me.