Refactor OTP to have new OTP classes to replace the OBA GTFS classes

[t2gran]

Goal

Remove the coupling from OTP's internal model and OTP's API to Conveyal´s fork of OneBusAway (OBA) GTFS classes. The OBA GTFS library should still be used for importing GTFS files. This change will make it easier in the feature to:

• Improve the internal model and extend it with new functionality. New concepts can be added by adding properties or relations to the existing classes, avoiding the decoration that is required today.
• Separate concerns, the GTFS model is made for data transfer while the internal OTP model have other concerns (finding the optimal route as fast as possible).

We want to do this with as few changes to the rest of the code as possible.

Proposed solution

1. Copy the remaining GTFS classes that are being used internally by OTP classes - into OTP.
2. Refactor OTP to use the copied classes (except for classes used by GTFS parsing)
3. Map GTFS classes to the new (copied) classes in the internal model.
4. Refactor the copied classes, remove CSV serialization and other dead code.
5. Write unit tests for the new mapping code.

Id generation is done today by the GTFS lib. For now we will also do id generation inside the "new" Dao. For consistency this should be done in the OTP import, not in the data transfer classes; We have kept the id generation in the "new" classes in the POC below. This ensure id generation for any new data format import.

Proof of concept

This is no longer a POC, but part of our NETEX solution as Entur.
See PR-2495
More on the design is described in the developer discussion group.

Java Packages

The GFTS classes are copied into OTP. Some classes are renamed to better describe their role, and put in the following packages:

• Model classes: Agency, Stop, Trip, Route ..
  Moved to package: org.opentripplanner.model

• Service interfaces: OtpTransitService(GtfsDao), calendar.CalendarService) Moved to package: org.opentripplanner.model`
  These two services play a role together with the model classes; their implementation on the other hand is hidden away in their own packages.

• Service implementation (GtfsDaoImpl, exceptions and new builder)
  Moved to package: org.opentripplanner.model.impl

• calendar.CalendarServiceImpl (move together with Factory and MultiCalendarServiceImpl)
  Moved to package: org.opentripplanner.calendar.impl

• Mapping classes (AgencyMapper, StopMapper, TripMapper, RouteMapper)
  New package: org.opentripplanner.gtfs.mapping

• MockGtfs. (Used for testing, read in GTFS CSV files and map to internal model).
  New package: org.opentripplanner.gtfs

• TimeToStringConverter and InvalidTimeException. Used in org.otp.updater and model classes.
  New package: org.opentripplanner.util

Renaming

The following names would be misleading after a refactoring, these should changed:

• Rename GtfsDaoto OtpTransitService. This apply to the Ìmpl` as well.

Dao Inheritance structure

The GTFS lib have a very complex inheritance structure without any polymorphy, this can easily be simplified and flattened into just one interface OtpTransitService (replace GtfsDao).

Version of OTP used

1.3-SNAPSHOT (dd48797) 2017-09-12

[t2gran]

We now consider this work complete and we have implemented the NETEX import with several NETEX specific features on top of this PR - we are currently doing the system test before the final release.

[skinkie]

So, if I would provide a Dutch NeTEx file as we speak, would that work?

[t2gran]

We have implemented support for the Norwegian profile of the NeTEx profile only. But, we have designed it with support for other ´profiles´ in mind, so there is two possible solutions:

• We can extend our implementation to also support your needs, or
• We can write a new Netex Module to import your needs.

The good news is that the import is now separated from the internal model, and writing a new import is just a matter of mapping. Functionality like the PatternHopFactory is generfied to support both GTFS and out new NeTEx import. Among the work we have done in the NeTEx import is to extract TripPattern generation out of the PatternHopFactory, so this step is only run in the GTFSModule.

We have not made a PR of the NeTEx work jet, we await the PR #2495 to be accepted. I will write a little about our thought and rodmap in a new post on the forum later(this week).

[abyrd]

Hi @t2gran, I appreciate all your work on this and have been working through it and the accompanying email chains. I hope to provide some responses today or tomorrow.

[t2gran]

@abyrd thank you, I look forward for your response.

[t2gran]

The PR is merged and every thing above is resolved so I will close this issue.