Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add validation of NeTEX timetabled passing times #5081

Conversation

vpaturet
Copy link
Contributor

@vpaturet vpaturet commented Apr 28, 2023

Summary

As detailed in #5060, OTP fails to filter NeTEx ServiceJourneys with incorrect passing times: validation errors are reported in the HTML validation report, but incorrect data are not removed from the graph.

In the current NeTEx import implementation, the validation, performed in ValidateAndInterpolateStopTimesForEachTrip.run(), occurs too late in the import workflow, after the ServiceJourneys have been mapped to TripTimes.
In the GTFS import case, ValidateAndInterpolateStopTimesForEachTrip is called early enough and invalid data are properly filtered out.

The design solution followed in this PR is to leverage the NeTEx validator framework (org.opentripplanner.netex.validation.Validator) and implement a custom validation rule for NeTEx passing times.
The benefits of this approach over using ValidateAndInterpolateStopTimesForEachTrip are:

  • it is in line with the error handling approach used throughout the NeTEx import, which is to filter out and report broken data without trying to fix it.
  • it takes into account the specificities of the NeTEx format, in particular the relation between ServiceJourney and JourneyPattern, and the data structures in NeTEx-flex.
  • it provides more specific error messages that makes debugging/analysis easier.

The validation rule covers all 4 combinations of stop sequences:

  • regular stop followed by regular stop
  • area stop followed by area stop
  • regular stop followed by area stop
  • area stop followed by regular stop

The call to ValidateAndInterpolateStopTimesForEachTrip.run() is kept unchanged for now, as it provides also side-effect-free validation for hop speed and hop time.
A refactoring can be performed in a subsequent PR for splitting ValidateAndInterpolateStopTimesForEachTrip in two classes: one class, used only in the GTFS import, that would fix/interpolate passing times, and another that would report suspicious hop time/hop distance.

Tested on the all-Norway and all-Sweden NeTEx datasets.

Issue

closes #5060

Unit tests

Added unit tests

@codecov
Copy link

codecov bot commented Apr 28, 2023

Codecov Report

Patch coverage: 86.84% and project coverage change: +0.27 🎉

Comparison is base (b9f4016) 64.57% compared to head (087f866) 64.85%.

Additional details and impacted files
@@              Coverage Diff              @@
##             dev-2.x    #5081      +/-   ##
=============================================
+ Coverage      64.57%   64.85%   +0.27%     
- Complexity     13889    14231     +342     
=============================================
  Files           1715     1740      +25     
  Lines          67112    67644     +532     
  Branches        7207     7239      +32     
=============================================
+ Hits           43338    43868     +530     
+ Misses         21348    21339       -9     
- Partials        2426     2437      +11     
Impacted Files Coverage Δ
...opentripplanner/netex/mapping/StopTimesMapper.java 67.77% <ø> (ø)
...er/netex/support/stoptime/AreaStopTimeAdaptor.java 60.00% <60.00%> (ø)
...netex/index/hierarchy/AbstractHierarchicalMap.java 87.87% <66.66%> (-12.13%) ⬇️
...netex/support/stoptime/RegularStopTimeAdaptor.java 80.00% <80.00%> (ø)
...dation/ServiceJourneyNonIncreasingPassingTime.java 82.85% <82.85%> (ø)
...etex/support/stoptime/AbstractStopTimeAdaptor.java 86.95% <86.95%> (ø)
...ripplanner/netex/support/ServiceJourneyHelper.java 100.00% <100.00%> (ø)
...ntripplanner/netex/support/ServiceJourneyInfo.java 100.00% <100.00%> (ø)
...lanner/netex/support/stoptime/StopTimeAdaptor.java 100.00% <100.00%> (ø)
...r/netex/validation/AbstractHMapValidationRule.java 100.00% <100.00%> (ø)
... and 1 more

... and 169 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@vpaturet vpaturet marked this pull request as ready for review April 28, 2023 12:11
@vpaturet vpaturet requested a review from a team as a code owner April 28, 2023 12:11
@t2gran t2gran added this to the 2.4 milestone May 2, 2023
TimetabledPassingTime currentTimetabledPassingTime
) {
int currentArrivalOrDepartureTime = normalizedArrivalTimeOrElseDepartureTime(
currentTimetabledPassingTime
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would I make sense to create a wrapper class for for TimetabledPassingTime, which would be similar to CallWrapper for Siri or indeed ServiceJourneyInfo?

That means you can move these helper method in there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you do, please use the Adaptor design pattern and naming convention. CallAdaptor not CallWrapper,
Ref: https://refactoring.guru/design-patterns/adapter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about calling it Adapter, but this is not strictly an implementation of the design pattern (no interface involved, just composition)

@leonardehrenfried leonardehrenfried added the NeTEx This issue is related to the Netex model/import. label May 4, 2023
Copy link
Member

@leonardehrenfried leonardehrenfried left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the architecture but I don't know a lot about the domain we are validating here.

@t2gran t2gran added the Entur test This is currently being tested at Entur label May 8, 2023
@vpaturet
Copy link
Contributor Author

@leonardehrenfried could you please review again? There is no need to rebase, this PR contains both commits from Thomas and me (which makes the reviewing process more complicated actually).
I reviewed the commits from Thomas and I am OK with them.

@vpaturet vpaturet merged commit e6a09cf into opentripplanner:dev-2.x May 25, 2023
5 checks passed
@t2gran t2gran deleted the otp2_validate_netex_increasing_passing_time branch May 25, 2023 08:52
t2gran pushed a commit that referenced this pull request May 25, 2023
EmmaSimon added a commit to mbta/OpenTripPlanner that referenced this pull request Jun 22, 2023
* Add changelog entry for opentripplanner#5100 [ci skip]

* refactor: Add proper progress tracking for GraphQL timeouts.

* refactor: Fix spelling 'guarantied' -> 'guaranteed'

* review: Fix documentation

* feature: Trace HTTP request headers

* Add changelog entry for opentripplanner#5081 [ci skip]

* Add changelog entry for opentripplanner#5091 [ci skip]

* Add changelog entry for opentripplanner#5133 [ci skip]

* Remove unused code

* Generate full POM for shaded jar [ci skip]

* Add elevation data to Transmodel API

* Finetune elevation format

* Update documentation

* Add changelog entry for opentripplanner#5142 [ci skip]

* fix: Make sure the log key is removed from the Grizzly thread log context

* Apply suggestions from code review

Co-authored-by: Thomas Gran <t2gran@gmail.com>

* Remove San Francisco fare calculator

* Remove TimeBasedVehicleRentalFareService

* Add documentation about deleted calculators

* Remove empty lines

* Refactor null check on from and to vertices

* review: Apply review feedback

* Apply suggestions from code review

Co-authored-by: Leonard Ehrenfried <mail@leonard.io>

* feature: Add sanity check for HTTP header value

The value must match `[^\p{Cntrl}\v]{1,512}`

* test: Add test regular expression for HTTP header value

* Apply suggestions from code review

Co-authored-by: Leonard Ehrenfried <mail@leonard.io>

* feature: Remove batch query from Transmodel API

* refactor: re-generate doc

* fix(deps): update dependency com.google.guava:guava to v32

* fix(deps): update dependency com.graphql-java:graphql-java to v20.3

* Add changelog entry for opentripplanner#5130 [ci skip]

* fix(deps): update dependency com.google.guava:guava to v32

* Use git to figure out last-modified date

* Debug last modified check

* Increase fetch depth

* Make update frequency a Duration

* fix(deps): update dependency com.fasterxml.jackson.core:jackson-annotations to v2.15.2

* Update log messages

* Update docs

* Remove link to 1.x-dev docs

* refactor: Avoid creating builders unnecessary.

* test: Add test for bug in TripRequestMapperTest

* Add changelog entry for opentripplanner#5131 [ci skip]

* Bump serialization version id for opentripplanner#5131

* Add changelog entry for opentripplanner#5141 [ci skip]

* Bump serialization version id for opentripplanner#5141

* Improve money documentation

* Implement stop sequence in GraphQL

* Add stop sequence and test

* Fix docs, formatting and tests

* Add test for walk step mapping

* fix: Fix bug in maxDirectDurationForMode and refactor

 - The code is not DRY, so I refactored it - this also fixes the bug

* refactor: Extract mappers out off PreferencesMapper

* fix(deps): update dependency com.google.cloud:libraries-bom to v26.15.0

* fix(deps): update dependency org.onebusaway:onebusaway-gtfs to v1.4.3

* Make absolute direction optional

* Add test for GraphQL API

* Add changelog entry for opentripplanner#5140 [ci skip]

* Update documentation

* Add changelog entry for opentripplanner#5145 [ci skip]

* Consider level and layer tags when linking public transit stop area nodes

E.g. transit entrances above ground should not get directly linked to subway platforms

* Add test for ensuring that entrances do not link to platforms across different layers/levels

* Update documentation and mapping

* test: Add regression test.

* fix: Fix validation of flex area, assert isComplete and isConsistent, before isStopTimesIncreasing

* refactor: Cleanup AbstractStopTimeAdaptor

* Improve documentation

* Handle stop areas with many platforms properly

Entrances and other unconnected nodes included in stop area relation were
linked only with the first platform area. Now they are matched with all platforms.
Walkable area builder won't create connections if node is outside a platform.

* Update documenation

* Create interline transfers for trips that share the same service date and block

* Connect area boundary to entrance points inside it to prevent pruning

* Add better test data for testing area processing of stop_area relations

* Validate to/from in routing request

* Add changelog entry for opentripplanner#5152 [ci skip]

* Bump serialization version id for opentripplanner#5152

* Stop area linking tests

* Changing default value for earlyStartSec

* Add some documetation about stop area relations

* Fix formatting

* Add support for mapping NeTEx operating day in operating period

* Add error mapping in REST API

* Update documentation

* Add new doc page to mkdocs.yml

* Move getLevel to OSMWithTags andd simplify it

* Test also layer tag relevance in stop area processing

* Fix misleading data report issue content from stop area processing

* Use multimap for storing stop area link nodes

* Remove obsolete issue

Stop area which does not have entrances or other link points is really not any kind of error

* Add changelog entry for opentripplanner#5147 [ci skip]

* Improve updater log messages

* refactor: Apply code review

 - Change `earlyStartSec:int` to `earlyStart:Duration`
 - Add more doc on parameter

* Relax validity check for flex trip with null duration

* Make FlexPath fields final

* refactor: Make `SiriSXUpdaterParameters#timeout` a Duration

* Apply suggestions from code review

* refactor: Make Siri Updaters use Duration, not int, for reminding parameters

 - This also remove a bit of unnecessary mapping code.

* Update src/main/java/org/opentripplanner/standalone/config/routerconfig/updaters/SiriSXUpdaterConfig.java

Co-authored-by: Leonard Ehrenfried <mail@leonard.io>

* chore(deps): update dependency org.apache.maven.plugins:maven-surefire-plugin to v3.1.2

* Refactor shutdown hook

* Add changelog entry for opentripplanner#5159 [ci skip]

* Update pull_request_template.md [ci skip]

* fix(deps): update dependency com.graphql-java:graphql-java to v20.4

* Use hashmultimap in  src/main/java/org/opentripplanner/graph_builder/module/osm/OsmDatabase.java>

Co-authored-by: Leonard Ehrenfried <mail@leonard.io>

* Add HashMultimap import

* Refactor first/last date getters

* Split method in two

* Update src/main/java/org/opentripplanner/graph_builder/module/interlining/InterlineProcessor.java

Co-authored-by: Thomas Gran <t2gran@gmail.com>

* More accurate tagging instructions, one example relation linked

* Log warning if GBFS status reports unexpected vehicle type

* Code cleanup

* refactor: Cleanup TransitRouter and AccessEgressMapper

* refactor: Sort values in DurationForEnum.toString to make it deterministic

* refactor: Add State.containsModeWalkOnly() and DefaultAccessEgress.isWalkOnly()

These methods will make it simpler to filter access/egress later

* refactor: Add Duration#requireNonNegative(Duration) : Duration to DurationUtils

* refactor: Implement openingHoursToString() for AccessEgress for testing

Having to brows through many classes and hairy logic is time-consuming when
debugging FLEX access/egress, this simplifies the process.

* refactor: Move RaptorConstants into raptor.api.model package

* feature: Search before the earliest-departure-time in Raptor with searchWindowAccessSlack.

* refactor: Move AccessEgresses to street package

* refactor: Cleanup DoubleUtils

* refactor: Add requireXyz to IntUtils

* refactor: Add Cost value-object

* refactor: Improve int and double utilities

* refactor: Small cleanups

* Apply suggestions from code review

Co-authored-by: Thomas Gran <t2gran@gmail.com>

* Formatting

* Add changelog entry for opentripplanner#5161 [ci skip]

* Add changelog entry for opentripplanner#5162 [ci skip]

* Add changelog entry for opentripplanner#5168 [ci skip]

* Apply review suggestions

* Apply review suggestions

* Fix bicyle optimise type in TransmodelApi

* Add test for optimize type in GraphQL API

* fix(deps): update dependency com.google.guava:guava to v32.0.1-jre

* Add changelog entry for opentripplanner#5167 [ci skip]

* Add reusable method

* Add union type for stop position

* Simplify type resolving

* Use interface type in data fetcher

* Applied review suggestion

* Add documentation

* Add documentation

* Apply review feedback

Co-authored-by: Joel Lappalainen <lappalj8@gmail.com>

* Generate new SiriUpdater doc

* Add changelog entry for opentripplanner#5169 [ci skip]

* Add changelog entry for opentripplanner#5175 [ci skip]

* Add changelog entry for opentripplanner#5164 [ci skip]

* Add changelog entry for opentripplanner#5165 [ci skip]

* review: Remove serialVersionUID

* Validate that from and to temporary vertices are distinct

* review: Extract TestVehicleRentalStationBuilder

* Reduce log severity for non-optimized transfers

* Return an int as the stopPosition for StopTimes

* doc: Move JavaDoc to accessors, fix typo.

* fix: Improve error handling and prevent OTP from going down when connecting to external http services.

The VehicleRentalServiceDirectoryFetcher went down with a IllegalSateException when the
http endpoint failed. Instead of returning null in some error-cases and throwing IOExceptions
in others the HttpUtils is changed to throw an IOException in all cases. This make it more
robust, and the checked exception forces the client to handle it.

* Use Finland OSM mapping as basis for constant speed mapper

* Add support for taxi mode

* Rename ConstansSpeedMapper to describe the new super class

* Add changelog entry for opentripplanner#5153 [ci skip]

* Update micrometer.version to v1.11.1

* Update src/main/java/org/opentripplanner/routing/algorithm/mapping/RaptorPathToItineraryMapper.java

Co-authored-by: Thomas Gran <t2gran@gmail.com>

* Apply review feedback

* Update dependency ch.qos.logback:logback-classic to v1.4.8

* Add changelog entry for opentripplanner#5135 [ci skip]

* Separate words with underscore in osm tag mapper enum value

* Update docs

* Bump serialization version id for opentripplanner#5176

* Make stop area linking more precise and capable to handle elevators

* Add changelog entry for opentripplanner#5181 [ci skip]

* Use EnumSet instead of Stream

* Add changelog entry for opentripplanner#5183 [ci skip]

* Add changelog entry for opentripplanner#5179 [ci skip]

* Fix formatting in RouterConfig.md

* improve: add language argument to Quay and StopPlace types

deprecate lang arguments

* fix: the overloaded getLocale method does not use the language argument

* improve: extract shared code into helper method in transmodel GqlUtil

* add test for GqlUtil.getLocale

* Fix default value for bicycle safety report

* Update dependency org.apache.maven.plugins:maven-shade-plugin to v3.5.0

* Update dependency net.logstash.logback:logstash-logback-encoder to v7.4

* Update dependency org.mockito:mockito-core to v5.4.0

* Add more tests for stop area linking
- Test that elevators get linked
- Test that stop positions will not get linked

* Fix typo

* Update src/main/java/org/opentripplanner/openstreetmap/model/OSMWithTags.java

Co-authored-by: Leonard Ehrenfried <mail@leonard.io>

* Return set

* Update src/main/java/org/opentripplanner/openstreetmap/model/OSMWithTags.java

Co-authored-by: Joel Lappalainen <lappalj8@gmail.com>

* Update src/main/java/org/opentripplanner/openstreetmap/model/OSMWithTags.java

Co-authored-by: Joel Lappalainen <lappalj8@gmail.com>

* Add changelog entry for opentripplanner#5166 [ci skip]

---------

Co-authored-by: Leonard Ehrenfried <mail@leonard.io>
Co-authored-by: OTP Changelog Bot <changelog-bot@opentripplanner.org>
Co-authored-by: Thomas Gran <t2gran@gmail.com>
Co-authored-by: vpaturet <46598384+vpaturet@users.noreply.github.com>
Co-authored-by: Vincent Paturet <vincent.paturet@entur.org>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: OTP Serialization Version Bot <serialization-version-bot@opentripplanner.org>
Co-authored-by: Vesa Meskanen <vesa.meskanen@cgi.com>
Co-authored-by: Joel Lappalainen <lappalj8@gmail.com>
Co-authored-by: Lasse Tyrihjell <lassetyr@gmail.com>
Co-authored-by: Vesa Meskanen <vesa@realsoft.com>
Co-authored-by: Tom Erik Støwer <testower@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Entur test This is currently being tested at Entur NeTEx This issue is related to the Netex model/import.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Trips with wrong chronology not filtered during NeTEx import
3 participants