Create a valid area even when it has too many vertices #5019

vesameskanen · 2023-04-01T08:10:58Z

Summary

Walkable area processing does not try to generate walking routes across the area, if the geometry is too complex. The complexity limit check terminated area processing so early that area data which is used in linking vertices inside the area was totally missing.

Now area processing initialises complex areas propely. Slow edge generation is mitigated by considering only maxAreaNodes subset from the visibility nodes.

Area complexity check terminated area processin too early, leaving the area data invalid for linking.

codecov · 2023-04-01T08:20:49Z

Codecov Report

Patch coverage: 90.19% and project coverage change: +0.11 🎉

Comparison is base (d7f43ca) 64.49% compared to head (216f6fc) 64.60%.

Additional details and impacted files

@@              Coverage Diff              @@
##             dev-2.x    #5019      +/-   ##
=============================================
+ Coverage      64.49%   64.60%   +0.11%     
- Complexity     13863    13890      +27     
=============================================
  Files           1710     1710              
  Lines          67017    67054      +37     
  Branches        7203     7206       +3     
=============================================
+ Hits           43221    43322     +101     
+ Misses         21376    21312      -64     
  Partials        2420     2420

Impacted Files	Coverage Δ
.../graph_builder/module/osm/OpenStreetMapModule.java	`90.92% <85.71%> (+0.53%)`	⬆️
.../graph_builder/module/osm/WalkableAreaBuilder.java	`89.53% <100.00%> (+8.33%)`	⬆️
...opentripplanner/standalone/config/BuildConfig.java	`98.46% <100.00%> (ø)`

... and 13 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

leonardehrenfried

This is good work, thanks @vesameskanen .

However since this code breaks a lot I think we really should be add tests for this behaviour. Personally, I would be ok if we read in real OSM data from Lund, run the area builder and assert a result. (https://docs.opentripplanner.org/en/dev-2.x/Preparing-OSM/#filtering-osm-data)

Since it's a bug that Skanetrafiken wants fixed, could we ask @Bartosz-Kruba and @jtorin to contribute the tests?

jtorin · 2023-04-03T15:21:29Z

This is good work, thanks @vesameskanen .

Indeed! Also, I have separately verified that it solves the problem we're seeing.

Since it's a bug that Skanetrafiken wants fixed, could we ask @Bartosz-Kruba and @jtorin to contribute the tests?

Lets discuss this on tomorrows dev-meeting.

leonardehrenfried · 2023-04-04T08:18:39Z

@jtorin Here is an example of a test: https://github.com/opentripplanner/OpenTripPlanner/blob/dev-2.x/src/test/java/org/opentripplanner/graph_builder/module/OsmBoardingLocationsModuleTest.java

leonardehrenfried · 2023-04-04T08:22:22Z

@jtorin Here is an example of how to extract data from a larger OSM file: https://github.com/leonardehrenfried/otp2-setup/blob/main/Makefile#L219

http://geojson.io/

And here is an example for filtering out unneeded ways: https://docs.opentripplanner.org/en/dev-2.x/Preparing-OSM/#filtering-osm-data

habrahamsson-skanetrafiken · 2023-04-05T09:39:14Z

src/main/java/org/opentripplanner/graph_builder/module/osm/WalkableAreaBuilder.java

      for (OSMNode nodeI : visibilityNodes) {
+        sum_i += skip_ratio;


Nice fix!

Just a note that summing floats like this can actually be a bit dangerous. If skip_ratio is sufficiently small (like 1e-8) the sum will never reach 1. It probably won't be issue in this case but you might want to write this logic using an integer counter instead

Right. In this case, float increment is typically greater than 0.1, and in some truly exceptional cases > 0.01. Lower values would mean that street graph is useless for routing, and it won't matter if area edges will not get added.

jtorin · 2023-04-13T06:56:54Z

@vesameskanen I'm working on a test for WalkableAreaBuilder with data covering Lund station.

Area complexity check terminated area processin too early, leaving the area data invalid for linking.

…g OpenStreetMapModule. Make Handler static, which means that it looses access to the fields in OSMModule. Solve this by a mix of passing copies of private values to a new instance, as well as funneling access to the data using Suppliers. A number of tests needs to adapt to the new extended contructor.

Uses OSM data from an area around Lund station, Sweden. Coverage as well as assertions are spotty and can be improved. With help from Leonard Ehrenfried.

…back

…lex-area-fallback

habrahamsson-skanetrafiken

Nice that you got the tests in!

Just a comment about how we might be able to clean the code up a bit. But we could also just merge this as is as far as I'm concerned.

habrahamsson-skanetrafiken · 2023-04-24T13:07:54Z

src/main/java/org/opentripplanner/graph_builder/module/osm/OpenStreetMapModule.java

  ) {
    this.providers = List.copyOf(providers);
    this.boardingAreaRefTags = boardingAreaRefTags;
    this.graph = graph;
    this.issueStore = issueStore;
    this.areaVisibility = areaVisibility;
+    this.banDiscouragedWalking = banDiscouragedWalking;
+    this.banDiscouragedBiking = banDiscouragedBiking;


Can't we add the staticParkAndRide, staticBikeParkAndRide, platformEntriesLinking, customNamer and maxAreaNodes to the constructor as well? Then we could avoid the Supplier-thing and we could make the fields private. IMO it's better to have an obscenely big constructor than exposing public mutable fields.

Unfortunately it's the other way around: the Supplier:s exists to couple the handler with the public fields in OpenStreetMapModule. I checked in each public field and it was only banDiscouragedWalking and banDiscouragedBiking that wasn't modified externally, thus made them private final (immutable) and passable to Handler.

Yes, this isn't especially pretty, but I tried to minimize the changes in the application code to avoid introducing bugs.

Since Handler is 1-to-1 instantiation-wise to the outer class OpenStreetMapModule maybe they should be merged, and Handler could be converted to an interface instead. Maybe WalkableAreaBuilder and some of the code in OpenStreetMapModule can be converted into smaller modules that are executed sequentially.

Yeah, it's true that the public fields are modified externally but from a quick glance it looked like all the modifications were made in tests that could trivially be changed to supply the values via constructor instead.

You're right that these classes would benefit from some refactoring :) But I guess that's out of scope for this PR.

I think we all agree that these public mutable fields are terrible and we should get rid of it.

But frankly I feel like I've tortured @jtorin enough so I will not insist on it.

The best solution IMO would be to replace the OSMModule constructor with a builder so that the default values area set appropriately. I might do that in the future.

leonardehrenfried · 2023-04-24T14:52:09Z

src/test/java/org/opentripplanner/graph_builder/module/osm/WalkableAreaBuilderTest.java

+import org.opentripplanner.street.model.edge.AreaEdge;
+import org.opentripplanner.transit.model.framework.Deduplicator;
+
+public class WalkableAreaBuilderTest {


This is a good test and the beginning of something new - thanks!

I've taken the liberty to remove a bit of duplication by using a parameterized test. It's not really a requirement to getting this merged as it comes down to personal preferences, but if you like it, you can also cherry-pick this commit: leonardehrenfried@6eed582

jtorin

I'm approving the actual code changes, @habrahamsson-skanetrafiken has reviewed and approved my tests (as well).

Create a valid area even when it has too many vertices

363052c

Area complexity check terminated area processin too early, leaving the area data invalid for linking.

vesameskanen requested a review from a team as a code owner April 1, 2023 08:10

vesameskanen added the bug label Apr 1, 2023

Remove hard limit on walkable area processing

6352131

leonardehrenfried reviewed Apr 3, 2023

View reviewed changes

Update docs

c8a677d

habrahamsson-skanetrafiken reviewed Apr 5, 2023

View reviewed changes

optionsome added the Digitransit test label Apr 6, 2023

vesameskanen and others added 7 commits April 20, 2023 12:05

Create a valid area even when it has too many vertices

7aa3fd6

Area complexity check terminated area processin too early, leaving the area data invalid for linking.

Remove hard limit on walkable area processing

de49bc5

Update docs

d6c71c1

An initial test for WalkableAreaBuilder.

b40354c

Uses OSM data from an area around Lund station, Sweden. Coverage as well as assertions are spotty and can be improved. With help from Leonard Ehrenfried.

Merge remote-tracking branch 'otp/dev-2.x' into too-complex-area-fall…

aecfe32

…back

Merge remote-tracking branch 'skane/jt-5019-unit-tests' into too-comp…

216f6fc

…lex-area-fallback

jtorin requested review from leonardehrenfried and habrahamsson-skanetrafiken April 24, 2023 11:52

t2gran added this to the 2.4 milestone Apr 24, 2023

habrahamsson-skanetrafiken reviewed Apr 24, 2023

View reviewed changes

leonardehrenfried reviewed Apr 24, 2023

View reviewed changes

leonardehrenfried approved these changes Apr 25, 2023

View reviewed changes

habrahamsson-skanetrafiken approved these changes Apr 25, 2023

View reviewed changes

jtorin approved these changes Apr 25, 2023

View reviewed changes

vesameskanen merged commit 3c59d1c into opentripplanner:dev-2.x Apr 25, 2023
5 checks passed

vesameskanen deleted the too-complex-area-fallback branch April 25, 2023 09:12

t2gran pushed a commit that referenced this pull request Apr 25, 2023

Add changelog entry for #5019 [ci skip]

f08aa9e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a valid area even when it has too many vertices #5019

Create a valid area even when it has too many vertices #5019

vesameskanen commented Apr 1, 2023 •

edited

codecov bot commented Apr 1, 2023 •

edited

leonardehrenfried left a comment

jtorin commented Apr 3, 2023

leonardehrenfried commented Apr 4, 2023

leonardehrenfried commented Apr 4, 2023 •

edited

habrahamsson-skanetrafiken Apr 5, 2023

vesameskanen Apr 13, 2023

jtorin commented Apr 13, 2023

habrahamsson-skanetrafiken left a comment

habrahamsson-skanetrafiken Apr 24, 2023

jtorin Apr 24, 2023

habrahamsson-skanetrafiken Apr 24, 2023

leonardehrenfried Apr 24, 2023

leonardehrenfried Apr 24, 2023

leonardehrenfried Apr 24, 2023

jtorin left a comment

Create a valid area even when it has too many vertices #5019

Create a valid area even when it has too many vertices #5019

Conversation

vesameskanen commented Apr 1, 2023 • edited

Summary

codecov bot commented Apr 1, 2023 • edited

Codecov Report

leonardehrenfried left a comment

Choose a reason for hiding this comment

jtorin commented Apr 3, 2023

leonardehrenfried commented Apr 4, 2023

leonardehrenfried commented Apr 4, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtorin commented Apr 13, 2023

habrahamsson-skanetrafiken left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtorin left a comment

Choose a reason for hiding this comment

vesameskanen commented Apr 1, 2023 •

edited

codecov bot commented Apr 1, 2023 •

edited

leonardehrenfried commented Apr 4, 2023 •

edited