New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thoughts on profile development and testing #116

Open
wetneb opened this Issue Oct 11, 2018 · 13 comments

Comments

Projects
None yet
4 participants
@wetneb
Copy link

wetneb commented Oct 11, 2018

I find it quite frustrating that #58 is not merged, but I understand the underlying issue: as a maintainer, how can you be confident that changes to your profiles will not badly screw up the results in some cases?

So, do we need a sort of test-driven profile development? Has this been done anywhere else? Here a few random thoughts of how this could work.

A test case could be an origin, destination and one or more test point(s) through which the computed route would need to pass to validate/fail the test. To ease maintenance, test cases could come with a short description of why the route is expected to pass there rather than elsewhere. Each profile would have its own test suite, but ideally it should be possible for these test suites to share common tests.

Should the tests be run against a frozen state of OSM? That would have the advantage of making the testing process more reliable and would lower the maintenance costs for the test suite. Or should they be run continuously as the map evolves? That would have the advantage of matching the real state of the map better, and potentially notifying the maintainer of any important tagging changes.

If we had a nice test suite for a few profiles, we could even try to derive profile weights from test cases themselves! Convert all the test cases to a big bunch of linear inequations, and let the solver come up with something.

I'm sure plenty of people have done similar things before (I am not familiar with the field at all) so it would be important to research what is already out there before.

@poutnikl

This comment has been minimized.

Copy link

poutnikl commented Oct 11, 2018

IMHO, the best test environment is area the profile developer/tester is familiar with, so it cannot be the common one.

It is on such a person familiar with area roads/ways to consider, what route the profile with given priorities should take and to check what it really chooses.

Also, concerning surface quality data, there is need to compare OSM data with reality, in case they do not match (well).

@wetneb

This comment has been minimized.

Copy link

wetneb commented Oct 11, 2018

Yes I agree that it helps a lot to know an area to design test cases. But I do not understand what you mean by "it cannot be the common one"? What "common one"?

@poutnikl

This comment has been minimized.

Copy link

poutnikl commented Oct 11, 2018

It was an Android typo, I have immediately corrected it as common, but you replied to the original. ( posting from an Android GitHub client.)

Like personal test cases are IMHO better than common test cases.

@wetneb

This comment has been minimized.

Copy link

wetneb commented Oct 11, 2018

Okay, I still don't understand with "common" instead of "coming", so I have also edited my post!

@poutnikl

This comment has been minimized.

Copy link

poutnikl commented Oct 11, 2018

In my understanding, you asked for some common test cases and test scenarios.

@wetneb

This comment has been minimized.

Copy link

wetneb commented Oct 11, 2018

I think a lot of basic tests can be understood without knowing the place… I think they should only be added in places where the decision can be made based on the OSM data (otherwise there is no point trying to figure out a routing profile that somehow manages to pass the test for an unrelated reason).

More specifically I am thinking of the following workflow:

  • I am using some profile to plan my trips;
  • One day I find out that one route was bad - I update the map to indicate why (steps, bad road surface, narrow corner, whatever);
  • I check that the routing profile avoids the problem with the updated map. If not, I add a test case for it, and try to find a sensible profile change to fix the issue.
  • I submit the profile update and the test as a PR (for instance) - the maintainer can review the change more confidently by checking that the change does not break existing tests, understand which issue prompted the change, and so on. If the maintainer does not agree with the change (we don't have the same taste, we live in very different countries) I just fork it and keep developing it with people who have the same riding style.

There are lots of small communities (your local cycling club, local bike advocacy group, city council) which could very well enjoy developing their own profiles based on their common taste or rules, I think.

@poutnikl

This comment has been minimized.

Copy link

poutnikl commented Oct 11, 2018

There will be always cases the profile would choose a different route then its end users, even if it matches their preferences quite well. And, OSM data versus subjective reality perception adds additional layer of the profile Vs user difference.

Unless there is a profile code bug causing some crazy routing at some scenario, it is often better to choose different profile.

(or different profile parameters. As you may know, my profiles comes from a tunable common profile template. A script can easily generate many different profiles from the template by adjusting different profile parameters.

https://github.com/poutnikl/Brouter-profiles/wiki

@wetneb

This comment has been minimized.

Copy link

wetneb commented Oct 11, 2018

I still do not understand your point but it seems that you would not find these tests useful. We seem to be talking past each other so I will not try to convince you further…

@poutnikl

This comment has been minimized.

Copy link

poutnikl commented Oct 11, 2018

It may be just mutual misunderstanding about the purpose. If it is test from developer or end user point of view.

If it behaves like it should,
or if it behaves a user would like to.

@poutnikl

This comment has been minimized.

Copy link

poutnikl commented Oct 12, 2018

P.S.: What I get is you may want to have some representative testing routes for modification of established built-in profiles, to see there was not introduced any regression.

Such a public routing test suite would obviously make sense.

OTOH, testing the introduced changes would rather have to rely on testing routes at a profile developer side, as it is not possible a priori predict what profile features and OSM data will be tested.

@abrensch

This comment has been minimized.

Copy link
Owner

abrensch commented Oct 12, 2018

A test case could be an origin, destination and one or more test
point(s) through which the computed route would need to pass
to validate/fail the test. To ease maintenance, test cases could
come with a short description of why the route is expected to
pass there rather than elsewhere. Each profile would have its
own test suite, but ideally it should be possible for these test
suites to share common tests.

Hi,

actually, there has ever been exactly one such test as part of the JUnit-Tests
that are part of the build-pocess:

https://github.com/abrensch/brouter/blob/master/brouter-server/src/test/java/btools/server/RouterTest.java

It checks that the result for a short route with the "trekking"-Profile is binary identical to a stored gpx-file. The map used is a very small map of the city "Dreieich" that is crossing th 50-degree-latitude.

The intention of that test is more technical, testing the basic function of the router and the profile, and testing the tile-boundary-crossing.

However, that would be a way to add more "microscopic tests" for special aspects: oneway-logic, access-restrictions, oneway:bicycle=opposite, etc. But as you see: just objective questions (Does it still work or is it broken?).

But I think it's not a way to test the fine-tuned behaviuor of a profile. There you would need something more "fuzzy". Maybe a set of random medium-distance-routes (on a frozen map), and checking some cost-deltas between new and reference results?

regards, Arndt

@Phyks

This comment has been minimized.

Copy link
Contributor

Phyks commented Oct 31, 2018

I guess @wetneb was referring mostly to simple examples such as the one in poutnikl/Trekking-Poutnik#23 (comment) (minimal distance, easy to understand / reproduce problem).

I guess this would mean storing an extract of the OSM data in the bounding box at this time, but this might be quite easy to do with Overpass.

@Phyks

This comment has been minimized.

Copy link
Contributor

Phyks commented Nov 3, 2018

I was thinking about this issue and trying to fine tune the trekking profile to avoid some weird features (TLDR, it seems that adding a penalty for highway=traffic_signals node gives really nice results, I still have some work to do on it).

I wrote a small iPython notebook to help test and compare profiles with both full routes (city / country along cycle routes / country without cycle routes) and special features that should be avoided: https://github.com/Phyks/BrouterTesting.

This is, of course, biased towards routes I am familiar with (and geographical area is France). I am using a fixed copy of the segments4 (https://pub.phyks.me/brouter-testing/segments4/) and tried to provide notes / explanations / "human route" (what I would personally do) to help reuse it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment