Automated City Acceptance tests

[orangejulius]

Hey team!

In our first community call a great question was raised around testing. Historically, the main source of answers for "does Pelias as a whole work?" and similar questions has been our acceptance-tests: a set of around 600 hand-curated tests that require a full planet instance of Pelias.

Setting up a full planet build of Pelias costs a minimum of a few hundred dollars in server costs, and takes a while to run. So it's a very slow feedback cycle that is also hard to get working at all. This was in fact a point of considerable pain and slowdown for us at Mapzen.

While full planet testing will always be valuable and useful, it's a good idea to think of other types of tests that can be used first.

From our discussion, it would be very helpful to select maybe 10 small regions, or combination of regions, curate a set of acceptance tests for each region, and run a build for all those regions very frequently. A city-sized Pelias build can be run with the dockerfiles setup in about 20 minutes on relatively small hardware.

I think we could even run something like this on TravisCI or CircleCI, so the setup and recurring costs would be minimal or zero. Does anyone have proposals for how we manage the regional acceptance tests or the builds?

We also need to choose areas to test against. Some cities are essentially guaranteed to be on that list, such as NYC, Portland, and Berlin. However I think we should strive for a city on every continent (except Antarctica :) ), and also some rural areas.

[adelcasse]

Great idea ! We can probably contribute with a french city (Paris ?) if needed/usefull

[ashleyha]

Great idea! A couple follow-on questions:

1. It might also be useful to discuss guidelines for developing good tests, as well as test coverage, to support consistency between regions. Is there existing Pelias documentation that might be a good starting point for this?

2. In addition to the developer community here, wondering if there might also be opportunities to reach out to and incorporate the open mapping community to leverage overlap in workflows and local knowledge of place locations and names. Would there be value in exploring some mapathon / testing hackathon options to generate user-provided test cases?

[adelcasse]

On the proposition 2. of @Ashleyholt, we could maybe imagine, for those who have an instance exposed to "real world users" (when integrated in an app for example), finding a way to save anonymized usage data and share these datas to get analytics on searches/autocompletes, results and accuracy ? (and it would be a good way to get test cases from real users)