How I got valhalla running #3540
timwis
started this conversation in
Show and tell
Replies: 2 comments
-
Wow, congrats you pulled through all the way and thanks for sharing! It’s really nice to have someone sorting out the transitland stuff again, was always too lazy myself😅 Didn’t read it all but couple of comments:
|
Beta Was this translation helpful? Give feedback.
0 replies
-
Was going to mention the centroid api as well for finding convenient places to meet. Also seems like your found some warts, we will get those patched over. Thanks for reporting! |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello all, I love what I've read about valhalla in its docs, and I'd like to use it for an application I'm building to help people find mutually convenient locations to meet. Specifically, I'd like to use the isochrone generation service, primarily with public transport data, but also with other modes of travel. My initial focus will be London and the UK, but would eventually like to support more areas, and it looks like I can do all of that with valhalla.
But I've had some trouble getting it running. Some of the documentation is still a work in progress, and I ran into issues with some of the build scripts on my M1 macbook pro (when running in docker), e.g.:
So I decided to fire up a digital ocean virtual private server (vps) to eliminate the M1 chip from the equation.
I decided to document everything so that (a) I could eventually reproduce it when it comes time to run it in production, and (b) in case anyone else comes across this repo and, like me, hasn't worked with C++ before and is struggling to get it running, and (c) to potentially surface areas where documentation could be improved (which I'm happy to help with).
It's more of a live coding exercise than a tutorial, but I hope it's helpful! Perhaps I'll adapt it into a tutorial if that would be useful.
setting up the server
I used a Digital Ocean virtual private server (vps) because they're cheap ($5 USD per month) and the interface is much simpler than the other cloud providers.
I could have used the documentation on running valhalla on linux, but I chose to use the docker image because I'd like to eventually run this in production, possibly on a PaaS, and perhaps even on my macbook during development, and the docker image is more portable.
So I installed docker and docker compose.
I then created a
/valhalla
directory with a/resources
directory inside it. I created/valhalla/docker-compose.yml
with these contents:This makes it easier to run the valhalla 'run' docker image, persisting the artefacts it generates, and exposing the server's port to the host machine.
I then ran this image and attached my terminal to it via:
This put me inside the container, where I navigated to
/valhalla/resources
, and ran all my commands from there.preparing to run valhalla
I originally assumed that the valhalla docker image was "ready to go" and I could immediate generate isochrones with it. In fact, you must first download and prepare data for it to use, as well as a config to point to that data.
I use the 'running' section of the README as guidance for this.
generating the config
Running
valhalla_build_config
throws an error:So we need to install distutils
That should probably be included in the docker image, though perhaps it's because we're using the 'run' image instead of the 'build' image.
Now we can generate the build config with default settings
We'll need to modify some of the default settings in the build config. I used
vim
since I'm on a VPS, but you can use any text editor./valhalla/resources/{service-name}.log
/data/valhalla/
to/valhalla/resources/
Next we'll download OSM data for greater london into the
/valhalla/resources
directoryNow we'll follow the steps in the documentation's optional prerequisites section for mjolnir.
Administrative areas
From
/valhalla/resources
, we'll run:This threw a handful of errors:
I think that's because we used greater london data instead of "planet" level data. So my guess is this step was probably somewhat useless with this data as there are no country crossings within greater london.
Timezones
This one was pretty simple:
Transit
First we'll make sure the target directory exists
Running
valhalla_build_transit
without any arguments shows the usage is:The bounding box argument isn't clear whether it's in the order of latitude-longitude or longitude-latitude. According to the OpenStreetMap Wiki:
From this we can infer that it's longitude-latitude.
That wiki page also happens to have an example for greater london, which saves us time looking that up:
-0.489,51.28,0.236,51.686
For the TransitLand API key, we'll need to sign up.
I'm not sure what
import_level
means, so we'll just use4
from the sample.To find Transport for London's Onestop ID, we can search
London
on TransitLand's operators index. It looks like the Onestop ID for TfL iso-gcpv-transportforlondon
, but according to the Onestop ID explanation, the 'o' is for operator, and thevalhalla_build_transit
usage prompt calls for afeed_onestop_id
, so we'll swap the 'o' for an 'f'.So the full command ended up being:
Unfortunately this threw an error:
I copied the URL into my browser and saw a more descriptive error message:
🤔 Hm. Now that I look again at the URL, I don't see the onestop_id I provided in there. Perhaps the cause for the error.
Maybe the usage sample on the
valhalla_build_transit
CLI is out of date. Let's look at TransitLand's API docs. (EDIT: I later realised valhalla uses an old version of the TransitLand API, v1)There's a parameter called
onestop_id
. If we append&onestop_id=f-gcpv-transportforlondon
to the URL in the browser, it works. Why isn'tvalhalla_build_transit
doing that?Let's search the valhalla codebase for
feeds.geojson
to see if we can find the code for constructing that URL.If I'm understanding this correctly—and I don't read C++ well—it looks like the onestop_id is not added to the URL; instead,
valhalla_build_config
looks for the result that has a matching onestop_id.So let's search valhalla's issues for any reference to this. It may also be a bug in transitland's api.
I don't see any reference to it in valhalla's issues. Poking around transitland's GitHub org, this repo looks like the one that powers the web service, which checks out because the error looks like a ruby error.
I searched transitland-datastore's issues for
NoMethodError
and saw some other issues, but not this one. So I searched its source code foronestop_id
in the/app
path, but there are 62 results, and without line or file information from the error, it would be very hard to narrow it down without running the whole app locally.In the process of posting an issue, though, I realised I was looking at the wrong version of the transitland api docs. Valhalla's making requests to
/v1/
, which has its own documentation page. Sure enough, there's an example query there in the same format:This suggests valhalla is making an appropriate query, and the issue is on the transitland side (albeit on an old and probably deprecated version of their API). I tried a few other variations of the URL to try and narrow down the issue, but they all yielded the same error message, so I posted an issue on the transitland-datastore repo and cross-posted on valhalla's repo for visibility.
I think for now we'll continue without transit data, and revisit when we hear back on those issues.
EDIT: The TransitLand team fixed this issue, and the
valhalla_build_transit
tool worked with the above command, but then quickly ran into a rate limiting issue. I think the way I'll get around this is by running transitland v1 locally, and replacing the TransitLand endpoint in the build command with my local server's endpoint.Building tiles
I originally assumed tiles were only for basemaps, which I didn't need from valhalla, but it turns out they're used for routing, which is necessary for generating isochrones. So we'll need to build them.
I can see an example query in valhalla's readme, so we'll try that:
This command generates a plethora of lines with green in them, and no red, which is always a great feeling.
After a minute or so, though, all I'm getting is this warning for about ten minutes:
My guess is this has to do with the small amount of RAM that comes with digital ocean's tiniest VPS. Fortunately, a bunch of green lines came after that, and it finished 🎉
Building an extract
Valhalla's readme lists another command after this:
valhalla_build_extract
, which it says is to build a tile index for faster graph loading times. We'll run the command used in the example:Unfortunately, this throws an error:
Typing
valhalla_
and pressing TAB shows me all the available valhalla scripts, and that isn't one of them.I can see
valhalla_build_extract
is still in the repo, but for some reason it's not available in the docker 'run' image.Looking at the run image's dockerfile, this may be deliberate. Perhaps we should have been using the build image this whole time, and perhaps that's why the python-distutils package was missing when we started.
Fortunately, we mounted the
/resources
directory when running the 'run' container, so if we stop the container and run a 'build' container with the same directory mounted, we shouldn't need to rebuild anything.Let's add the 'build' image to our
docker-compose.yml
file, indented underservices
like the 'run' image:(We'll omit the
ports
property since we can assume build isn't meant to be the one running a web server)We'll run it using:
And then we'll try that
valhalla_build_extract
command again.Unfortunately, this fails too, with the same
command not found
error. Pressing TAB after typingvalhalla_
doesn't show any scripts this time.I can see in the dockerfile for the build image it runs
install-shared-deps.sh
, so I tried running that manually, but that didn't fix the issue.Looking at the Dockerfile and install-shared-deps.sh, neither appear to copy the valhalla source code (which includes the
valhalla_build_extract
script), which the 'run' Dockerfile does, so maybe we were right to use the 'run' container after all.The valhalla readme does offer an alternative to the
valhalla_build_extract
command, of simply tarring up the tiles, so let's try that.First, let's switch back to the 'run' image by pressing CTRL+C to detach from and stop the 'build' container, then running the 'run' container:
Once we're in the container, navigate back to the
/valhalla/resources
directory and try the command from the readme, changing the namevalhalla_tiles
totiles
, to match the paths in our config file:That's all the steps listed in the readme, so I think we're ready to try running the server!
Running the server
The
docker-compose run
command can't expose ports, so we'll need to exit the container (CTRL+C) and usedocker-compose up
. This will immediately stop the container, because the run container doesn't have a CMD in it (deliberately). So we'll need to add a command to ourdocker-compose.yml
file:I believe the
1
at the end is the number of cores. We'll run this service using:We get 1 success message and 3 warnings:
These warnings make sense because we didn't load or process traffic or elevation data, but our config file still has the default paths in it, so valhalla looked for it.
Let's see if it works anyway. In another terminal tab, we'll SSH into our digital ocean vps again:
Then we can try making a request to the server we're running via docker.
From the isochrone docs, we can see that the server runs on port
8002
(which is why we exposed that port in ourdocker-compose.yml
), and it gives a sense of the expected URL structure, along with a sample request. Let's usecurl
to send that sample request:This returns an error:
If you look closely, this is actually a curl related issue: the double quotes in the URL is likely the problem. curl probably expects them to be encoded (e.g.
%22
).This article offers a very elegant solution, which we'll try:
This returns an error from valhalla:
If we search valhalla's issues for that error message, we can see a comment from Kevin that it means either:
Now that I think about it, we only loaded in data for greater london, but we copied and pasted a query from the isochrone docs containing a lat/lon in New York.
Let's try it with a London lat/lon:
It worked!! 🎉 We got back a GeoJSON LineString object. And if we try to access the digital ocean server directly from my laptop's browser (rather than via SSH) on port 8002, it works! That means the server must have port 8002 open to the public internet by default.
Beta Was this translation helpful? Give feedback.
All reactions