Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Demoserver - call for hosting volunteers #5655

Closed
danpat opened this issue Jan 24, 2020 · 19 comments
Closed

Demoserver - call for hosting volunteers #5655

danpat opened this issue Jan 24, 2020 · 19 comments

Comments

@danpat
Copy link
Member

danpat commented Jan 24, 2020

On Mar 09, 2020, the TLS certificate for router.project-osrm.org is going to expire.  At the same time, Mapbox is going to shutdown our hosting of that service.  As many of you have probably noticed, our investment in the OSRM codebase has greatly decreased over the last year.

This is a call to the community to see if someone else is willing to take up the mantle for running the demo server.  If nobody steps up, then the service will stop working on Mar 09, 2020.

Below is a chart that shows the request load for the last two weeks - the value here is "requests per minute".

Screen Shot 2020-01-24 at 11 12 41 AM

I have also posted this notice on the osrm-talk mailing list for wider discussion.

@1ec5
Copy link
Member

1ec5 commented Jan 28, 2020

This might be worth cross-posting to a wider audience, such as the main talk list. I also posted a mention in OSMUS Slack.

Note that router.project-osrm.org is distinct from router.openstreetmap.de.

@SrihariThalla
Copy link

SrihariThalla commented Feb 5, 2020

I can host the demo server.

Here is an existing one - Germany Car profile:

I have also posted on the OSRM mailing list (https://lists.openstreetmap.org/pipermail/osrm-talk/2020-February/001837.html)

@akashihi
Copy link
Contributor

akashihi commented Feb 10, 2020

What are the current host specs?

@SrihariThalla
Copy link

SrihariThalla commented Feb 11, 2020

What are the current host specs?

If you are asking me, I'm using a 4GB RAM with a 2 vCPU server. I am hosting Germany's Car profile. Since there is very little traffic, I didn't need higher specs for now.

@akashihi
Copy link
Contributor

akashihi commented Feb 12, 2020

No, i asked @danpat for server specs, so i'll be able to estimate the expenses.

@danpat
Copy link
Member Author

danpat commented Feb 13, 2020

Unfortunately, I don't have a great answer to the size question. Currently, the demoserver is running on Mapbox's shared infrastructure, and we've partitioned the data out across several machines (smaller geographic regions), which also serve some of our other traffic, so I can't easily point to one box in our system and say "that's the demoserver". We've simply absorbed it as overhead.

If I were contemplating running the service globally, I'd be aiming for:

  1. Enough RAM to load all files into memory (about 70GB for a car.lua CH dataset last time I looked, but I haven't processed the whole planet for a while)
  2. Enough CPU cores to keep up with the demoserver request volume (250/sec). Estimating 50ms per request, you'd need 12-16 cores to do that.

Here's the daily network traffic for the last couple of weeks to get an idea of bandwidth volumes. The p?? values are response time percentiles, in seconds. This is across all request types (/route, /table, /match, /trips, /nearest):

  dt total_requests outgoing_bytes incoming_bytes p50 p90 p95 p99
1 2020-02-12 20957484 32020135708 8899911485 0.007 0.052 0.052 0.053
2 2020-02-11 18375314 34171759908 7888859078 0.007 0.052 0.052 0.054
3 2020-02-10 17536583 37150229241 7326692479 0.007 0.052 0.052 0.054
4 2020-02-09 11766243 31729906622 4949341021 0.007 0.051 0.052 0.053
5 2020-02-08 12630320 33114101719 5324081819 0.007 0.052 0.052 0.054
6 2020-02-07 18440429 41877401205 7786411052 0.008 0.052 0.052 0.054
7 2020-02-06 16306322 45444040204 6970381051 0.007 0.052 0.052 0.054
8 2020-02-05 15910468 40002820959 6950761916 0.007 0.052 0.052 0.054
9 2020-02-04 16412076 35710951466 6948442645 0.007 0.052 0.052 0.054
10 2020-02-03 15240168 43539650561 6408017674 0.007 0.052 0.052 0.053
11 2020-02-02 11824870 38352572280 5136492785 0.007 0.051 0.052 0.054
12 2020-02-01 13011131 41814895623 5702334096 0.007 0.052 0.052 0.054

You also need to factor in the data processing costs - the expense here depends highly on how often you refresh the map. Currently it's done daily. You'd want 256 or more GB of RAM for the fastest osrm-extract/osrm-contract turnaround time, and the more CPUs you can give it the better. I think you can turnaround a planet-sized CH dataset in 12-16 hours with a 32 core machine and 256GB of RAM.

@akashihi
Copy link
Contributor

akashihi commented Feb 14, 2020

@danpat It looks more demanding than i expect. I sent you a mail with more private details.

@SrihariThalla
Copy link

SrihariThalla commented Feb 15, 2020

It's too much for me. Sorry, I had to drop my proposal.

@akashihi
Copy link
Contributor

akashihi commented Mar 2, 2020

@danpat We still can do that, but i can't reach you to discuss the details

@rCarto
Copy link

rCarto commented Mar 12, 2020

Hello,
What is the current situation? What is the foreseen future of the demo server?
As of today it seems that the http://router.project-osrm.org/ server is still running.

@danpat
Copy link
Member Author

danpat commented Mar 12, 2020

Sorry, I've been totally slammed by life and work.

Yes, the demoserver is still running, it's not being turned off on our side (yet, no direct timeline for that yet, but it'll happen eventually), but the https:// certificate (if you access it over https) has expired and won't be renewed by Mapbox.

A couple of people have offered to host a box to replace it, I need to find time to follow-up with them.

@datendelphin
Copy link

datendelphin commented Mar 21, 2020

@danpat I opened the following pull request to fix the demo site for now: Project-OSRM/osrm-frontend#319

@answerquest
Copy link

answerquest commented Mar 21, 2020

@danpat A suggestion: How about splitting this country-wise (or region wise) and restructuring on the demo website also to let the user pick a country first? Then things can be divided among volunteers. Or you can just add links on the site to take users to country-specific URLs.

I'd like to volunteer to host OSRM for India. I'm using OSRM (our own hosted) heavily in my work and have even customized the profile a bit to have left-side driving and to permit pedestrians on main roads. I have a digitalocean server droplet mapped to my personal domain and have free LetsEncrypt SSL on it . It's working fine since over 2 years. Here's a page I'm hosting that has a backend daily fetching latest OSM data and processing it: https://server.nikhilvj.co.in/OSM/roads.html

Additional request : Permit the country-wise volunteers to put a small link to accept contributions towards the server or contact the hoster for professional enquiries. This way if there are heavier users in a particular place, they can get in touch with the hoster, take ownership instead of socializing the burden to the world. (I have a hunch that 80% of your usage load is coming from very few people who have hooked it up to their forever-running programs - if we split by region, majority of places probably won't have much traffic)

I know this isn't a fancy full-planet solution, but hey if you have to choose between complete shutdown and decentralizing, why not decentralize.

@datendelphin
Copy link

datendelphin commented Mar 25, 2020

@answerquest That is a good idea. But it requires a bit of development on the demo site. Would anyone be willing to do this? I incorporated a drop down for routing.osm.ch and routing.openstreetmap.de but that was already time consuming and too much javascript for me.

@mwargan
Copy link

mwargan commented May 6, 2020

Can we not use https://routing.openstreetmap.de?

@datendelphin
Copy link

datendelphin commented May 31, 2020

map.project-osrm.org is now switched to the fossgis server as the backend. https://github.com/fossgis-routing-server/osrm-frontend

@danpat
Copy link
Member Author

danpat commented Jun 4, 2020

Looks like somebody redirected router.project-osrm.org at router.openstreetmap.de as well.

@danpat danpat closed this as completed Jun 4, 2020
@mandeepsandhu
Copy link
Contributor

mandeepsandhu commented Jun 4, 2020

Looks like somebody redirected router.project-osrm.org at router.openstreetmap.de as well.

It was announced few comments above I suppose?

@danpat
Copy link
Member Author

danpat commented Jun 5, 2020

@mandeepsandhu The comment only mentioned map.project-osrm.org, not the router.project-osrm.org CNAME (first one hosted the osrm-frontend project via Github Pages, the second one hosted an osrm-routed API instance).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants