Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

get the public endpoint running again (use BVG endpoint for now) #29

Closed
berlincount opened this issue Jun 19, 2018 · 66 comments
Closed

get the public endpoint running again (use BVG endpoint for now) #29

berlincount opened this issue Jun 19, 2018 · 66 comments
Labels

Comments

@berlincount
Copy link

@berlincount berlincount commented Jun 19, 2018

$ curl -v 'https://2.vbb.transport.rest/stations/9007102/departures?when=1529416634&duration=30'
* Connected to 2.vbb.transport.rest (54.37.75.136) port 443 (#0)
> GET /stations/9007102/departures?when=1529416634&duration=30 HTTP/1.1
> Host: 2.vbb.transport.rest
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 504 Gateway Time-out
< Server: nginx/1.10.3 (Ubuntu)
< Date: Tue, 19 Jun 2018 13:53:00 GMT
< Content-Type: text/html
< Content-Length: 192
< Connection: keep-alive
<
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

thanks for having a look!

@joecorcoran

This comment has been minimized.

Copy link

@joecorcoran joecorcoran commented Jun 19, 2018

Yeah the V2 departures endpoint has been down since sometime yesterday. Looks like an upstream connection is gone? Not clear whether it's this project or the actual underlying API that has the problem. Anything we can do to help?

https://status.transport.rest/779961406

@joecorcoran

This comment has been minimized.

Copy link

@joecorcoran joecorcoran commented Jun 20, 2018

I can run this server locally and the departures endpoint seems fine, so that makes me think the problem is not upstream. Maybe just needs restarting?

curl http://localhost:3000/stations/900000120004/departures | jq .[0].journeyId

"1|31739|0|86|20062018"
@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jun 20, 2018

The upstream endpoint has often been down for very brief periods of time in the past days.

I can run this server locally and the departures endpoint seems fine, so that makes me think the problem is not upstream.

I can confirm this. Seems like they have blocked the IP address of my server or something. Will look into it later.

@rejas

This comment has been minimized.

Copy link

@rejas rejas commented Jun 23, 2018

Any news on this?

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jun 25, 2018

I requested a new IPv4 address for my VPS. Would like to avoid to moving the API over to another VPS.

If someone wants to host the endpoint or a fallback, let me know!

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jun 28, 2018

Still waiting for an IP address and currently don't have the time to migrate to a different VPS/VPS provider.

Again: If anyone is interested in hosting it, let me know! A really small VPS with nginx and Node.js installed is sufficient. @juliuste are you interested?

@juliuste

This comment has been minimized.

Copy link

@juliuste juliuste commented Jun 28, 2018

I can do it.

@juliuste

This comment has been minimized.

Copy link

@juliuste juliuste commented Jun 28, 2018

Endpoint should be up again.

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jun 28, 2018

Closing for now. We should find a long-term solution though for what I think is the cause: The VBB can block the IP address of the API server.

@derhuerst derhuerst closed this Jun 28, 2018
@joecorcoran

This comment has been minimized.

Copy link

@joecorcoran joecorcoran commented Jun 28, 2018

Thanks for solving this. I'm happy to contribute some € to the running costs. Let me know if that's possible.

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jun 28, 2018

@juliuste Do you have a PayPal/Patreon/OpenCollective/bank/crypto currency account?

You can support me on my Patreon page or via SEPA if you want to skip the fees. Keep in mind though:

Recently, I've started to work at Deutsche Bahn. I'm aware that I'm in a privileged position. Nevertheless, the sector of public transport is a place where open source work like mine is often not valued, let alone seen as necessary. Especially connecting public transport networks, for the benefit of everyone, is something that almost no company pursues. Independent support will allow me to keep working on what I – and assume you as well – think is truly important for passenger-friendly, accessible public transport services.

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jun 28, 2018

BTW @joecorcoran what are you building with the API?

@juliuste

This comment has been minimized.

Copy link

@juliuste juliuste commented Jun 28, 2018

Thanks for solving this. I'm happy to contribute some € to the running costs. Let me know if that's possible.

@juliuste Do you have a PayPal/Patreon/OpenCollective/bank/crypto currency account?

Thank you very much, but I'm alright, just support @derhuerst if you want to. 🙂

@rejas

This comment has been minimized.

Copy link

@rejas rejas commented Jun 28, 2018

@derhuerst how much traffic would a VPS generate if I'd rent something to be a fallback?

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jun 28, 2018

@rejas Not a lot. For a while I've hosted it on a 3€ Scaleway VPS. It should be around 50k-100k requests per day, but vbb-rest is very light on the CPU, it basically just parses JSON.

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jun 28, 2018

Not sure how one would do a fallback for this though without adding too much tooling. Maybe a load balancer that supports an /health HTTP route.

@derhuerst derhuerst changed the title The endpoint is broken? implement a fallback endpoint Jun 28, 2018
@derhuerst derhuerst reopened this Jun 28, 2018
@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jun 28, 2018

I will also investigate if vbb-rest runs out of the box on Now. Then we'd have one fallback endpoint for free.

@joecorcoran

This comment has been minimized.

Copy link

@joecorcoran joecorcoran commented Jun 29, 2018

BTW @joecorcoran what are you building with the API?

I'm building a departure board app, designed for leaving open on a screen. Taking it slowly as I've never used React before. :)

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jun 30, 2018

Have deployed a fallback endpoint at https://vbb-rest-ribczgxrzz.now.sh. We still need to figure out how to do the load balancing though.

@juliuste

This comment has been minimized.

Copy link

@juliuste juliuste commented Jul 12, 2018

Seems like the new endpoint has been blocked as well. @derhuerst we could use a load balancer + different now instances, for example…

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jul 12, 2018

@joecorcoran @rejas @berlincount can you deploy vbb-rest via Now? With a (free) account, run now in the vbb-rest repo and post me the URL it returns.

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jul 12, 2018

  • https://vbb-rest-usfolnrykj.now.sh
  • https://vbb-rest-ribczgxrzz.now.sh
@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Jul 12, 2018

Didn't get it to work using nginx, trying haproxy now. If someone knows any other simple and efficient load balancing solution, let me know.

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Aug 8, 2018

It works with railway station 900000310042 but not with the bus station 650030840706 (HCI Service: location missing or invalid), but that's okay for me ;)

What's the name of this 650030840706 station? Where did you get the ID from?

@timaschew

This comment has been minimized.

Copy link

@timaschew timaschew commented Aug 9, 2018

From the stops.txt of VBB

$ cat stops.txt | grep "Bad Saarow" | grep "Klinikum"
900000310109,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",1,
650030840701,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",0,900000310109
650030840706,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",0,900000310109
900000310042,"","Bad Saarow, Klinikum Bhf",,"52.284505000000","14.065870000000",1,
450009310042,"","Bad Saarow, Klinikum Bhf",,"52.284505000000","14.065870000000",0,900000310042

Seems that only 9xx are working, the others are duplicates anyway.

So 900000310042 is the railway station
and 900000310109 is the bus station which are both working.

Problem solved

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Aug 9, 2018

From the stops.txt of VBB

$ cat stops.txt | grep "Bad Saarow" | grep "Klinikum"
900000310109,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",1,
650030840701,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",0,900000310109
650030840706,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",0,900000310109
900000310042,"","Bad Saarow, Klinikum Bhf",,"52.284505000000","14.065870000000",1,
450009310042,"","Bad Saarow, Klinikum Bhf",,"52.284505000000","14.065870000000",0,900000310042

The 6* stops are stops of the 9* stations. The API doesn't seem to know about this hierarchy (as you said, it doesn't know about them at all).

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Aug 17, 2018

To re-post from above:

Good news everyone!

I have set up a new API: 1.bvg.transport.rest provides data from the BVG endpoint. It returns almost exactly the same data as 2.vbb.transport.rest, but in the output format of hafas-client@3.0.0-alpha.20, so you will have to make minor adjustments. Check the docs at derhuerst/bvg-rest!

Let's see for how long this API will stay up.

@derhuerst derhuerst changed the title get the public endpoint running again get the public endpoint running again (use BVG endpoint for now) Aug 24, 2018
@derhuerst derhuerst added the upstream label Aug 24, 2018
@michaelcheers

This comment has been minimized.

Copy link

@michaelcheers michaelcheers commented Nov 30, 2018

Seems like the vbb endpoint is working now.

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Nov 30, 2018

Seems like the vbb endpoint is working now.

Can't reproduce. 😕

curl 'https://2.vbb.transport.rest/stations/900000013102/departures'
@michaelcheers

This comment has been minimized.

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Nov 30, 2018

Hm, it seems like half of the features of the API are working again.

@deg0nz

This comment has been minimized.

Copy link

@deg0nz deg0nz commented Apr 21, 2019

I just discovered dfb6c4a. Does this mean, we will have a new working endpoint for VBB data? Are the rate-problems with VBB solved?

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Apr 24, 2019

As mentioned above, I introduce breaking changes in hafas-client or any of its wrapper libraries (such as vbb-hafas or db-hafas) every now an then. When doing this, I increase the major version number by 1.

I've recently released hafas-client@4 and – building on top of that – vbb-hafas@6, db-hafas@4, bvg-hafas@2, etc. For the wrapping REST APIs, I've created a branch each for the old version (2 in vbb-rest, 2 for db-rest and 1 for bvg-rest) and adapted master to the aforementioned latest *-hafas versions.

Long story short: 3.vbb.transport.rest/3.db.transport.rest/2.bvg.transport.rest represent the latest data format based on hafas-client@4. I haven't announced these APIs publicly, so expect sudden changes.

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Apr 24, 2019

Does this mean, we will have a new working endpoint for VBB data?

With the new 3.vbb.transport.rest API the fundamental problem is not solved: The VBB endpoint may still block the servers' IP address(es) if there are too many requests.

We can reduce the chance by load-balancing over multiple servers though, so please deploy the latest REST APIs on your servers or cloud services (such as DigitalOcean, Scaleway, AWS, Now or Heroku):

  1. Check out vbb-rest#3/db-rest#3/bvg-rest#2.
  2. Deploy it and check its health with the /health HTTP route (e.g. curl 'https://my-vbb-rest-3-instance.example.org/health).
  3. Post its URL here.
@deg0nz

This comment has been minimized.

Copy link

@deg0nz deg0nz commented Apr 24, 2019

Alright, thank you for clarifying.

I guess I was a little confused by the 3 in 3.vbb.transport.rest I thought that you maybe had a talk with the VBB people... But yeah, your explanation makes sense :)

@jonathan-reisdorf

This comment has been minimized.

Copy link

@jonathan-reisdorf jonathan-reisdorf commented May 1, 2019

3. Post its URL here.

Hi, I can offer these for the lb:

https://db.fishlevel.com
https://vbb.fishlevel.com
https://bvg.fishlevel.com

The instances run on a bare-metal server located in a data center in Berlin.
I could also add the version prefix in the domain if you prefer.
Do you know how much traffic is generated on average per month?

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented May 1, 2019

@jonathan-reisdorf

Thank you! 💚

I could also add the version prefix in the domain if you prefer.

Yes please, it helps me not to misconfigure something by accident. You could adopt the naming scheme of @juliuste and me (Julius will run b.*, I run a.*):

Do you know how much traffic is generated on average per month?

All 6 endpoints together get receive ~50-500k requests per day. On my (really cheap) VPS, they usually contribute to a load of about 0.15.

@jonathan-reisdorf

This comment has been minimized.

Copy link

@jonathan-reisdorf jonathan-reisdorf commented May 4, 2019

@derhuerst ok, I have set these up and also c.3.db.fishlevel.com. The load is no problem, just the traffic might be, bc the data center where my server is housed has a fair-use policy regarding traffic (-> if it's "a couple of hundred GB" per month I might have to pay an additional fee), so let's see how much traffic is generated after a month :)

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented May 7, 2019

We can also define your nodes as fallback, so that they won't receive any requests if a.* or b.* are up.

@jonathan-reisdorf

This comment has been minimized.

Copy link

@jonathan-reisdorf jonathan-reisdorf commented May 8, 2019

👍 sounds great!

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented May 8, 2019

@joecorcoran @deg0nz @ferby09 @k-nut @berlincount @timaschew Do you happen to have the knowledge and time to set up mirror/load balancing nodes as well? I would appreciate it.

@k-nut

This comment has been minimized.

Copy link

@k-nut k-nut commented May 13, 2019

Sorry, I don't have capacities for this right now...

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Oct 28, 2019

I've set up 3.vbb.transport.rest a while ago (forgot to let everyone know about this), and so far it has been running reasonably well, so please use this endpoint or 2.bvg.transport.rest. Unfortunately both have a slightly updated output format, so make sure to check their docs.

I'm going to close this issue now, as the VBB admins haven't blocked 3.vbb.transport.rest so far. Thanks for you patience!

@derhuerst

This comment has been minimized.

Copy link
Owner

@derhuerst derhuerst commented Mar 23, 2020

I have opened a thread in the transport.rest meta-repo to discuss hosting further: public-transport/transport.rest#2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.