get the public endpoint running again (use BVG endpoint for now) #29

berlincount · 2018-06-19T13:55:28Z

$ curl -v 'https://2.vbb.transport.rest/stations/9007102/departures?when=1529416634&duration=30'
* Connected to 2.vbb.transport.rest (54.37.75.136) port 443 (#0)
> GET /stations/9007102/departures?when=1529416634&duration=30 HTTP/1.1
> Host: 2.vbb.transport.rest
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 504 Gateway Time-out
< Server: nginx/1.10.3 (Ubuntu)
< Date: Tue, 19 Jun 2018 13:53:00 GMT
< Content-Type: text/html
< Content-Length: 192
< Connection: keep-alive
<
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

thanks for having a look!

The text was updated successfully, but these errors were encountered:

joecorcoran · 2018-06-19T16:20:59Z

Yeah the V2 departures endpoint has been down since sometime yesterday. Looks like an upstream connection is gone? Not clear whether it's this project or the actual underlying API that has the problem. Anything we can do to help?

https://status.transport.rest/779961406

joecorcoran · 2018-06-20T07:52:57Z

I can run this server locally and the departures endpoint seems fine, so that makes me think the problem is not upstream. Maybe just needs restarting?

curl http://localhost:3000/stations/900000120004/departures | jq .[0].journeyId

"1|31739|0|86|20062018"

derhuerst · 2018-06-20T08:41:14Z

The upstream endpoint has often been down for very brief periods of time in the past days.

I can run this server locally and the departures endpoint seems fine, so that makes me think the problem is not upstream.

I can confirm this. Seems like they have blocked the IP address of my server or something. Will look into it later.

rejas · 2018-06-23T09:36:34Z

Any news on this?

derhuerst · 2018-06-25T09:17:27Z

I requested a new IPv4 address for my VPS. Would like to avoid to moving the API over to another VPS.

If someone wants to host the endpoint or a fallback, let me know!

derhuerst · 2018-06-28T09:39:32Z

Still waiting for an IP address and currently don't have the time to migrate to a different VPS/VPS provider.

Again: If anyone is interested in hosting it, let me know! A really small VPS with nginx and Node.js installed is sufficient. @juliuste are you interested?

juliuste · 2018-06-28T12:13:43Z

I can do it.

juliuste · 2018-06-28T13:19:04Z

Endpoint should be up again.

derhuerst · 2018-06-28T14:08:06Z

Closing for now. We should find a long-term solution though for what I think is the cause: The VBB can block the IP address of the API server.

joecorcoran · 2018-06-28T14:10:30Z

Thanks for solving this. I'm happy to contribute some € to the running costs. Let me know if that's possible.

derhuerst · 2018-06-28T14:54:35Z

@juliuste Do you have a PayPal/Patreon/OpenCollective/bank/crypto currency account?

You can support me on my Patreon page or via SEPA if you want to skip the fees. Keep in mind though:

Recently, I've started to work at Deutsche Bahn. I'm aware that I'm in a privileged position. Nevertheless, the sector of public transport is a place where open source work like mine is often not valued, let alone seen as necessary. Especially connecting public transport networks, for the benefit of everyone, is something that almost no company pursues. Independent support will allow me to keep working on what I – and assume you as well – think is truly important for passenger-friendly, accessible public transport services.

derhuerst · 2018-06-28T14:55:28Z

BTW @joecorcoran what are you building with the API?

juliuste · 2018-06-28T14:57:27Z

Thanks for solving this. I'm happy to contribute some € to the running costs. Let me know if that's possible.

@juliuste Do you have a PayPal/Patreon/OpenCollective/bank/crypto currency account?

Thank you very much, but I'm alright, just support @derhuerst if you want to. 🙂

rejas · 2018-06-28T15:12:57Z

@derhuerst how much traffic would a VPS generate if I'd rent something to be a fallback?

derhuerst · 2018-06-28T15:33:24Z

@rejas Not a lot. For a while I've hosted it on a 3€ Scaleway VPS. It should be around 50k-100k requests per day, but vbb-rest is very light on the CPU, it basically just parses JSON.

derhuerst · 2018-06-28T15:34:20Z

Not sure how one would do a fallback for this though without adding too much tooling. Maybe a load balancer that supports an /health HTTP route.

derhuerst · 2018-06-28T19:16:44Z

I will also investigate if vbb-rest runs out of the box on Now. Then we'd have one fallback endpoint for free.

joecorcoran · 2018-06-29T07:56:41Z

BTW @joecorcoran what are you building with the API?

I'm building a departure board app, designed for leaving open on a screen. Taking it slowly as I've never used React before. :)

derhuerst · 2018-06-30T10:52:26Z

Have deployed a fallback endpoint at https://vbb-rest-ribczgxrzz.now.sh. We still need to figure out how to do the load balancing though.

juliuste · 2018-07-12T11:38:09Z

Seems like the new endpoint has been blocked as well. @derhuerst we could use a load balancer + different now instances, for example…

derhuerst · 2018-07-12T16:49:04Z

@joecorcoran @rejas @berlincount can you deploy vbb-rest via Now? With a (free) account, run now in the vbb-rest repo and post me the URL it returns.

derhuerst · 2018-07-12T16:51:55Z

https://vbb-rest-usfolnrykj.now.sh
https://vbb-rest-ribczgxrzz.now.sh

derhuerst · 2018-07-12T18:33:44Z

Didn't get it to work using nginx, trying haproxy now. If someone knows any other simple and efficient load balancing solution, let me know.

michaelcheers · 2018-11-30T16:37:23Z

Seems like the vbb endpoint is working now.

derhuerst · 2018-11-30T17:59:39Z

Seems like the vbb endpoint is working now.

Can't reproduce. 😕

curl 'https://2.vbb.transport.rest/stations/900000013102/departures'

michaelcheers · 2018-11-30T20:42:03Z

@derhuerst, try https://2.vbb.transport.rest/stations?query=oranienburg

derhuerst · 2018-11-30T22:31:41Z

Hm, it seems like half of the features of the API are working again.

deg0nz · 2019-04-21T16:27:36Z

I just discovered dfb6c4a. Does this mean, we will have a new working endpoint for VBB data? Are the rate-problems with VBB solved?

derhuerst · 2019-04-24T09:31:46Z

As mentioned above, I introduce breaking changes in hafas-client or any of its wrapper libraries (such as vbb-hafas or db-hafas) every now an then. When doing this, I increase the major version number by 1.

I've recently released hafas-client@4 and – building on top of that – vbb-hafas@6, db-hafas@4, bvg-hafas@2, etc. For the wrapping REST APIs, I've created a branch each for the old version (2 in vbb-rest, 2 for db-rest and 1 for bvg-rest) and adapted master to the aforementioned latest *-hafas versions.

Long story short: 3.vbb.transport.rest/3.db.transport.rest/2.bvg.transport.rest represent the latest data format based on hafas-client@4. I haven't announced these APIs publicly, so expect sudden changes.

derhuerst · 2019-04-24T09:49:59Z

Does this mean, we will have a new working endpoint for VBB data?

With the new 3.vbb.transport.rest API the fundamental problem is not solved: The VBB endpoint may still block the servers' IP address(es) if there are too many requests.

We can reduce the chance by load-balancing over multiple servers though, so please deploy the latest REST APIs on your servers or cloud services (such as DigitalOcean, Scaleway, AWS, Now or Heroku):

Check out vbb-rest#3/db-rest#3/bvg-rest#2.
Deploy it and check its health with the /health HTTP route (e.g. curl 'https://my-vbb-rest-3-instance.example.org/health).
Post its URL here.

deg0nz · 2019-04-24T16:08:48Z

Alright, thank you for clarifying.

I guess I was a little confused by the 3 in 3.vbb.transport.rest I thought that you maybe had a talk with the VBB people... But yeah, your explanation makes sense :)

jonathan-reisdorf · 2019-05-01T13:14:57Z

3. Post its URL here.

Hi, I can offer these for the lb:

https://db.fishlevel.com
https://vbb.fishlevel.com
https://bvg.fishlevel.com

The instances run on a bare-metal server located in a data center in Berlin.
I could also add the version prefix in the domain if you prefer.
Do you know how much traffic is generated on average per month?

derhuerst · 2019-05-01T16:32:40Z

@jonathan-reisdorf

Thank you! 💚

I could also add the version prefix in the domain if you prefer.

Yes please, it helps me not to misconfigure something by accident. You could adopt the naming scheme of @juliuste and me (Julius will run b.*, I run a.*):

c.3.vbb.fishlevel.com -> derhuerst/vbb-rest#3
c.2.vbb.fishlevel.com -> derhuerst/vbb-rest#2
c.2.bvg.fishlevel.com -> derhuerst/bvg-rest#2
c.1.bvg.fishlevel.com -> derhuerst/bvg-rest#1
c.2.db.fishlevel.com -> derhuerst/db-rest#2
c.1.db.fishlevel.com -> derhuerst/db-rest#1

Do you know how much traffic is generated on average per month?

All 6 endpoints together get receive ~50-500k requests per day. On my (really cheap) VPS, they usually contribute to a load of about 0.15.

jonathan-reisdorf · 2019-05-04T11:35:02Z

@derhuerst ok, I have set these up and also c.3.db.fishlevel.com. The load is no problem, just the traffic might be, bc the data center where my server is housed has a fair-use policy regarding traffic (-> if it's "a couple of hundred GB" per month I might have to pay an additional fee), so let's see how much traffic is generated after a month :)

derhuerst · 2019-05-07T15:48:41Z

We can also define your nodes as fallback, so that they won't receive any requests if a.* or b.* are up.

jonathan-reisdorf · 2019-05-08T05:56:59Z

👍 sounds great!

derhuerst · 2019-05-08T18:33:41Z

@joecorcoran @deg0nz @ferby09 @k-nut @berlincount @timaschew Do you happen to have the knowledge and time to set up mirror/load balancing nodes as well? I would appreciate it.

k-nut · 2019-05-13T06:57:39Z

Sorry, I don't have capacities for this right now...

derhuerst · 2019-10-28T16:03:13Z

I've set up 3.vbb.transport.rest a while ago (forgot to let everyone know about this), and so far it has been running reasonably well, so please use this endpoint or 2.bvg.transport.rest. Unfortunately both have a slightly updated output format, so make sure to check their docs.

I'm going to close this issue now, as the VBB admins haven't blocked 3.vbb.transport.rest so far. Thanks for you patience!

derhuerst · 2020-03-23T17:35:53Z

I have opened a thread in the transport.rest meta-repo to discuss hosting further: public-transport/transport.rest#2

derhuerst · 2020-05-09T14:33:43Z

I have set up v5.vbb.transport.rest, the successor of 3.vbb.transport.rest. As usual, unfortunately the response format has changed slightly (to the format of hafas-client@5, make sure to check its migration guide).

I will keep 3.vbb.transport.rest running for a while, and announce its shutdown via RSS before.

berlincount mentioned this issue Jun 19, 2018

"Fehler bei Daten abrufen" - Problems to fetch Data deg0nz/MMM-PublicTransportBerlin#32

Closed

derhuerst closed this as completed Jun 28, 2018

derhuerst added enhancement help wanted labels Jun 28, 2018

derhuerst changed the title ~~The endpoint is broken?~~ implement a fallback endpoint Jun 28, 2018

derhuerst reopened this Jun 28, 2018

deg0nz mentioned this issue Jul 12, 2018

Departure board in vue an electron joecorcoran/abfahrten.berlin#11

Closed

derhuerst changed the title ~~get the public endpoint running again~~ get the public endpoint running again (use BVG endpoint for now) Aug 24, 2018

derhuerst added the upstream label Aug 24, 2018

This was referenced Aug 31, 2018

Bad Gateway at request derhuerst/hafas-rest-api-client#11

Closed

Endpoints down returning 502 error #34

Closed

derhuerst mentioned this issue Sep 14, 2018

directionStationId not working deg0nz/MMM-PublicTransportBerlin#39

Closed

derhuerst mentioned this issue Sep 29, 2018

run bot on a new server derhuerst/vbb-telegram#7

Open

derhuerst closed this as completed Oct 28, 2019

derhuerst mentioned this issue Feb 4, 2020

Unstable API noxer/departures#13

Closed

derhuerst mentioned this issue Mar 23, 2020

sustainable server setup: collaborate on hosting public-transport/transport.rest#2

Closed

derhuerst mentioned this issue Dec 15, 2020

Question regarding license of provided data public-transport/transport.rest#4

Closed

derhuerst mentioned this issue Dec 13, 2021

find replacement for defunct oebb endpoint public-transport/hafas-client#195

Open

derhuerst added the ops label Apr 3, 2024

get the public endpoint running again (use BVG endpoint for now) #29

get the public endpoint running again (use BVG endpoint for now) #29

Comments

berlincount commented Jun 19, 2018 • edited by derhuerst Loading

joecorcoran commented Jun 19, 2018

joecorcoran commented Jun 20, 2018

derhuerst commented Jun 20, 2018

rejas commented Jun 23, 2018

derhuerst commented Jun 25, 2018

derhuerst commented Jun 28, 2018

juliuste commented Jun 28, 2018 • edited Loading

juliuste commented Jun 28, 2018

derhuerst commented Jun 28, 2018

joecorcoran commented Jun 28, 2018

derhuerst commented Jun 28, 2018 • edited Loading

derhuerst commented Jun 28, 2018

juliuste commented Jun 28, 2018 • edited Loading

rejas commented Jun 28, 2018

derhuerst commented Jun 28, 2018

derhuerst commented Jun 28, 2018 • edited Loading

derhuerst commented Jun 28, 2018

joecorcoran commented Jun 29, 2018

derhuerst commented Jun 30, 2018

juliuste commented Jul 12, 2018

derhuerst commented Jul 12, 2018

derhuerst commented Jul 12, 2018

derhuerst commented Jul 12, 2018

michaelcheers commented Nov 30, 2018

derhuerst commented Nov 30, 2018

michaelcheers commented Nov 30, 2018

derhuerst commented Nov 30, 2018 • edited Loading

deg0nz commented Apr 21, 2019

derhuerst commented Apr 24, 2019

derhuerst commented Apr 24, 2019

deg0nz commented Apr 24, 2019

jonathan-reisdorf commented May 1, 2019

derhuerst commented May 1, 2019 • edited Loading

jonathan-reisdorf commented May 4, 2019

derhuerst commented May 7, 2019

jonathan-reisdorf commented May 8, 2019

derhuerst commented May 8, 2019

k-nut commented May 13, 2019

derhuerst commented Oct 28, 2019

derhuerst commented Mar 23, 2020 • edited Loading

derhuerst commented May 9, 2020

berlincount commented Jun 19, 2018 •

edited by derhuerst

Loading

juliuste commented Jun 28, 2018 •

edited

Loading

derhuerst commented Jun 28, 2018 •

edited

Loading

juliuste commented Jun 28, 2018 •

edited

Loading

derhuerst commented Jun 28, 2018 •

edited

Loading

derhuerst commented Nov 30, 2018 •

edited

Loading

derhuerst commented May 1, 2019 •

edited

Loading

derhuerst commented Mar 23, 2020 •

edited

Loading