Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a MQTT GTFS-RT trip update updater #2516

Open
wants to merge 1 commit into
base: dev-1.x
from

Conversation

@hannesj
Copy link
Contributor

commented Dec 7, 2017

The GTFS-RT standard doesn't limit the transport method for the protobufs for the realtime messages. We are in the process of implementing a MQTT-based transport and have created the required GTFS-RT updater for that based on the Apache Paho library.

LOG.debug("Connected");
client.subscribe(topic, qos);
} catch (MqttException e) {
LOG.warn("Could not subsctibe to: " + topic);

This comment has been minimized.

Copy link
@haphut

haphut Dec 7, 2017

"subscribe"

@hannesj hannesj force-pushed the HSLdevcom:mqtt-gtfs-rt branch from a306434 to bc2a645 Dec 7, 2017

@barbeau

This comment has been minimized.

Copy link
Collaborator

commented Dec 7, 2017

@hannesj Somewhat off topic (apologies) - would you be willing to share the URL of your MQTT feed when available so we can add support for this transport method in the GTFS-realtime validator?

I've opened an issue here for topic if you'd like to continue the discussion there - CUTR-at-USF/gtfs-realtime-validator#317.

@hannesj

This comment has been minimized.

Copy link
Contributor Author

commented Dec 7, 2017

@barbeau sure! Just a correction, the GTFS-RT feed is still encoded as protobufs, they are just transported as MQTT messages, instead of polling from a server or using websocket as the transport.

@barbeau

This comment has been minimized.

Copy link
Collaborator

commented Dec 7, 2017

@hannesj Ah, gotcha, that makes sense (you can tell I haven't worked with MQTT before ;) ). But the MQTT messages contain full messages with header.incrementality: FULL_DATASET, right? Not header.incrementality: DIFFERENTIAL?

@barslett

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2017

I told some of you today about our ITxPT standardisation for messages between vehicles and backend. the message transport is also here based on MQTT and protobuf. Let's see if we should use MQTT/protobuf for our SIRI updaters as well, @t2gran

@hannesj

This comment has been minimized.

Copy link
Contributor Author

commented Dec 7, 2017

@barbeau we would be differential messages in order to better preserve the history in OTP. Also the size difference would be quite considerable, as we push updated predictions with up to 1/s frequency

@barbeau

This comment has been minimized.

Copy link
Collaborator

commented Dec 8, 2017

@hannesj Interesting - GTFS-rt says the following about differential messages:

currently, this mode is unsupported and behavior is unspecified for feeds that use this mode. There are discussions on the GTFS Realtime mailing list around fully specifying the behavior of DIFFERENTIAL mode and the documentation will be updated when those discussions are finalized.

Just curious if you have additional design documentation to specify some of this behavior? If so and it works well in practice it may be worth trying to get some of this officially adopted into the spec. I'm looking at this from the GTFS-realtime validation point of view too. Right now there are no validation rules for differential feeds because of the above. If we're seeing these in production in the wild it would be good to try and formalize some of the common behavior being used.

@abyrd

This comment has been minimized.

Copy link
Member

commented Dec 8, 2017

The specification may say that it's unsupported, but that text has been around for a long time, and significant practical work has been done toward demonstrating how this mode could/should work. As part of a Dutch transportation ministry project in 2013, OpenTripPlanner was modified to consume a continuous stream of GTFS-RT messages in differential mode over websockets. The transport protocol was chosen over our own initial choice (ZeroMQ) to conform to existing practice in One Bus Away and following advice from other developers who had experimented with the same thing.

The Plannerstack platform is providing OTP-based routing on a complete open data set for the Netherlands, with updates at least once a minute from every vehicle in the country. This is all being done with differential GTFS-RT over Websockets interfacing with One Bus Away modules.

We have well-established semantics and implementations for these ideas that have been around for several years. If there is still ambiguity about this we can certainly document the way this existing system interprets the spec.

@abyrd

This comment has been minimized.

Copy link
Member

commented Dec 8, 2017

It's actually pretty simple. To summarize, our differential GTFS-RT uses trip-level granularity, and the effects of successive messages about the same trip do not accumulate.

An update about a trip is always cleanly applied to the base trip from the original static feed, with no consideration for any previously received message. The differential effect comes from accumulating changes to different trips across the whole data set.

@abyrd

This comment has been minimized.

Copy link
Member

commented Dec 8, 2017

If people in Norway and Finland are using these same rules (which is probably the case if we're all using OTP just with different transport layers) then we can create a joint document promoting this interpretation of differential mode.

@barbeau

This comment has been minimized.

Copy link
Collaborator

commented Dec 8, 2017

Thanks @abyrd! Yes, I'd love to get this behavior documented and adopted into the spec. I've opened an issue at google/transit#84 where we can start tracking documentation related to differential messages, with the goal of creating a proposal to get the documented behavior into the GTFS-realtime spec. Anyone, please feel free to comment more there.

@hannesj hannesj force-pushed the HSLdevcom:mqtt-gtfs-rt branch from bc2a645 to 699e88f Dec 9, 2017

@abyrd abyrd requested a review from opentripplanner/plc Sep 11, 2018

@abyrd
Copy link
Member

left a comment

This PR itself looks good to me, the main remaining action is to make this widely-accepted definition of differential GTFS-RT official, and @barbeau has already created an issue for that elsewhere.

@barbeau

This comment has been minimized.

Copy link
Collaborator

commented Sep 11, 2018

FYI - GTFS-rt issue for formalizing header.incrementality: DIFFERENTIAL is google/transit#84. I'll add a comment there referencing this PR.

@hannesj

This comment has been minimized.

Copy link
Contributor Author

commented Sep 11, 2018

If anyone wants to test the implementation using HSL real-time data, it is available with the following updater configuration. This is in conjunction with this GTFS file

    {
      "type": "MQTT-gtfs-rt-updater",
      "url": "tcp://mqtt.hsl.fi",
      "topic": "gtfsrt/v1/fi/hsl/tu",
      "feedId": "HSL",
      "fuzzyTripMatching": true
    }
@abyrd

This comment has been minimized.

Copy link
Member

commented Sep 11, 2018

Thanks for the config @hannesj, we should indeed test this out before merging just as a matter of good procedure.

@barbeau

This comment has been minimized.

Copy link
Collaborator

commented Sep 11, 2018

Thanks @hannesj! Could you share URLs to your feed (and any documentation) as well as links to the user-facing deployment of OTP that consumes this feed in the GTFS-realtime issue google/transit#84?

@hannesj

This comment has been minimized.

Copy link
Contributor Author

commented Sep 11, 2018

The GTFS-RT-feed is published on mqtt.hsl.fi, using the topic gtfsrt/v1/hsl/tu, as specified by the config. The architecture and service documentation unfortunately still refer to the old APIs, but will be updated soon. The public facing ui is located at https://www.reittiopas.fi/, and the OTP API is available at https://api.digitransit.fi/routing/v1/routers/hsl/.

@abyrd

This comment has been minimized.

Copy link
Member

commented Jan 23, 2019

@barbeau did you ever get a chance to test this PR? I'm going to change the base to the new dev-1.x, maybe we can get this merged sometime soon.

@abyrd abyrd changed the base branch from master to dev-1.x Jan 23, 2019

@abyrd

This comment has been minimized.

Copy link
Member

commented Jan 23, 2019

Oh, looks like it no longer compiles when updated to dev-1.x. But it might be a quick fix.

@barbeau

This comment has been minimized.

Copy link
Collaborator

commented Jan 23, 2019

@abyrd said:

@barbeau did you ever get a chance to test this PR? I'm going to change the base to the new dev-1.x, maybe we can get this merged sometime soon.

Unfortunately not yet. DIFFERENTIAL GTFS-rt has current taken a backseat compared to other priorities at the moment, but it's definitely something I'd like to return to. Yes, any changes that would get the PR back into a runnable state would be great!

@hannesj hannesj force-pushed the HSLdevcom:mqtt-gtfs-rt branch from 301b3cf to 695de81 Feb 10, 2019

@hannesj

This comment has been minimized.

Copy link
Contributor Author

commented Feb 10, 2019

@abyrd @barbeau This is now updated similarly how it was done in 544e1b1

Also the URL for the HSL feed has changed to:

    {
      "id": "hsl-trip-updates",
      "type": "MQTT-gtfs-rt-updater",
      "url": "tcp://mqtt.cinfra.fi",
      "topic": "gtfsrt/v1/fi/hsl/tu",
      "feedId": "HSL",
      "fuzzyTripMatching": true
    },
* <pre>
* bessersmith.type = mqtt-gtfs-rt-updater
* bessersmith.feedId = hsl
* bessersmith.url = ssl://mqtt.hsl.fi:443

This comment has been minimized.

Copy link
@optionsome

optionsome Feb 12, 2019

Member

@hannesj pointed out that the URL is currently different. However, this old URL given here should be production ready soon again. If I'm not mistaken, the new port for MQTT will be 8883 (as recently updated in the documentation here)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.