Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.

General

This system will read real-time data from various sources (for example, Pubtrans database regarding departures and arrivals) and convert those to GTFS real-time messages using a pipeline created with Apache Pulsar. The final step will publish the messages to different locations (such as MQTT brokers and blob storage).

General usage pattern is to build Docker images and then run them with docker-compose. Services are separated to different GitHub repositories, each containing the source code and the Dockerfile.

/bin-folder contains scripts to launch Docker images for Pulsar and Redis which are requirements for some of the services.

Requirements

Overall system requirements for running the system are:

  • Docker
  • Redis
  • Pulsar
  • Connection to a Pubtrans SQL Server database
  • Connection to an MQTT broker

System Architecture & Components

Transitdata

Alt text

Transitdata input

  • mqtt.hsl.fi vehicle positions in HFP format (all vehicles)
  • hsl-mqtt-lab-a.cinfra.fi estimates for stop time (metros)
  • Pubtrans ROI: estimates for stop time (bus and trams)
  • Pubtrans DOI: static data (schedule, stops, routes)
  • api.digitransit.fi/realtime/service-alerts/v1: not used
  • OMM DB: service alerts (cancellations, disruptions)
  • raildigitraffic2gtfsrt: train stop estimates (to be replaced by a service connected to ratadigitraffic)
  • sm5.rt.hsl.fi: EKE message from SM5 trains

Transitdata output

  • MQTT Broker cmqttdv.cinfra.fi: vehicle position in GTFS-RT format (HSL displays at stops)
  • Azure Blob storage: used for publishing vehicle position, trip updates and service alerts in GTFS-RT format (for Google Maps and 3rd-party applications) and for archiving messages (HFP and EKE) in CSV files
  • MQTT Broker pred.rt.hsl.fi -> Reittiopas.fi: stop estimates in GTFS-RT format
    • Note: this MQTT broker is intended to be used by HSL systems only. Its functionality can be changed without a notice and there is no guarantee that it will work for third-party applications.
  • Graylog server: logs from all the microservices.

Transitlog

Alt text

Components are stored in their own Github Repositories:

Common dependencies

Transitdata components

Sources
Processors
Publishers
Unused components
  • transitdata-hslalert-source reads trip cancellations from HSL public HTML API and generates TripUpdate cancellations. Not in use anymore.

Transitlog HFP components

Versioning

All the components in this project use semver, but the output conforms always to the GTFS Realtime standard. Some vendor-specific extensions might be added, which require incrementing the major version. Otherwise, new features should only increment the minor version, but some exceptions might arise. TripUpdate and ServiceAlert APIs are versioned independently.

Implementation notes

Pulsar seems to cause approximately 5ms of latency for each message, which is consistent with their promise. The latency is not a problem in itself, and is well within acceptable bounds. However, the latency means that a single-threaded consumer-producer loop can only process 200 messages per second.

About

transitdata publishes GTFS Realtime interfaces

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages