Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support for GTFS-RT and other real-time transit data #77

Open
drewda opened this issue Aug 1, 2016 · 4 comments
Assignees
Labels

Comments

@drewda
Copy link
Member

@drewda drewda commented Aug 1, 2016

Within Mapzen and with a number of potential outside collaborators, we've chatted on and off about adding support to Transitland for real-time data. We've also run some internal experiments of connecting the Transitland stack with GTFS-RT and similar formats.

Creating this ticket as a placeholder for future additions to the Datastore, the Feed Registry, and dependencies that will need to be updated or created. When a wide enough range of users are satisfied that Transitland can provide a solid foundation of static schedule data (and stable Onestop IDs), we'll be well position to work together to aggregate real-time from authoritative sources and offer it as an overlay.

A specific example of where we'll need a "solid foundation" before we can aggregate and combine static and real-time data: stable trip identifiers ( transitland/transitland-datastore#713 )

Timeframe: Not urgent.

//cc @doublestranded @barbeau among others

@drewda

This comment has been minimized.

Copy link
Member Author

@drewda drewda commented May 8, 2017

In order to start cataloging GTFS-RT feeds within Transitland:

  • turn the existing single Feed model into two models using single-table inheritance:
    • FeedGtfsStatic model (with feed_format="gtfs") will have the existing set of attributes, validation, and behavior
    • FeedGtfsRealtime model (with feed_format="gtfs-rt") will have additional attributes (to support URLs for alerts, trip updates, and vehicle position), different validation, and no association to FeedVersions
    • the base Feed model can continue to have the association with Operator records, since both static and real-time feeds are associated with operators/agencies
  • update API serializers and controllers to allow both static and real-time feed records, with the option to filter for one or the other
  • add in some GTFS-RT feeds from this spreadsheet created by @barbeau: https://docs.google.com/spreadsheets/d/1EACeuYP3Yirm2OwBnFAHlFaNlZyX6PODF9DZo8s5oEI/edit#gid=0
  • Feed Registry: should list out GTFS-RT feeds on operator detail pages (for example: https://transit.land/feed-registry/operators/o-dr5r-nyct) and show links off to alerts, trip updates, and vehicle position URLs
  • write a blog post and some documentation on how to use Transitland Dispatcher to create a changeset that adds in a new GTFS-RT feed

Note that for now it will only be possible to add and edit FeedGtfsRealtime models using a changeset sent in to the API -- we won't be expanding the Feed Registry "add a feed" flow to support GTFS-RT.

@barbeau

This comment has been minimized.

Copy link

@barbeau barbeau commented May 8, 2017

I also recently found this list of GTFS-rt feeds in Transitfeeds.com:
http://transitfeeds.com/search?q=gtfsrt

I've also been cataloging some feeds by size of protobuf file in this issue to ensure our gtfs-rt-validator can process large feeds:
CUTR-at-USF/gtfs-realtime-validator#123

@nvkelso

This comment has been minimized.

Copy link

@nvkelso nvkelso commented May 12, 2017

As first pass it would be nice to know if a realtime feed variant was available for a generic GTFS feed :)

See also:

@drewda drewda referenced this issue Oct 5, 2017
@barbeau

This comment has been minimized.

Copy link

@barbeau barbeau commented Oct 10, 2017

We now have a very practical use case for retrieving a list of GTFS-realtime feeds - bulk quality analysis of GTFS-realtime feeds.

We got tired of doing this manually, so we created the transit-feed-quality-calculator, a project that:

  1. Fetches the URLs for all known GTFS-realtime feeds and corresponding GTFS data from the TransitFeeds.com GetFeeds API and downloads them each to a subdirectory
  2. Runs the gtfs-realtime-validator Batch Processor on each of the subdirectories
  3. Produces summary statistics and graphs (In progress - see Issue #2 and WIP Pull Request #3))

See example WIP output here.

It's written to be fairly agnostic to what API the GTFS and GTFS-realtime feed URLs come from. We're currently using TransitFeeds.com because it already has a directory of GTFS-realtime feeds, but I'd like to add Transitland as an option too.

Adding Transitland as an option to our project would mean mirroring the TransitFeedsDownloader class in a new TransitlandDownloader class.

TransitFeedsDownloader does the following:

  1. Gets a list of GTFS-realtime feeds from TransitFeeds.com, which each have a Location ID (defined by TransitFeeds.com), creates a subdirectory for that Location ID, and downloads the GTFS-realtime protocol buffer files for each URL to that directory
  2. Gets a list of GTFS feeds, and downloads each GTFS zip file if we have a corresponding GTFS-realtime feed for that same Location ID

So, we'd need to following a similar process for Transitland:

  1. Get a list of GTFS-realtime feed URLs, each of which has some identifier that links it to the corresponding GTFS data
  2. Get a list of GTFS feeds, each of which has some identifier that links it to the corresponding GTFS-realtime feeds

This could happen in a single API call, or several - the main requirement is that there is a link between the GTFS and GTFS-realtime feeds, and that we can easily download GTFS data only for agencies that have GTFS-realtime feeds. Note that GTFS-realtime feeds typically have 2-3 URLs, one for each TripUpdate, VehiclePositions, and ServiceAlerts.

I opened an issue for adding Transitland to our tool here - CUTR-at-USF/transit-feed-quality-calculator#4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.