It's become harder to reproduce gtfs examples

[mpadge]

It used to be quite easy to reproduce example GTFS analyses by just including a download.file or whatever line at the outset to grab the feed and demonstrate the code. That is now more difficult because the only general feed providers left in the game (early 2022) are transitfeeds and transit.land. Both of these services rely on private amazon aws s3 buckets, with transit.land being definitely private (and also now pretty definitely a commercial service, a move itself worthy of another discussion sometime ...), and transitfeeds being semi-open but still difficult to access openly. This makes it nigh on impossible to do a simply download.file to generate reproducible examples. It'd be good to establish general principles of how we as a community might continue to create reproducible workflows with GTFS feeds. Ideas anyone? Explicit pings for input @dhersz @rafapereirabr @polettif @tbuckl

[polettif]

To be quite honest, I've never used transit.land or transitfeed as the go-to spot for feeds, especially in the last few years when it was hard to figure out how they operate. I've always seen them as a registry to find actual feed providers for a region and move on from there.

However, I don't see the issue with downloads? transit.land still provides permalinks to external URL where the feeds are hosted by agencies or feed providers. Or at least that's the case for the few feeds I've checked. These links can then be used in reproducible examples? I guess I'm missing something.

[mpadge]

Thanks @polettif - i was thinking more at a "higher level" for anybody else wishing to use our tools for reproducible GTFS stuff. Like a set of guidelines for how best to do that, and those should arguably extend out of general feed "provider" services. Among the many problems there are that feeds on those platforms don't always come with external links to actual providers, and the only required links are to internal s3 buckets. No idea on statistics, but when i semi-randomly look, I mostly find either no external URLs, or ones that resolve to intermediate API Endpoints or s3 buckets or other stuff than can't simply be used as download points.

If we were to write a "Best practices for working with GTFS Data", it would have to start with a section along the lines of "Getting GTFS data", and that realistically would have to refer to at least those two "services" as primary collectors of feed (meta-)data. We'd then have to recommend how to get actual feeds, which is easy in a dumb click-and-wait way, but is now really difficult to automate. The increasing difficulty of automating that is what got me thinking here.

[polettif]

Like a set of guidelines for how best to do that, and those should arguably extend out of general feed "provider" services.

That's a very good point but this covers quite a bigger scope than R?

Among the many problems there are that feeds on those platforms don't always come with external links to actual providers, and the only required links are to internal s3 buckets.

I guess I'm still missing something, because looking through transit.land atlas there are a lot of external URLs to zip files? Now, I don't know how many of those URLs actually work, I did get some access denied or no response. Most of them did seem to work though.

# working in cloned transit.land-atlas repo
lf = list.files("feeds", full.names = T)

urls_list = lapply(lf, \(jsonfile) {
    json = jsonlite::read_json(jsonfile)

    static_current_urls = lapply(json$feeds, \(f) {
        if(f$spec == "gtfs-rt") {
            return(f$urls$realtime_trip_updates)
        } else {
            return(f$urls$static_current)
        }
    })
    unlist(static_current_urls)
})
urls = unlist(urls_list)

file_extenstions = tools::file_ext(urls)
table(file_extenstions)
#> file_extenstions
#>        ashx  aspx   bin   dat  gtfs    pb   pbf phtml proto   zip   ZIP 
#>   207     1     7     8     1    15    37     1     1     2  1135     3

However, these URLs always link to the current file which of course is bad for reproducibility. I think I see where you're coming from... But other than getting involved with transit.land or urging feed providers to supply timestamped feeds, I don't see a (realistic) solution. For my use cases, Swiss GTFS feeds are already archived/timestamped. Happy to read other input but I guess I can't help much further, sorry.

[mpadge]

Thanks for the insights @polettif - i just thought it was something worth having a public record of. We'll see where this goes ... You're definitely right that the scope is more general than just R. My one thought is the need to at least write an interface to the v2 REST API of transit.land, and maybe put that in gtfstools. If i find time, I'll have a play and see what i can come up with.

[dhersz]

Great discussion! Writing on best practices/workflows to work with GTFS data in R is something that we have recently started to look at in our project, and a section on how to get GTFS data is definitely important.

To be honest, however, I don't have much experience with neither of these services because we usually only deal with GTFS data that we either get directly from official data repositories hosted by agencies or from direct contact with them (e.g. via email). Writing an interface to the their APIs seems to be the way to go, and I'd be happy to accept an PR on this direction in gtfstools (though I think that maybe in the long run it'd be better to create separate packages for these APIs, since that would introduce a whole new set of dependencies and testing infrastructure - httr and http-testing-packages come to my mind).

I'd be happy to procrastinate on contribute to such a package/function if you want to to kickstart something @mpadge.

[rafapereirabr]

Hi all. Just my quick 2 cents here.

If the peoblem is creating reproducible gtfs examples, I believe the best approach would be to have two or three sample GTFS feeds in the package. This is what we currently do, but perhaps we need more diverse gtfs feeds with different characteristics (e.g frequency-base vs time table based, etc) ?

Having said that, it would be great to have something like a download_gtfs() function in gtfstools. I agree downloading GTFS data can be tricky sometimes, specially as some websites that host GTFS feeds become commercial-oriented. I just don't think this would solve the problem of creating reproducible GTFS examples because of the many points you all raised in the discussion above