Additional OpenStreetMap, ESRI ShapeFile and Geodatabase, and GTFS (General Transit Feed Service) importing

[d-wasserman]

#I read in the documentation that this is already implemented on the C++ side? I was curious what the most current information on shapefile/geodatabase import is? A lot of analysis requires custom edge weights based on prior analysis on the network. I was reviewing NetworkX implementation of the graph network importer, but it is pure Python implementation that has been shown to be slower relative to IGraph/graph-tool.
Questions about the implementation:
Does the C++ shapefile import functionality allow import of networks that have connections based on vertices along with line end points? Or just end points?
What are the dependencies on the C++ implementation on import?
Depending on where this was on your priority queue I was thinking about implementing something myself. Any thoughts or updates on this would be great.

[fscottfoti]

Good question. Yeah, OSRM has a lot of great new import functionality. Unfortunately we're tied to a custom version where the author of OSRM wrote some custom functionality for OSRM which never made it into the trunk. I once looked at updating to the latest OSRM and decided it would be very challenging, nigh impossible. I can see the value of a shapefile to network conversion tool usable by Pandana and for lots of other reasons too I imagine. NetworkX is maybe the best available right now?

[waddell]

Regarding performance on a variety of network queries once a network is imported, yes, I've seen benchmarking showing igraph to be faster than NetworkX. We have not run comparable benchmarks, but I suspect the pandana queries are at least as fast as the igraph benchmarks based on our use with large networks, at least for the kinds of accessibility queries we optimized pandana for (walkscore type aggregations over a network). There is always room for improvement of course. As for network importing from shapefiles, we have not implemented that yet, and you are welcome to contribute it. We do have OSM loaders in pandana already, and will soon release an urbanaccess library that provides a lot of functionality for creating transit networks from GTFS data, and fusing this with OSM data.

[d-wasserman]

@fscottfoti That is unfortunate to hear. Might have saved me a step.
NetworkX is the only one I can see that has directly shapefile import, but it is pretty simple and requires a hidden dependency that can be a little annoying to integrate into the conda stack (but is possible and done often). My plan was just to make an Arcpy/Other library script that meets my desired criteria (arbitrary number of weight fields/vertices linkage) and just have it load those results into a node/edge/weight table to match the API. I was going to try to keep it general so an open source distribution could use most of the code as well (Shapely/OSGeo4W). Should I just plan on an intermediate converter and link it back here?

@waddell I would be interested in any final benchmarking done, but based on what I have read I think Pandana is comparable to i-graph. My concern with OSM is it is really good for aggregate regional analysis like you are doing usually with this package, but I can't really inspect the topology of the graph in an interactive manner. There are some types of detailed analysis that might have features that OSM does not have (planned overpasses being a good example) that need to have their network topology validated to get a good graph.

[waddell]

You might want to look at the osmnx project for some additional network importing and cleaning processes, as well as graph analytics. It leverages NetworkX. https://github.com/gboeing/osmnx

[d-wasserman]

@waddell That is a very useful library. I will have to check it out later. The dependency concern I mentioned before is still an issue for this library but this is clearly a tool I am looking for other uses.

[semcogli]

Let me chime in here regarding shape file importing. We do have a tool to convert shape files to network graphs based on NetworkX. And we store the converted nodes, edges and weights in HDF so the future loading and accessibility calculation can be done very fast.

The tool is pretty simple and you can access it on our Github repo.
https://github.com/SEMCOG/semcog_urbansim/blob/semcog/notebooks/shape2hdf5_w_local.ipynb

[d-wasserman]

Thanks for the notebook link. I am starting to lean towards using networkX as potential solution after reviewing the suggested tools, and you have a good conversion example to follow.

[sablanchard]

@Holisticnature we have just released alpha versions of a new OSM network tool called OSMnet: https://github.com/UDST/osmnet and a new GTFS network tool call UrbanAccess: https://github.com/UDST/urbanaccess that all produce networks for use with Pandana. We have not yet created a shapefile loader, however anyone who wishes to start a contribution for Pandana is welcome. I will close this issue as it seems you have a workaround for the lack of a shapefile loader at the moment. If you would like to still see a shapefile loader please open a new issue specifically for that.

[sablanchard]

Link to related comment: #36 (comment)