Initial draft for review of NaPTAN point locations

[anisotropi4]

The proposed code downloads National Passenger Transport Access Node (NaPTAN) data from the current DfT Universal Resource Identifier, filters on heavy and light rail, and ferry locations, adds the National Rail CRS (Computer Reservation System) codes for heavy rail stations and outputs the as GeoJSON and GeoPackage in EPSG:32630 projection, as well as the National Rail CRScode lookup data in a slightly odd CSV file format.

Any thoughts and comments welcome

[anisotropi4]

The current build appears to fail but I'm not sure whether this is due to this push or something else

[Robinlovelace]

At around 1 MB the stations.geojson and stations.gpkg objects are a bit big. We want to keep the repo as small as possible (currently around 2-3MB I think) to make it easy to download. I suggest we either only release these datasets in the releases section where we have saved other dataset https://github.com/geocompr/py/releases or release the data there and add minimal (e.g. 100 KB max) in the commit history. Apologies but would you mind creating a new PR from a different branch so this big files do not go in the commit history? Overall big 👍 from me though and we can write some nice text around this outstanding worked example.

[Robinlovelace]

Regarding the repo size, you may want to re-clone this repo and create a create a new branch e.g. along the lines of

gh repo clone geocompr/py
cd py
git checkout -b naptan
git remote add a1 https://github.com/anisotropi4/py.git
# make + commit changes
git push naptan a1

Not sure if that will actually work, hope so!

[anisotropi4]

Thanks for the feedback and clear steer on data.

I'll wait for any more comments and then drop the branch and resubmit this with cut-down size. This will probably involve a reduced geographic scope and sorting out some of the daft levels of precision. The GeoJSON is at something like 11 significant figures.

[anisotropi4]

As I won't get to look at this until the weekend I will leave it open for comments until Friday. I will then fix the PR based on Yorkshire, and look to host the full data set somewhere else

[Robinlovelace]

Any feedback on this @michaeldorman, @anitagraser, @Nowosad ? Feedback on the great worked example in the Python script here https://github.com/geocompr/py/pull/57/files is what I think @anisotropi4 is after, will be glad to get your thoughts on it if you get a chance. Many thanks again Will for this valuable contribution.

[michaeldorman]

The code works for me, and looks great! Thank you very much @anisotropi4 for the contribution, and @Robinlovelace !

One suggestion is that in the final mapping part might be better to focus on pure Python methods, because it's the book focus and because it would be beyond the scope to explain the HTML/JavaScript code in the text. Perhaps there can be few examples of .plot to show the data in static maps, and then .explore to create an interactive map?

[anisotropi4]

The code works for me, and looks great! Thank you very much @anisotropi4 for the contribution, and @Robinlovelace !

One suggestion is that in the final mapping part might be better to focus on pure Python methods, because it's the book focus and because it would be beyond the scope to explain the HTML/JavaScript code in the text. Perhaps there can be few examples of .plot to show the data in static maps, and then .explore to create an interactive map?

There may be another way round this. Although it isn't something I have used it would appear that folium is a way of generating Leaflet.js interactive html files. I just need to work out how to replicate the existing interactive index.html using the library.

[michaeldorman]

The code works for me, and looks great! Thank you very much @anisotropi4 for the contribution, and @Robinlovelace !
One suggestion is that in the final mapping part might be better to focus on pure Python methods, because it's the book focus and because it would be beyond the scope to explain the HTML/JavaScript code in the text. Perhaps there can be few examples of .plot to show the data in static maps, and then .explore to create an interactive map?

There may be another way round this. Although it isn't something I have used it would appear that folium is a way of generating Leaflet.js interactive html files. I just need to work out how to replicate the existing interactive index.html using the library.

Sure, excellent idea!

I suggest perhaps to also demonstrate some of the extra capabilities of folium, to justify using it instead of .explore. I haven't used folium before so don't have a specific recommendation at the moment, but according to the docs seems like tile layer selection using radio buttons is something that .explore doesn't provide, so perhaps that could be an option.

[anisotropi4]

One suggestion is that in the final mapping part might be better to focus on pure Python methods, because it's the book focus and because it would be beyond the scope to explain the HTML/JavaScript code in the text. Perhaps there can be few examples of .plot to show the data in static maps, and then .explore to create an interactive map?

There may be another way round this. Although it isn't something I have used it would appear that folium is a way of generating Leaflet.js interactive html files. I just need to work out how to replicate the existing interactive index.html using the library.

Sure, excellent idea!

I suggest perhaps to also demonstrate some of the extra capabilities of folium, to justify using it instead of .explore. I haven't used folium before so don't have a specific recommendation at the moment, but according to the docs seems like tile layer selection using radio buttons is something that .explore doesn't provide, so perhaps that could be an option.

My issue is I don't know what .explore does, I haven't ever used it myself and a quick search of "python .explore" didn't really help

FWIW my dev environment is typically linux command line witth emacs, a python -m http.server and chromium to hack python and JavaScript/html. Using folium to generate Leaflet.js and html is closest to what I am used to. I'd not advocate this as a reasonable or sane method of working however...

[michaeldorman]

One suggestion is that in the final mapping part might be better to focus on pure Python methods, because it's the book focus and because it would be beyond the scope to explain the HTML/JavaScript code in the text. Perhaps there can be few examples of .plot to show the data in static maps, and then .explore to create an interactive map?

There may be another way round this. Although it isn't something I have used it would appear that folium is a way of generating Leaflet.js interactive html files. I just need to work out how to replicate the existing interactive index.html using the library.

Sure, excellent idea!
I suggest perhaps to also demonstrate some of the extra capabilities of folium, to justify using it instead of .explore. I haven't used folium before so don't have a specific recommendation at the moment, but according to the docs seems like tile layer selection using radio buttons is something that .explore doesn't provide, so perhaps that could be an option.

My issue is I don't know what .explore does, I haven't ever used it myself and a quick search of "python .explore" didn't really help

FWIW my dev environment is typically linux command line witth emacs, a python -m http.server and chromium to hack python and JavaScript/html. Using folium to generate Leaflet.js and html is closest to what I am used to. I'd not advocate this as a reasonable or sane method of working however...

.explore is a wrapper around folium, intended for GeoDataFrame input:
https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.explore.html

[anisotropi4]

Following on from the discussion this week, please find an update which is based on Yorkshire rather than GB with precision rounded to sensible values, which significantly reduces the data size. With an interactive map based on folium. This is still beta as it still needs description and a narrative placing round the content, but thoughts and comments welcome.

If acceptable I will look to provide an interactive rail-line map patch for the same geography based on OSMnx and then an interactive population density vectortile view as part of this chapter.

[Robinlovelace]

One question: how did you remove the big files from the commit history @anisotropi4 ? I assumed they would be there for eternity! In any case as shown above I've approved it. Would like to get feedback from Michael, Anita and Jakub before merging this. One thing to note: code/chapters/09-mapping.py is an ephemeral file and will get overwritten each time we run convert.sh either locally on on Actions. Worth moving your example to somewhere more like code/transport-example.py?

[anisotropi4]

One question: how did you remove the big files from the commit history @anisotropi4 ? I assumed they would be there for eternity!
The big files are dynamically generated by the script so if you make a "big file" smaller the automation just seems to clobber it. In this case I did what I said I would:

1. Restrict the mapping area to Yorkshire. There is now a very small boundary GeoJSON file "data/yorkshire.json". Anything that falls outside this is dropped as follows:

YORKSHIRE = gpd.read_file('data/yorkshire.json').iloc[0, 0]
IDX = STATIONS.within(YORKSHIRE)
STATIONS = STATIONS[IDX]

2. Rounded to three d.p. for WGS84 coordinates and to whole numbers for the EPSG:32630 projection in $m$ using some coding slight-of-hand:

2a. Define a function constructor:

def _set_precision(precision=0):
  """returns function that rounds a geometry to a given precision""" 
  from functools import partial
  from shapely.ops import transform

  def _precision(x, y, z=None):
    return tuple([round(i, precision) for i in [x, y, z] if i])
  return partial(transform, _precision)

2b. Create a function that rounds to the required precision:

_precision = _set_precision(0)

2c. Then .apply it to the appropriate geometry column

OUTPUT = STATIONS.copy() 
CRS = 'EPSG:32630'
OUTPUT['geometry'] = OUTPUT['geometry'].to_crs(CRS).apply(_precision)

One thing to note: code/chapters/09-mapping.py is an ephemeral file and will get overwritten each time we run convert.sh either locally on on Actions. Worth moving your example to somewhere more like code/transport-example.py?

I've created the stations-example.py file and will create a track-example.py working as a further example for LineString geometries and OSMnx once this PR is cleared.

Census population density and Polygon geometry example will appear later if required.

[michaeldorman]

Look great! I suggest converting to .ipynb to be in agreement with the other content, and also it will be easier to add text, execute the code, and view the output.

[Robinlovelace]

Agree re. converting to .ipynb but suggest we can do that post merge. Also I suggest converting to .ipynb via .qmd for consistency. Ultimately at least some of this code deserves to be in the final book so we can incorporate it, into a later section of the visualisation chapter, currently chapter 9, is my current thinking.

[michaeldorman]

Agree re. converting to .ipynb but suggest we can do that post merge. Also I suggest converting to .ipynb via .qmd for consistency. Ultimately at least some of this code deserves to be in the final book so we can incorporate it, into a later section of the visualisation chapter, currently chapter 9, is my current thinking.

Agree, sounds good!

[anisotropi4]

Bump.

As I understand that this is waiting on formal approval from @michaeldorman, @Nowosad and @anitagraser I would ask that you either reject or approve this PR.

I ask as I believe I unable to raise another PR against the repository and unless I code branch and then enter branch dependency hell, I am stuck until this is approved.

Thoughts or advice welcome.

[Robinlovelace]

Thanks for the bump and agree we should move on this. I think Michael has already signaled 👍 on this with this comment:

Agree, sounds good!

Will await comment from Jakub and Anita, happy for this to be merged? Please signal either way with a Review from the files tab or just a 👍 in the comment. I'm confident it's good to go but as a community project don't want to make unilateral decisions. Many thanks for the contribution, it's looking really good to me and sure it will massively benefit the book.

[anitagraser]

Looks good 👍 but I don't have time for an in-depth review this weekend

[Nowosad]

You have my 👍🏻

[Robinlovelace]

🎉 thanks for the quickfire responses guys