Add basic support for remote resources to `read_file` #531

rgieseke · 2017-09-01T11:56:05Z

Tested with GeoJSON files.

rgieseke · 2017-09-01T11:57:44Z

This might need more work, let me know if you have any suggestions!

(Started at EuroScipy 2017 sprint, see also the related issue #441)

rgieseke · 2017-09-01T11:59:33Z

I've added a custom marker for pytest, so one can use

pytest geopandas/io/tests -m "webtest"

and

pytest geopandas/io/tests -m "not webtest"

Maybe there is already someone similar set-up already?

codecov · 2017-09-01T12:21:50Z

Codecov Report

Merging #531 into master will decrease coverage by 0.06%.
The diff coverage is 91.3%.

@@            Coverage Diff             @@
##           master     #531      +/-   ##
==========================================
- Coverage   93.73%   93.66%   -0.07%     
==========================================
  Files          14       14              
  Lines        1037     1058      +21     
==========================================
+ Hits          972      991      +19     
- Misses         65       67       +2

Impacted Files	Coverage Δ
geopandas/io/file.py	`94.28% <91.3%> (-1.64%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7101fc9...13664f8. Read the comment docs.

rgieseke · 2017-09-01T17:51:48Z

Related: #464

jdmcbr · 2017-09-04T20:48:32Z

geopandas/io/file.py

+    be opened and *kwargs* are keyword args to be passed to the `open`
+    or (`BytesCollection`) method in the fiona library when opening the file.
+    For more information on possible keywords, type:
+    ``import fiona; help(fiona.open)``


I think it would be valuable to expand the docs a bit here now with this added functionality. In the from_file method on a geodataframe, which wraps read_file, there's a basic example. Something like that would be useful here. Additionally, putting it in numpy style to help with effort to make formatting of docs consistent.

rgieseke · 2017-09-05T10:06:35Z

I was trying to add a URL example as well but I can't find a simple one with a short URL - I think I'll add something to the docs which then could be served via http://geopandas.org/

rgieseke · 2017-09-05T11:50:26Z

Just fixed some syntax in the docs, should be ok now.

jdmcbr · 2017-09-10T20:41:37Z

doc/source/io.rst


-    gpd.read_file("https://d2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_110m_land.geojson")
+    url = "http://geopandas.orgd2ad6b4ur7yvpq.cloudfront.net/naturalearth-3.3.0/ne_110m_land.geojson"
+    gpd.read_file(url)


The url you changed it to isn't valid; it also isn't quite as clear anymore with the preceding sentence about reading a GeoJSON file from geojson.xyz (which is the case for the original url, though given that geojson.xyz doesn't appear in the url anywhere, it is not totally transparent then either).

Whoops, I had started thinking about using a "geopandas.org" URL, then realized it would probably not be a nice URL either and concentrated on fixing the syntax errors.

I amended the commit and all should now work again!

Did you push the change? I am still seeing the url with geopandas in the diff online?

Just did, Monday morning ... (thanks for the heads-up!)

jorisvandenbossche

Looks good!
Added a few minor comments

jorisvandenbossche · 2017-09-11T07:15:24Z

doc/source/io.rst

@@ -19,6 +19,12 @@ Any arguments passed to ``read_file()`` after the file name will be passed direc

 Among other things, one can explicitly set the driver (shapefile, GeoJSON) with the ``driver`` keyword, or pick a single layer from a multi-layered file with the ``layer`` keyword.

+Where supported in ``fiona`` *geopandas* can also load resources directly from


I would put a comma between fiona and geopandas

jorisvandenbossche · 2017-09-11T07:22:38Z

geopandas/io/file.py

+
+    Examples
+    --------
+        >>> df = geopandas.read_file("nybb.shp")


no indentation is needed here

jorisvandenbossche · 2017-09-11T07:25:20Z

geopandas/io/tests/test_io.py

@@ -54,6 +54,13 @@ def test_read_file(self):
        lower_columns = [c.lower() for c in self.columns]
        assert (df.columns[:-1] == lower_columns).all()

+    @pytest.mark.webtest


maybe just "web" instead of "webtest" ?
But it's fine to add a mark like this, we don't have anything set up like this yet.

Good suggestion, had just taken it from the py.test example in their docs.

jdmcbr · 2017-09-14T04:13:13Z

Thanks @rgieseke!

rgieseke · 2017-09-14T04:24:07Z

Thanks @jdmcbr and @jorisvandenbossche !

* Add basic support for remote resources to `read_file` Tested with GeoJSON files. * TST: Use GeoJSON file from GeoPandas repo * Switch to numpydoc style * Fix syntax in docs * Formatting and rename test marker (cherry picked from commit a43d166)

linwoodc3 · 2017-09-20T12:04:35Z

When will the pypi version of geopandas include this feature? I installed from the master branch and was able to load the remote file. However, I couldn't read the remote file from the pypi pip install geopandas version.

Also, how feasible is it to add an optional argument for a cert file to handle requests to an https file?

rgieseke · 2017-09-20T12:43:11Z

On the https - good question, what Python version are you using?

For Python versions earlier than 2.7.9, urllib does not attempt to validate the server certificates of HTTPS URIs.

https://docs.python.org/2/library/urllib.html

linwoodc3 · 2017-09-20T13:44:14Z

Always above 2.7.9 and up to 3.6. Anaconda versions.

rgieseke · 2017-09-20T22:00:08Z

I might have mis-understood your use-case, do you want to use a local cert file?

linwoodc3 · 2017-09-20T23:18:35Z

Yes, you are correct.

This is the requests equivalent:

import requests

cert = <path to cert>
r = requests.get(url,cert =cert, verify=False)

Documented here: http://docs.python-requests.org/en/master/user/advanced/#client-side-certificates

If we can pass a cert argument in with this, it would do the trick I think. The only thing is, I've never done this in urllib..but sure there's a way.

jorisvandenbossche · 2017-09-21T19:36:14Z

There has recently been some discussion about this on the pandas issue tracker: pandas-dev/pandas#16716 (and subsequent PRs). Can be used as inspiration or reused what comes out of that

Add basic support for remote resources to read_file

54f0f47

Tested with GeoJSON files.

TST: Use GeoJSON file from GeoPandas repo

f961370

jdmcbr reviewed Sep 4, 2017

View reviewed changes

Switch to numpydoc style

269e847

jdmcbr reviewed Sep 10, 2017

View reviewed changes

Fix syntax in docs

78008b4

jorisvandenbossche reviewed Sep 11, 2017

View reviewed changes

Formatting and rename test marker

13664f8

jdmcbr merged commit a43d166 into geopandas:master Sep 14, 2017

rgieseke deleted the remote-geojson branch September 14, 2017 04:24

dzbee mentioned this pull request Nov 8, 2017

GeoDataFrame.from_url() #464

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add basic support for remote resources to `read_file` #531

Add basic support for remote resources to `read_file` #531

rgieseke commented Sep 1, 2017

rgieseke commented Sep 1, 2017

rgieseke commented Sep 1, 2017

codecov bot commented Sep 1, 2017 •

edited

rgieseke commented Sep 1, 2017

jdmcbr Sep 4, 2017

rgieseke commented Sep 5, 2017 •

edited

rgieseke commented Sep 5, 2017

jdmcbr Sep 10, 2017

rgieseke Sep 11, 2017

jorisvandenbossche Sep 11, 2017

rgieseke Sep 11, 2017

jorisvandenbossche left a comment

jorisvandenbossche Sep 11, 2017

rgieseke Sep 11, 2017

jorisvandenbossche Sep 11, 2017

rgieseke Sep 11, 2017

jorisvandenbossche Sep 11, 2017

rgieseke Sep 11, 2017

jdmcbr commented Sep 14, 2017

rgieseke commented Sep 14, 2017

linwoodc3 commented Sep 20, 2017

rgieseke commented Sep 20, 2017

linwoodc3 commented Sep 20, 2017

rgieseke commented Sep 20, 2017

linwoodc3 commented Sep 20, 2017

jorisvandenbossche commented Sep 21, 2017

		@@ -19,6 +19,12 @@ Any arguments passed to ``read_file()`` after the file name will be passed direc

		Among other things, one can explicitly set the driver (shapefile, GeoJSON) with the ``driver`` keyword, or pick a single layer from a multi-layered file with the ``layer`` keyword.

		Where supported in ``fiona`` geopandas can also load resources directly from

Add basic support for remote resources to read_file #531

Add basic support for remote resources to read_file #531

Conversation

rgieseke commented Sep 1, 2017

rgieseke commented Sep 1, 2017

rgieseke commented Sep 1, 2017

codecov bot commented Sep 1, 2017 • edited

Codecov Report

rgieseke commented Sep 1, 2017

Choose a reason for hiding this comment

rgieseke commented Sep 5, 2017 • edited

rgieseke commented Sep 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdmcbr commented Sep 14, 2017

rgieseke commented Sep 14, 2017

linwoodc3 commented Sep 20, 2017

rgieseke commented Sep 20, 2017

linwoodc3 commented Sep 20, 2017

rgieseke commented Sep 20, 2017

linwoodc3 commented Sep 20, 2017

jorisvandenbossche commented Sep 21, 2017

Add basic support for remote resources to `read_file` #531

Add basic support for remote resources to `read_file` #531

codecov bot commented Sep 1, 2017 •

edited

rgieseke commented Sep 5, 2017 •

edited