FEAT: PostGIS support #1787

ian-r-rose · 2019-05-16T19:32:50Z

This is in an extremely early state, but I wanted to open this for comment now as I'm not likely to get the chance to work much on it for a few days.

ian-r-rose · 2019-05-16T19:33:33Z

ibis/sql/alchemy.py

@@ -14,6 +14,14 @@
 from sqlalchemy.ext.compiler import compiles as sa_compiles
 from sqlalchemy.sql.elements import Over as _Over

+try:


We may want to put all this geoalchemy logic in another submodule, but for now I have it inline with the rest of the sqlalchemy stuff.

ibis/sql/alchemy.py

ian-r-rose · 2019-05-16T19:35:54Z

ibis/sql/alchemy.py

@@ -881,7 +939,17 @@ def _fetch(self, cursor):
            columns=cursor.proxy.keys(),
            coerce_float=True,
        )
-        return self.schema().apply_to(df)
+        df = self.schema().apply_to(df)
+        if GEO:


We may want to move this logic elsewhere (possibly even to Schema.apply_to), but for now this is convenient.

xmnlab · 2019-05-21T21:04:17Z

it is looking pretty good :)

it seems there are some pep8 issues ... if you want you can run make develop that will install git pre-commit hook. it has some useful things such as flake8, black and isort

ian-r-rose · 2019-05-21T21:59:42Z

Thanks for the tip @xmnlab. At this point a lot of the functionality I'm looking for is in, but it needs a lot of testing. Any tips as to how to go about that? In particular, it would be useful to have a PostGIS-enabled PostgreSQL and a Spatialite-enabled SQLite in CI.

codecov · 2019-05-21T23:04:30Z

Codecov Report

Merging #1787 into master will decrease coverage by 1.87%.
The diff coverage is 51.74%.

@@            Coverage Diff             @@
##           master    #1787      +/-   ##
==========================================
- Coverage   87.33%   85.45%   -1.88%     
==========================================
  Files          81       80       -1     
  Lines       15266    15219      -47     
  Branches     1965     1956       -9     
==========================================
- Hits        13333    13006     -327     
- Misses       1567     1860     +293     
+ Partials      366      353      -13

Impacted Files	Coverage Δ
ibis/expr/rules.py	`99.41% <100%> (+0.01%)`	⬆️
ibis/expr/operations.py	`95.85% <100%> (+0.11%)`	⬆️
ibis/expr/datatypes.py	`94.91% <100%> (-0.11%)`	⬇️
ibis/sql/alchemy.py	`89.14% <15.38%> (-3.9%)`	⬇️
ibis/expr/api.py	`89.53% <33.33%> (-4.27%)`	⬇️
ibis/bigquery/client.py	`41.1% <0%> (-53.39%)`	⬇️
ibis/bigquery/compiler.py	`58.67% <0%> (-38.76%)`	⬇️
ibis/bigquery/udf/api.py	`80.48% <0%> (-14.64%)`	⬇️
ibis/bigquery/api.py	`63.33% <0%> (-10.01%)`	⬇️
ibis/impala/compiler.py	`91.28% <0%> (-5.27%)`	⬇️
... and 31 more

xmnlab · 2019-05-22T13:54:45Z

hey @ian-r-rose you can add the tests here: https://github.com/ibis-project/ibis/blob/f6220bb29ea05035ebeb44437c7db36ce0ba8562/ibis/tests/all/test_geospatial.py

related to postgis, you can try to change

ibis/ci/docker-compose.yml

Line 5 in f6220bb

image: postgres

to one docker image from https://wiki.osgeo.org/wiki/DockerImages or https://hub.docker.com/r/geographica/postgis/ (this last one I just find right now)

probably you will need to add the the new libs you are using to ci/requirements-dev yaml files

related to spatialite extension ... not sure .. maybe you need to add libspatialite to requirements-dev.yml files and change the import here:

ibis/ci/datamgr.py

Line 13 in f6220bb

import sqlalchemy as sa

maybe you also will need to change something here:

ibis/ci/datamgr.py

Line 40 in f6220bb

def init_database(driver, params, schema=None, recreate=True, **kwargs):

@cpcloud what do you think about this?

ibis/sql/postgres/tests/test_client.py

ibis/sql/alchemy.py

ian-r-rose · 2019-06-05T20:22:58Z

Okay, I think this is close-to-ready for some feedback. Tests are mostly passing except for docs (any idea what is going on there?), and most of the functionality I am looking for is in.

There are still some tests to write, and some coverage to improve, but querying geospatial data with PostGIS basically works!

ci/docker-compose.yml

xmnlab · 2019-06-06T20:37:24Z

hey @ian-r-rose is this still WIP or is it done for a review?

ian-r-rose · 2019-06-06T20:40:22Z

I think it will need some more test coverage (hence still having WIP), but some review would be helpful at this point.

xmnlab · 2019-06-06T20:42:53Z

OK ... I will review it today :)

ian-r-rose · 2019-06-06T20:45:13Z

Thanks!

xmnlab · 2019-06-06T22:49:22Z

.isort.cfg

@@ -1,2 +1,2 @@
 [settings]
-known_third_party = asv,click,clickhouse_driver,dateutil,google,graphviz,impala,jinja2,kudu,multipledispatch,numpy,pandas,pkg_resources,plumbum,psycopg2,pyarrow,pydata_google_auth,pymapd,pymysql,pytest,pytz,regex,requests,ruamel,setuptools,sphinx_rtd_theme,sqlalchemy,toolz
+known_third_party = asv,click,clickhouse_driver,dateutil,geopandas,google,graphviz,impala,jinja2,kudu,multipledispatch,numpy,pandas,pkg_resources,plumbum,psycopg2,pyarrow,pydata_google_auth,pymapd,pymysql,pytest,pytz,regex,requests,ruamel,setuptools,sphinx_rtd_theme,sqlalchemy,toolz


maybe also gdal and geoalchemy2 should be added here.

This was automatically added by the pre-commit hook. Should I edit it manually?

actually not sure .. @cpcloud do you know?

No, this is fine. Third-party dependencies get automatically added when they are detected.

We should only add gdal and geoalchemy2 if they are imported directly inside of ibis and they are not detected by seed-isort-config.

Yes, please do.

Hmm, seed-isort-config doesn't seem to like my manually adding imports here. It is automatically removing them in the pre-commit hook.

Ok, I think that's probably fine. Does it cause issues elsewhere? If not then I'd say leave as is.

Doesn't seem to cause any issues.

And now the file is unchanged. It seems that isort isn't smart enough to see pytest.importorskip or imports in a try/except block.

ci/datamgr.py

ibis/sql/alchemy.py

ibis/sql/postgres/tests/test_functions.py

xmnlab

looks awesome @ian-r-rose ! I just added small comments.

xmnlab · 2019-06-06T23:19:18Z

mmm it seems now there is a file with conflict

ci/datamgr.py

ibis/sql/alchemy.py

ci/requirements-3.5-dev.yml

cpcloud · 2019-06-07T15:57:44Z

This is looking pretty close, just a few more review comments and the merge conflict and we should be good to go.

ian-r-rose · 2019-06-12T19:55:59Z

I've made a demonstration gist showing this PR in action: https://gist.github.com/ian-r-rose/255295a5a98ca59259b55aee9a7cfdae

It's a little hokey, but shows some non-trivial geospatial operations.

xmnlab · 2019-06-12T20:56:42Z

@ian-r-rose your notebook looks great!

ibis/sql/postgres/tests/test_postgis.py

cpcloud · 2019-06-12T22:16:51Z

Agree with @xmnlab: sweet notebook.

cpcloud · 2019-06-12T22:17:43Z

@ian-r-rose As a follow up we might consider adding it to our tutorial notebook collection.

ian-r-rose · 2019-06-13T03:57:57Z

Okay, I think I am happy with this, feature-wise. Ideas for follow-ups:

Implement some missing operations
More/better tests
Tutorial notebook
Clean up some of the logic around loading geo data in CI

xmnlab

LGTM! thanks @ian-r-rose ! @cpcloud do you think it is done to be merged?

cpcloud

Bravo! LGTM, merging.

cpcloud · 2019-06-17T15:22:51Z

@ian-r-rose Thanks for all your work on this!

ian-r-rose · 2019-06-17T15:25:19Z

Thanks for the patient review @cpcloud and @xmnlab!

xmnlab · 2019-06-17T15:31:16Z

thanks for you hard work @ian-r-rose these new features are awesome!

xmnlab · 2019-06-17T17:26:48Z

it seems also it is necessary to add smoke tests for the new ibis expressions: ibis/expr/tests/test_geospatial.py

maybe parametrize the tests would be good

I will open an issue for that.

cpcloud · 2019-06-17T19:06:39Z

Cool. @ian-r-rose Can you open up issues for your bullet points above if you haven't already?

xmnlab · 2019-06-18T21:36:17Z

@ian-r-rose just a question, did you add translation for literals for geotypes on postgis? (ex: point_0 = ibis.literal((0, 0), type='point:geometry').name('p'))

ian-r-rose · 2019-06-18T22:01:51Z

No, I don't think so. If you can point me to some examples, I can try to do that in a follow-up.

xmnlab · 2019-06-18T22:06:06Z

awesome! this is how geo literals were implemented on mapd: https://github.com/ibis-project/ibis/blob/master/ibis/mapd/operations.py#L390

I think you can add these translations here for postgis: https://github.com/ibis-project/ibis/blob/master/ibis/sql/postgres/compiler.py#L585

Follow up to #1787 and #1925 . This adds a number of new operations for the PostGIS backend: * `ST_GeometryType` * `ST_GeometryN` * `ST_IsValid` * `ST_LineLocatePoint` * `ST_LineMerge` * `ST_LineSubstring` * `ST_OrderingEquals` * `ST_Union` The last one requires special attention. That [`ST_Union`](http://postgis.net/docs/ST_Union.html) is available as both a value operation and a column aggregation operation. To avoid a name collision in the ibis APIs, I have named the aggregation variant `unary_union`. This is following the same choice made by [GeoPandas](h ttp://geopandas.org/reference.html#geopandas.GeoSeries.unary_union). Author: Ian Rose <ian.r.rose@gmail.com> Closes #1987 from ian-r-rose/more-postgis-ops and squashes the following commits: aa3de1d [Ian Rose] Work on docstring styles. 379156f [Ian Rose] Export new ops as part of the top-level expr api. dfba5de [Ian Rose] Add ST_Union functionality. 02ba505 [Ian Rose] Add some tests for the new functions. 259a477 [Ian Rose] Work on some additional geospatial operations and their implementation in postgis.

ian-r-rose commented May 16, 2019

View reviewed changes

ian-r-rose force-pushed the postgis branch from a5e0fbc to a573f8d Compare May 21, 2019 21:58

ian-r-rose commented May 28, 2019

View reviewed changes

ibis/sql/postgres/tests/test_client.py Outdated Show resolved Hide resolved

ian-r-rose force-pushed the postgis branch from 7f1a979 to df9235a Compare May 28, 2019 18:12

ian-r-rose changed the title ~~[WIP] PostGIS suppot~~ [WIP] PostGIS support Jun 4, 2019

ian-r-rose force-pushed the postgis branch from bcd7c7d to ab48fd6 Compare June 4, 2019 20:15

ian-r-rose commented Jun 5, 2019

View reviewed changes

ibis/sql/alchemy.py Show resolved Hide resolved

ian-r-rose commented Jun 5, 2019

View reviewed changes

ci/docker-compose.yml Show resolved Hide resolved

ian-r-rose force-pushed the postgis branch 2 times, most recently from cb9c296 to b1d8aef Compare June 6, 2019 18:18

xmnlab reviewed Jun 6, 2019

View reviewed changes

ci/datamgr.py Show resolved Hide resolved

xmnlab reviewed Jun 6, 2019

View reviewed changes

ibis/sql/alchemy.py Outdated Show resolved Hide resolved

xmnlab reviewed Jun 6, 2019

View reviewed changes

ibis/sql/postgres/tests/test_functions.py Outdated Show resolved Hide resolved

xmnlab reviewed Jun 6, 2019

View reviewed changes

cpcloud reviewed Jun 7, 2019

View reviewed changes

ci/datamgr.py Show resolved Hide resolved

cpcloud reviewed Jun 7, 2019

View reviewed changes

ibis/sql/alchemy.py Show resolved Hide resolved

cpcloud reviewed Jun 7, 2019

View reviewed changes

ci/requirements-3.5-dev.yml Outdated Show resolved Hide resolved

cpcloud added the postgres The PostgreSQL backend label Jun 7, 2019

ian-r-rose added 2 commits June 12, 2019 12:54

Add postgis marker to setup.cfg

0a5daca

Only load geospatial data into postgis when python >= 3.6.

10a10fc

ian-r-rose force-pushed the postgis branch from 2d064b1 to 10a10fc Compare June 12, 2019 19:54

cpcloud reviewed Jun 12, 2019

View reviewed changes

ibis/sql/postgres/tests/test_postgis.py Outdated Show resolved Hide resolved

ian-r-rose added 3 commits June 12, 2019 16:26

Use importorskip for geopandas in test suite.

38aa985

Add ST_DWithin and ST_DFullyWithin.

90673fc

Ensure that empty resulting dataframes work okay.

db50529

ian-r-rose changed the title ~~[WIP] PostGIS support~~ PostGIS support Jun 12, 2019

xmnlab approved these changes Jun 13, 2019

View reviewed changes

xmnlab requested a review from cpcloud June 17, 2019 14:43

cpcloud approved these changes Jun 17, 2019

View reviewed changes

cpcloud merged commit 52cb979 into ibis-project:master Jun 17, 2019

cpcloud changed the title ~~PostGIS support~~ FEAT: PostGIS support Jun 17, 2019

xmnlab mentioned this pull request Jun 17, 2019

TST: Add tests for new geospatial operators added for postgis #1835

Closed

ian-r-rose deleted the postgis branch June 17, 2019 21:44

ian-r-rose mentioned this pull request Oct 2, 2019

FEAT: Add more POSTGIS operations #1987

Merged

FEAT: PostGIS support #1787

FEAT: PostGIS support #1787

Conversation

ian-r-rose commented May 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmnlab commented May 21, 2019

ian-r-rose commented May 21, 2019

codecov bot commented May 21, 2019 • edited Loading

Codecov Report

xmnlab commented May 22, 2019

ian-r-rose commented Jun 5, 2019

xmnlab commented Jun 6, 2019

ian-r-rose commented Jun 6, 2019

xmnlab commented Jun 6, 2019

ian-r-rose commented Jun 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmnlab left a comment

Choose a reason for hiding this comment

xmnlab commented Jun 6, 2019

cpcloud commented Jun 7, 2019

ian-r-rose commented Jun 12, 2019

xmnlab commented Jun 12, 2019

cpcloud commented Jun 12, 2019

cpcloud commented Jun 12, 2019 • edited Loading

ian-r-rose commented Jun 13, 2019

xmnlab left a comment

Choose a reason for hiding this comment

cpcloud left a comment

Choose a reason for hiding this comment

cpcloud commented Jun 17, 2019

ian-r-rose commented Jun 17, 2019

xmnlab commented Jun 17, 2019

xmnlab commented Jun 17, 2019

cpcloud commented Jun 17, 2019

xmnlab commented Jun 18, 2019

ian-r-rose commented Jun 18, 2019

xmnlab commented Jun 18, 2019

codecov bot commented May 21, 2019 •

edited

Loading

cpcloud commented Jun 12, 2019 •

edited

Loading