FEAT: Add geopandas as output for omniscidb #1858

xmnlab · 2019-06-25T04:42:52Z

In this PR:

Add geopandas output for omniscidb
Add postgresql backend on geospatial tests
Add SetSRID to postgresql backend
Add geo spatial rules for literal on postgresql backend

ibis/mapd/operations.py

xmnlab · 2019-06-25T17:30:32Z

@ian-r-rose I am adding new features on postgis implementation. if you have a time, could you take a look into that?

codecov · 2019-06-26T18:50:01Z

Codecov Report

❗ No coverage uploaded for pull request base (master@323967f). Click here to learn what that means.
The diff coverage is 80.48%.

@@            Coverage Diff            @@
##             master    #1858   +/-   ##
=========================================
  Coverage          ?   85.68%           
=========================================
  Files             ?       81           
  Lines             ?    15537           
  Branches          ?     1994           
=========================================
  Hits              ?    13313           
  Misses            ?     1856           
  Partials          ?      368

Impacted Files	Coverage Δ
ibis/sql/alchemy.py	`92.33% <ø> (ø)`
ibis/mapd/operations.py	`71.95% <100%> (ø)`
ibis/sql/postgres/compiler.py	`91.05% <62.5%> (ø)`
ibis/mapd/client.py	`60.99% <91.3%> (ø)`

xmnlab · 2019-06-26T19:30:15Z

@cpcloud it is done for review :) thanks!

ibis/mapd/client.py

ibis/sql/postgres/compiler.py

ibis/tests/all/test_geospatial.py

xmnlab

@cpcloud it is done for a new review! thanks!

ibis/tests/all/test_geospatial.py

xmnlab · 2019-07-09T01:52:56Z

@cpcloud when you have time could you review this one? so I can move forward with the remains PR

cpcloud · 2019-07-09T16:52:28Z

Reviewing now.

ci/impalamgr.py

ibis/common/geospatial.py

ibis/mapd/client.py

ibis/tests/all/test_geospatial.py

ibis/mapd/client.py

xmnlab · 2019-07-09T20:45:21Z

thanks @cpcloud for the feedback! I will work on that!

xmnlab

@cpcloud thanks for the review. I applied the suggestion you made. it is done for a new review!

ci/impalamgr.py

ibis/common/geospatial.py

xmnlab · 2019-07-13T21:11:11Z

ibis/mapd/client.py

+    """
+
+    def to_df(self):
+        if not isinstance(self.cursor, Cursor):


xmnlab · 2019-07-13T21:11:31Z

ibis/mapd/client.py

+    def to_df(self):
+        if not isinstance(self.cursor, Cursor):
+            if self.cursor is None:
+                return geopandas.GeoDataFrameDataFrame([])


thanks for catch that up!

ibis/mapd/client.py

ibis/tests/all/test_geospatial.py

xmnlab · 2019-07-19T06:39:14Z

@cpcloud, when you have time, could you review again this PR? Thanks!

xmnlab · 2019-07-23T22:29:31Z

FAILED ibis/tests/all/test_temporal.py::test_timestamp_truncate[Spark-ms] - A...
FAILED ibis/tests/all/test_temporal.py::test_timestamp_truncate[Spark-us] - A...
FAILED ibis/tests/all/test_temporal.py::test_timestamp_truncate[Spark-ns] - A...
FAILED ibis/tests/all/test_temporal.py::test_date_truncate[Spark-M] - Attribu...
FAILED ibis/tests/all/test_temporal.py::test_date_truncate[Spark-D] - Attribu...

cpcloud

LGTM! Merging! Thanks @xmnlab

cpcloud · 2019-07-24T17:56:26Z

Actually @xmnlab Can you rebase on top of master to pick up the fix from #1893

xmnlab · 2019-07-25T18:10:50Z

@cpcloud rebase done! but it seems it is raising some errors.

cpcloud · 2019-07-25T19:44:47Z

@xmnlab Can you rebase and push again to pick up fixes and resolve the conflicts?

xmnlab · 2019-07-26T21:51:07Z

FAILED ibis/tests/all/test_temporal.py::test_timestamp_truncate[Spark-ms] - A...
FAILED ibis/tests/all/test_temporal.py::test_timestamp_truncate[Spark-us] - A...
FAILED ibis/tests/all/test_temporal.py::test_timestamp_truncate[Spark-ns] - A...
FAILED ibis/tests/all/test_temporal.py::test_date_truncate[Spark-M] - Attribu...
FAILED ibis/tests/all/test_temporal.py::test_date_truncate[Spark-D] - Attribu...
= 5 failed, 6826 passed, 1039 skipped, 564 xfailed, 44 warnings in 612.67 seconds =

xmnlab · 2019-07-27T16:21:01Z

@cpcloud do you have any idea about these errors?

cpcloud · 2019-07-29T16:04:09Z

@xmnlab Looking now. Looks like there's a conflict again here, sorry about that. Can you fix up and push again?

xmnlab · 2019-07-29T16:19:50Z

rebased! :)

xmnlab · 2019-07-29T17:03:12Z

it seems the errors related to test_temporal and spark still occurs

cpcloud · 2019-07-29T18:39:19Z

@xmnlab We've merged some code in the meantime that uses the old version of common.py so you need to change the import ibis.common as com to import ibis.common.exception as com to fix those errors.

cpcloud · 2019-07-29T18:39:37Z

After that, this is good to merge.

xmnlab · 2019-07-29T22:23:27Z

thanks @cpcloud I didn't realize that the problem could be in new files.

xmnlab · 2019-08-01T15:41:15Z

ERROR at setup of test_compile ________________________
[gw1] linux -- Python 3.7.3 /opt/conda/envs/ibis-env/bin/python

data_directory = PosixPath('/tmp/ibis-testing-data')

    @pytest.fixture(scope='session')
    def client(data_directory):
        pytest.importorskip('pyspark')
>       return get_spark_testing_client(data_directory)

ibis/spark/tests/conftest.py:9:

xmnlab · 2019-08-01T18:47:30Z

@cpcloud it is done for a new review! thanks!

cpcloud

LGTM, merging!

cpcloud added this to the Next Feature Release milestone Jun 25, 2019

cpcloud added the omnisci label Jun 25, 2019

cpcloud reviewed Jun 25, 2019

View reviewed changes

ibis/mapd/operations.py Outdated Show resolved Hide resolved

xmnlab marked this pull request as ready for review June 26, 2019 19:29

xmnlab requested a review from cpcloud June 26, 2019 19:30

xmnlab mentioned this pull request Jun 27, 2019

FEAT: Add shapely geometries as input for literals #1860

Merged

cpcloud reviewed Jun 30, 2019

View reviewed changes

cpcloud requested changes Jun 30, 2019

View reviewed changes

ibis/tests/all/test_geospatial.py Outdated Show resolved Hide resolved

ibis/tests/all/test_geospatial.py Outdated Show resolved Hide resolved

ibis/tests/all/test_geospatial.py Outdated Show resolved Hide resolved

xmnlab force-pushed the add-omniscidb-geopandas-output branch 2 times, most recently from 703e488 to cffc14a Compare July 3, 2019 18:35

xmnlab requested a review from cpcloud July 3, 2019 20:59

xmnlab commented Jul 3, 2019

View reviewed changes

ibis/tests/all/test_geospatial.py Outdated Show resolved Hide resolved

cpcloud requested changes Jul 9, 2019

View reviewed changes

xmnlab force-pushed the add-omniscidb-geopandas-output branch from 5d5e9be to 6194340 Compare July 14, 2019 14:33

xmnlab commented Jul 14, 2019

View reviewed changes

xmnlab force-pushed the add-omniscidb-geopandas-output branch from 6194340 to 95ec563 Compare July 23, 2019 21:03

xmnlab requested a review from cpcloud July 23, 2019 21:03

cpcloud approved these changes Jul 24, 2019

View reviewed changes

xmnlab force-pushed the add-omniscidb-geopandas-output branch from 95ec563 to 263255c Compare July 25, 2019 17:30

xmnlab force-pushed the add-omniscidb-geopandas-output branch from 263255c to e5995a3 Compare July 26, 2019 01:57

xmnlab force-pushed the add-omniscidb-geopandas-output branch from 440b339 to 59d2d2c Compare July 29, 2019 16:19

add geopandas as output for omniscidb

d273c52

Fixed typo issues

0088881

xmnlab force-pushed the add-omniscidb-geopandas-output branch from 560d677 to 0088881 Compare August 1, 2019 15:58

xmnlab mentioned this pull request Aug 2, 2019

Update intake-mapd Quansight/omnisci#51

Closed

cpcloud approved these changes Aug 2, 2019

View reviewed changes

cpcloud merged commit b3d0400 into ibis-project:master Aug 2, 2019

dharhas mentioned this pull request Aug 7, 2019

Integrate geopandas and shapely to ibis Quansight/omnisci#49

Closed

2 tasks

cpcloud modified the milestones: Next Feature Release, Next Major Release Aug 26, 2019

xmnlab deleted the add-omniscidb-geopandas-output branch September 27, 2019 21:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT: Add geopandas as output for omniscidb #1858

FEAT: Add geopandas as output for omniscidb #1858

xmnlab commented Jun 25, 2019 •

edited

Loading

xmnlab commented Jun 25, 2019

codecov bot commented Jun 26, 2019 •

edited

Loading

xmnlab commented Jun 26, 2019

xmnlab left a comment

xmnlab commented Jul 9, 2019

cpcloud commented Jul 9, 2019

xmnlab commented Jul 9, 2019

xmnlab left a comment

xmnlab Jul 13, 2019

xmnlab Jul 13, 2019

xmnlab commented Jul 19, 2019

xmnlab commented Jul 23, 2019

cpcloud left a comment

cpcloud commented Jul 24, 2019

xmnlab commented Jul 25, 2019

cpcloud commented Jul 25, 2019

xmnlab commented Jul 26, 2019

xmnlab commented Jul 27, 2019

cpcloud commented Jul 29, 2019

xmnlab commented Jul 29, 2019

xmnlab commented Jul 29, 2019

cpcloud commented Jul 29, 2019

cpcloud commented Jul 29, 2019

xmnlab commented Jul 29, 2019

xmnlab commented Aug 1, 2019

xmnlab commented Aug 1, 2019

cpcloud left a comment

FEAT: Add geopandas as output for omniscidb #1858

FEAT: Add geopandas as output for omniscidb #1858

Conversation

xmnlab commented Jun 25, 2019 • edited Loading

xmnlab commented Jun 25, 2019

codecov bot commented Jun 26, 2019 • edited Loading

Codecov Report

xmnlab commented Jun 26, 2019

xmnlab left a comment

Choose a reason for hiding this comment

xmnlab commented Jul 9, 2019

cpcloud commented Jul 9, 2019

xmnlab commented Jul 9, 2019

xmnlab left a comment

Choose a reason for hiding this comment

xmnlab Jul 13, 2019

Choose a reason for hiding this comment

xmnlab Jul 13, 2019

Choose a reason for hiding this comment

xmnlab commented Jul 19, 2019

xmnlab commented Jul 23, 2019

cpcloud left a comment

Choose a reason for hiding this comment

cpcloud commented Jul 24, 2019

xmnlab commented Jul 25, 2019

cpcloud commented Jul 25, 2019

xmnlab commented Jul 26, 2019

xmnlab commented Jul 27, 2019

cpcloud commented Jul 29, 2019

xmnlab commented Jul 29, 2019

xmnlab commented Jul 29, 2019

cpcloud commented Jul 29, 2019

cpcloud commented Jul 29, 2019

xmnlab commented Jul 29, 2019

xmnlab commented Aug 1, 2019

xmnlab commented Aug 1, 2019

cpcloud left a comment

Choose a reason for hiding this comment

xmnlab commented Jun 25, 2019 •

edited

Loading

codecov bot commented Jun 26, 2019 •

edited

Loading