FEAT: Add new geospatial functions to OmniSciDB backend #1836

xmnlab · 2019-06-17T19:55:47Z

Add 2 new geo spatial functions to OmniSciDB backend:

set_srid
transform

Add a cast to geographic type

codecov · 2019-06-19T15:46:11Z

Codecov Report

Merging #1836 into master will decrease coverage by 1.77%.
The diff coverage is 96.22%.

@@            Coverage Diff             @@
##           master    #1836      +/-   ##
==========================================
- Coverage   87.46%   85.69%   -1.78%     
==========================================
  Files          81       81              
  Lines       15458    15501      +43     
  Branches     1977     1985       +8     
==========================================
- Hits        13520    13283     -237     
- Misses       1571     1853     +282     
+ Partials      367      365       -2

Impacted Files	Coverage Δ
ibis/expr/operations.py	`95.92% <100%> (ø)`	⬆️
ibis/expr/datatypes.py	`95.31% <100%> (+0.17%)`	⬆️
ibis/expr/api.py	`92.32% <100%> (-0.05%)`	⬇️
ibis/mapd/operations.py	`72.08% <60%> (+0.76%)`	⬆️
ibis/bigquery/client.py	`41.1% <0%> (-53.39%)`	⬇️
ibis/bigquery/compiler.py	`59.92% <0%> (-37.5%)`	⬇️
ibis/bigquery/udf/api.py	`80.48% <0%> (-14.64%)`	⬇️
ibis/bigquery/api.py	`63.33% <0%> (-10.01%)`	⬇️
ibis/impala/compiler.py	`91.29% <0%> (-5.26%)`	⬇️
ibis/pandas/client.py	`85.54% <0%> (-3.47%)`	⬇️
... and 5 more

xmnlab · 2019-06-20T13:36:42Z

@cpcloud it seems it is done for a review .. there is just one error related to impala test_drop_view - ibis.impala.tests.test_ddl

ibis/expr/datatypes.py

ibis/expr/operations.py

cpcloud · 2019-06-20T15:00:05Z

ibis/mapd/operations.py

+        dtype_from = arg._dtype.geotype
+        dtype_target = target.geotype
+
+        if dtype_from == dtype_target:


Why is this an error and not a no-op?

I think I am not following you here. could you provide more details?

I think I'm not understanding why it's an error to cast from a geotype to the same geotype. That is what you're doing here is it not?

Oh I see .. you're right. I will fix that! thanks!

ibis/tests/all/test_geospatial.py

ibis/tests/backends.py

xmnlab

Thanks @cpcloud for the feedback.
I will apply the changes you requested.
I added some comments and question for some topics here. thanks!

ibis/tests/all/test_geospatial.py

ibis/tests/backends.py

ibis/expr/datatypes.py

ibis/expr/operations.py

xmnlab · 2019-06-20T16:39:44Z

ibis/mapd/operations.py

+        dtype_from = arg._dtype.geotype
+        dtype_target = target.geotype
+
+        if dtype_from == dtype_target:


I think I am not following you here. could you provide more details?

cpcloud · 2019-06-20T17:25:47Z

@xmnlab Looks like conda-forge is not working at the moment, lots of 503s.

xmnlab · 2019-06-20T20:32:44Z

@cpcloud it seems yesterday conda-forge was also unstable ...

xmnlab · 2019-06-20T20:36:00Z

@cpcloud I fix the code related to cast with same type (#1836 (comment)) let me know your thoughts about the approach used.

cpcloud · 2019-06-20T20:59:17Z

ibis/expr/api.py

+        target_type.lower() in ('geometry', 'geography')
+        and (
+            arg.type().geotype == target_type
+            or (arg.type().geotype is None and target_type == 'geometry')


Can you explain the behavior here? I'm not sure I follow purpose of this additional logic. In particular, what is the rationale for not performing a cast in these conditions?

It looks like you're trying to enable generic "cast to geometry or geography" functionality, and trying to avoid generating code for cast that would essentially be a no-op. Is there any reason to avoid generating this code?

I want to make sure we're not baking in assumptions that are tied to the OmniSci backend in the generic expression API, so if you could walk me through the reasoning here that would be helpful.

It might help to add a unit test just for this functionality.

well I will try to explain me.

In this PR the cast operation for geospatial works from point, linestring, polygon or multipolygon to geometry or geography.

point, linestring, polygon and multipolygon have a attribute called geotype that specify it is a geometry or geography (also it could be None, that is treated as geometry)

so cast from geometry to geometry is the same to cast int to int ... so I am follow the same idea of if op.to.equals(arg.type()): ...

currently omniscidb doesn't support cast to geometry ... but I am treating that just inside the mapd backend.

Ok, makes a bit more sense now. Can you perform the comparison with op.to instead of looking at target_type? op.to is an instance of DataType as opposed to target_type which can be either a string or a DataType.

cpcloud · 2019-06-20T21:53:48Z

ibis/expr/api.py

+        target_type.lower() in ('geometry', 'geography')
+        and (
+            arg.type().geotype == target_type
+            or (arg.type().geotype is None and target_type == 'geometry')


Ok, makes a bit more sense now. Can you perform the comparison with op.to instead of looking at target_type? op.to is an instance of DataType as opposed to target_type which can be either a string or a DataType.

xmnlab · 2019-06-20T22:02:04Z

@cpcloud Ok .. I will change that! thanks!

xmnlab · 2019-06-20T22:59:17Z

there are more bug to fix .. I am working on that

xmnlab · 2019-06-21T01:53:05Z

@cpcloud it is done for a new review!

py35_test failed because a CondaHTTPError: HTTP 504 GATEWAY_TIMEOUT

cpcloud · 2019-06-21T17:00:01Z

ibis/expr/api.py

+        to_geotype = op.to.geotype
+        if (
+            from_geotype == to_geotype
+            or (from_geotype is None and to_geotype == 'geometry')


I think this can be reduced to

from_geotype = arg.type().geotype or 'geometry' to_geotype = op.to.geotype if from_geotype == to_geotype: ...

looks much better. thanks! I will change that

cpcloud · 2019-06-24T02:33:45Z

@xmnlab Can you rebase to pick up the fix for the failing tests?

cpcloud · 2019-06-24T02:33:57Z

Once this is green, I'll merge.

xmnlab · 2019-06-24T03:27:16Z

rebased! thanks @cpcloud !

cpcloud · 2019-06-24T10:25:36Z

Thanks @xmnlab!

xmnlab changed the title ~~FEAT: Added new geospatial functions to OmniSciDB backend~~ FEAT: Add new geospatial functions to OmniSciDB backend Jun 17, 2019

xmnlab force-pushed the add-more-geo-funcs-mapd branch 2 times, most recently from b8e88b3 to a6704ec Compare June 19, 2019 20:57

xmnlab marked this pull request as ready for review June 19, 2019 23:59

xmnlab requested a review from cpcloud June 20, 2019 13:34

cpcloud reviewed Jun 20, 2019

View reviewed changes

ibis/expr/datatypes.py Outdated Show resolved Hide resolved

cpcloud reviewed Jun 20, 2019

View reviewed changes

ibis/expr/datatypes.py Outdated Show resolved Hide resolved

cpcloud requested changes Jun 20, 2019

View reviewed changes

cpcloud mentioned this pull request Jun 20, 2019

SUPP: Skip SQLAlchemy backend tests in connect method in backends.py #1846

Closed

xmnlab commented Jun 20, 2019

View reviewed changes

xmnlab force-pushed the add-more-geo-funcs-mapd branch from a6704ec to 7ebcecf Compare June 20, 2019 17:04

cpcloud reviewed Jun 20, 2019

View reviewed changes

cpcloud requested changes Jun 20, 2019

View reviewed changes

cpcloud added this to the 1.2.0 milestone Jun 20, 2019

xmnlab force-pushed the add-more-geo-funcs-mapd branch from 36c305b to d171051 Compare June 21, 2019 00:22

cpcloud reviewed Jun 21, 2019

View reviewed changes

Added new geo functions to omniscidb backend

9eaf660

xmnlab force-pushed the add-more-geo-funcs-mapd branch from 4fc33ed to 9eaf660 Compare June 24, 2019 03:26

cpcloud merged commit 8427e9e into ibis-project:master Jun 24, 2019

cpcloud self-assigned this Jun 24, 2019

cpcloud added feature Features or general enhancements omnisci labels Jun 24, 2019

xmnlab deleted the add-more-geo-funcs-mapd branch June 24, 2019 14:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT: Add new geospatial functions to OmniSciDB backend #1836

FEAT: Add new geospatial functions to OmniSciDB backend #1836

xmnlab commented Jun 17, 2019 •

edited

Loading

codecov bot commented Jun 19, 2019 •

edited

Loading

xmnlab commented Jun 20, 2019

cpcloud Jun 20, 2019

xmnlab Jun 20, 2019

cpcloud Jun 20, 2019

xmnlab Jun 20, 2019

xmnlab left a comment

xmnlab Jun 20, 2019

cpcloud commented Jun 20, 2019

xmnlab commented Jun 20, 2019

xmnlab commented Jun 20, 2019

cpcloud Jun 20, 2019

cpcloud Jun 20, 2019

xmnlab Jun 20, 2019

cpcloud Jun 20, 2019

cpcloud Jun 20, 2019

xmnlab commented Jun 20, 2019

xmnlab commented Jun 20, 2019

xmnlab commented Jun 21, 2019

cpcloud Jun 21, 2019

xmnlab Jun 21, 2019

cpcloud commented Jun 24, 2019

cpcloud commented Jun 24, 2019

xmnlab commented Jun 24, 2019

cpcloud commented Jun 24, 2019

FEAT: Add new geospatial functions to OmniSciDB backend #1836

FEAT: Add new geospatial functions to OmniSciDB backend #1836

Conversation

xmnlab commented Jun 17, 2019 • edited Loading

codecov bot commented Jun 19, 2019 • edited Loading

Codecov Report

xmnlab commented Jun 20, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmnlab left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Jun 20, 2019

xmnlab commented Jun 20, 2019

xmnlab commented Jun 20, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmnlab commented Jun 20, 2019

xmnlab commented Jun 20, 2019

xmnlab commented Jun 21, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Jun 24, 2019

cpcloud commented Jun 24, 2019

xmnlab commented Jun 24, 2019

cpcloud commented Jun 24, 2019

xmnlab commented Jun 17, 2019 •

edited

Loading

codecov bot commented Jun 19, 2019 •

edited

Loading