[MapD] Added Geospatial functions #1678

xmnlab · 2018-12-18T20:17:07Z

This PR solves #1665 and solves #1707

Add Geo Spatial functions on the main structure and define these functions inside MapD backend.

References:

Depends on #1666 ( PR 1666 was used as base for the current PR)

Geospatial functions

Geometry/Geography Constructors
- ST_GeomFromText(WKT) - using literals
- ST_GeogFromText(WKT) - using literals
Geometry Editors
- ~~ST_Transform (Returns a new geometry with its coordinates transformed to a different spatial reference system.)~~
- ~~ST_SetSRID (Sets the SRID on a geometry to a particular integer value.)~~
Geometry Accessors
- ST_X (Return the X coordinate of the point, or NULL if not available. Input must be a point.)
- ST_Y (Return the Y coordinate of the point, or NULL if not available. Input must be a point.)
- ST_XMin (Returns Y minima of a bounding box 2d or 3d or a geometry.)
- ST_XMax (Returns X maxima of a bounding box 2d or 3d or a geometry.)
- ST_YMin (Returns Y minima of a bounding box 2d or 3d or a geometry.)
- ST_YMax (Returns Y maxima of a bounding box 2d or 3d or a geometry.)
- ST_StartPoint (Returns the first point of a LINESTRING geometry as a POINT or NULL if the input parameter is not a LINESTRING.)
- ST_EndPoint (Returns the last point of a LINESTRING geometry as a POINT or NULL if the input parameter is not a LINESTRING.)
- ST_PointN (Return the Nth point in a single linestring in the geometry. Negative values are counted backwards from the end of the LineString, so that -1 is the last point. Returns NULL if there is no linestring in the geometry.)
- ST_NPoints (Return the number of points in a geometry. Works for all geometries.)
- ST_NRings (If the geometry is a polygon or multi-polygon returns the number of rings. It counts the outer rings as well.)
- ST_SRID (Returns the spatial reference identifier for the ST_Geometry)
Spatial Relationships and Measurements
- ST_Distance
- ST_Contains
- ST_Area
- ST_Perimeter
- ST_Length
- ST_MaxDistance
Extra
- ~~CastToGeography~~ TODO: will be added in a new PR.

codecov · 2018-12-18T20:44:29Z

Codecov Report

Merging #1678 into master will decrease coverage by 2.51%.
The diff coverage is 94.06%.

@@            Coverage Diff             @@
##           master    #1678      +/-   ##
==========================================
- Coverage      90%   87.49%   -2.52%     
==========================================
  Files         186      188       +2     
  Lines       27486    27701     +215     
  Branches     2344     2351       +7     
==========================================
- Hits        24739    24236     -503     
- Misses       2338     3055     +717     
- Partials      409      410       +1

Impacted Files	Coverage Δ
ibis/tests/all/test_geospatial.py	`100% <100%> (ø)`
ibis/mapd/operations.py	`73.59% <100%> (+0.99%)`	⬆️
ibis/expr/tests/test_geospatial.py	`100% <100%> (ø)`
ibis/expr/rules.py	`99.4% <100%> (ø)`	⬆️
ibis/expr/operations.py	`95.73% <100%> (+0.12%)`	⬆️
ibis/mapd/client.py	`51.02% <100%> (+0.16%)`	⬆️
ibis/tests/backends.py	`90.82% <100%> (+1.05%)`	⬆️
ibis/tests/util.py	`100% <100%> (ø)`	⬆️
ibis/expr/tests/conftest.py	`94.11% <100%> (+0.78%)`	⬆️
ibis/expr/api.py	`93.53% <100%> (+0.06%)`	⬆️
... and 24 more

xmnlab · 2018-12-19T00:50:24Z

@kszucs @cpcloud

I added Geo prefix for all new functions and classes. should ST_ be better?

xmnlab · 2018-12-31T15:12:33Z

@cpcloud @kszucs any feedback?

kszucs · 2018-12-31T16:24:30Z

@xmnlab Please add more tests. Then it'll be easier to see whether the API is convenient enough or still needs some tuning. In general it looks good though.

xmnlab · 2018-12-31T16:54:14Z

@kszucs That sounds good, thanks for the feedback. I will work on that today.

xmnlab · 2019-01-03T22:57:07Z

@cpcloud @kszucs I am adding a new table for geo data (https://github.com/ibis-project/ibis/blob/c4572177b45420343b2580396460e9e5ca8f66fd/ci/ibis-testing-data/geo.csv)

it seems download function tries to get data files from https://storage.googleapis.com/ibis-testing-data, is it right ? how can I save geo.csv there?

is there another way to do that and avoid using a "external" storage? maybe creating a new repo just for data on ibis-project.

xmnlab · 2019-01-09T23:03:39Z

@cpcloud @kszucs any thoughts about the previous comment (#1678 (comment)) ?

xmnlab · 2019-01-14T03:23:42Z

@cpcloud @kszucs

it seems download function tries to get data files from https://storage.googleapis.com/ibis-testing-data, is it right ? how can I save geo.csv there?

is there another way to do that and avoid using a "external" storage? maybe creating a new repo just for data on ibis-project.

xmnlab · 2019-01-21T03:30:25Z

@cpcloud @kszucs any idea about this?

kszucs · 2019-01-31T14:19:13Z

@xmnlab please rebase

xmnlab · 2019-02-06T02:43:47Z

hey @kszucs @cpcloud any feedback about including geo.csv on tests?

kszucs · 2019-02-22T10:17:40Z

ci/datamgr.py

+    # get data for geospatial support tests
+    geo_url = (
+          'https://raw.githubusercontent.com/Quansight/ibis/'
+          'geospatial_functions/ci/ibis-testing-data/geo.csv'


Please create a PR to https://github.com/ibis-project/testing-data with the geo.csv added and change the download script to fetch https://api.github.com/repos/ibis-project/testing-data/tarball/master

@kszucs that sounds pretty good .. I am going to do that now 👍

PR opened ibis-project/testing-data#1

I will now change download function

kszucs · 2019-02-22T10:18:48Z

@xmnlab Please rebase. Is this PR still work in progress?

xmnlab · 2019-02-22T15:14:20Z

@kszucs OK I am going to rebase that now. thanks!
It is missing just 3 new methods. I was working to to that .. but my mapd on my environment was not running good with geospatial ... I cleaned my docker environment here and it seems it is working now .. I am going to add these new functions and their functions now :) thanks!

xmnlab · 2019-02-22T22:37:19Z

code rebased.

xmnlab · 2019-02-23T17:19:18Z

hey @kszucs I think I will work on these 3 functions in a new PR. I will remove the on the title [wip] and I am working to fix the error on CI related to the testing data folder.

xmnlab · 2019-02-23T17:27:51Z

@kszucs is it possible to remove the jobs on circle ci related to python35?

kszucs · 2019-02-23T17:42:16Z

@xmnlab I think those jobs are stuck, because this PR was created before their removals.

xmnlab · 2019-02-23T20:06:49Z

oh I see .. thanks for the explanation @kszucs

I think it is done for a review 👍

kszucs · 2019-02-23T20:17:53Z

First of all, We're going to need more tests. Please improve the coverage.

xmnlab · 2019-02-23T21:11:06Z

@kszucs sounds good. I am working on that now! thanks!

xmnlab · 2019-02-23T23:36:16Z

hey @kszucs I added tests for each function .. I just need to fix 3 tests/functions (tomorrow) ... if you have any tip about improving this test code coverage let me know. thanks!

kszucs · 2019-02-24T09:19:50Z

ibis/tests/all/test_geospatial.py

+
+
+@tu.skipifnot_backend(MapD)
+def test_geo_op(backend, geo):


@xmnlab Could You decompose and/or pytest.parametrize it to multiple test functions?

sounds good @kszucs . thanks for the feedback. I am working on that.

xmnlab · 2019-02-25T13:37:20Z

@kszucs it seems codecov now is green :) I think it is ok for a new review :)
I am not so happy if some tests .. maybe it could be improved .. but I think we can do it in a new PR.
let me know your thoughts 👍

xmnlab · 2019-03-02T00:10:31Z

@kszucs thanks for the feedback! I am going to work on that now :)

xmnlab · 2019-03-02T06:39:02Z

@kszucs I made the changes you suggested.

let me know any thought about the last changes :)

xmnlab · 2019-03-06T12:00:33Z

@kszucs I am back! thansk for the commit! do you already know about AttributeError: module 'pyarrow' has no attribute 'compat' ?

kszucs · 2019-03-06T13:12:40Z

@xmnlab I gave You the wrong github url, sorry for that :)
I'm looking at the pyarrow import issue.

xmnlab · 2019-03-06T15:26:18Z

it seems that impyla v0.14.2.2 (latest release, 28 days ago) doesn't use run_async:

docker-compose run -e LOGLEVEL=DEBUG ibis ci/load-data.sh
Traceback (most recent call last):
  File "ci/datamgr.py", line 17, in <module>
    import ibis
  File "/ibis/ibis/__init__.py", line 32, in <module>
    import ibis.impala.api as impala  # noqa: F401
  File "/ibis/ibis/impala/api.py", line 1, in <module>
    from ibis.impala.client import (ImpalaConnection,  # noqa: F401
  File "/ibis/ibis/impala/client.py", line 28, in <module>
    from ibis.impala.compat import impyla, ImpylaError, HS2Error
  File "/ibis/ibis/impala/compat.py", line 17, in <module>
    import impala.dbapi as impyla  # noqa
  File "/opt/conda/lib/python3.7/site-packages/impala/dbapi.py", line 28, in <module>
    import impala.hiveserver2 as hs2
  File "/opt/conda/lib/python3.7/site-packages/impala/hiveserver2.py", line 340
    async=True)

if is not possible to ask for a new release there, maybe would be good to keep the url for the commit as you have made before.

xmnlab · 2019-03-06T17:03:53Z

after this PR be merged I will create an issue about the improvement of documentation about geo spatial data types and functions with explanation and examples.

xmnlab · 2019-03-06T17:15:49Z

not sure if this is relevant for this PR but pymapd v0.8.0 should be available in some minutes (conda-forge/pymapd-feedstock#30)

kszucs · 2019-03-06T21:00:47Z

Cool, thanks @xmnlab!

xmnlab · 2019-03-06T22:09:44Z

@kszucs do you think it is ready to be merged?

kszucs · 2019-03-06T22:19:01Z

It is green so yes! Thanks for the hard work @xmnlab!

xmnlab · 2019-03-06T22:19:49Z

thank you so much @kszucs for all the help!

kszucs added the expressions Issues or PRs related to the expression API label Dec 31, 2018

xmnlab force-pushed the geospatial_functions branch from 05da7b2 to f1271c0 Compare February 6, 2019 01:05

kszucs reviewed Feb 22, 2019

View reviewed changes

xmnlab force-pushed the geospatial_functions branch from 3e0fa42 to 72a8990 Compare February 22, 2019 21:32

xmnlab force-pushed the geospatial_functions branch from 560386c to 47abb55 Compare February 23, 2019 17:14

xmnlab changed the title ~~[MapD] [WIP] Added Geospatial functions~~ [MapD] Added Geospatial functions Feb 23, 2019

xmnlab force-pushed the geospatial_functions branch from 4ce593d to 41be8dc Compare February 23, 2019 23:32

kszucs reviewed Feb 24, 2019

View reviewed changes

xmnlab force-pushed the geospatial_functions branch 4 times, most recently from 6ec0c65 to d2ac169 Compare March 2, 2019 05:19

Added more tests

4c3969c

xmnlab force-pushed the geospatial_functions branch from d2ac169 to 4c3969c Compare March 2, 2019 06:13

use the zip github endpoint to download the repository

bb4c8f4

kszucs added 2 commits March 6, 2019 14:59

more robost testing data download script; updated requirements

e4d4410

fox download path in docker-compose

2cc2285

use pkg_resources to get pymapd's version

a1e177d

xmnlab and others added 2 commits March 6, 2019 16:40

Added pymapd dependence from conda

a96c0fd

remove IBIS_TEST_DOWNLOAD_DIRECTORY from the docs container as well

44dfaa6

update docs conda dependencies

4f94baf

kszucs approved these changes Mar 6, 2019

View reviewed changes

kszucs closed this in 8842547 Mar 6, 2019

This was referenced Mar 6, 2019

Update the datamgr.py script to download the testing data from github #1707

Closed

Add Ibis support for geospatial data types #1665

Closed

xmnlab deleted the geospatial_functions branch March 6, 2019 22:21

ian-r-rose mentioned this pull request May 15, 2019

PostGIS support #1786

Closed



		@tu.skipifnot_backend(MapD)
		def test_geo_op(backend, geo):

[MapD] Added Geospatial functions #1678

[MapD] Added Geospatial functions #1678

Conversation

xmnlab commented Dec 18, 2018 • edited Loading

Geospatial functions

codecov bot commented Dec 18, 2018 • edited Loading

Codecov Report

xmnlab commented Dec 19, 2018

xmnlab commented Dec 31, 2018

kszucs commented Dec 31, 2018

xmnlab commented Dec 31, 2018

xmnlab commented Jan 3, 2019

xmnlab commented Jan 9, 2019

xmnlab commented Jan 14, 2019

xmnlab commented Jan 21, 2019

kszucs commented Jan 31, 2019

xmnlab commented Feb 6, 2019

kszucs Feb 22, 2019

Choose a reason for hiding this comment

xmnlab Feb 22, 2019

Choose a reason for hiding this comment

xmnlab Feb 22, 2019

Choose a reason for hiding this comment

xmnlab Feb 22, 2019

Choose a reason for hiding this comment

kszucs Feb 22, 2019

Choose a reason for hiding this comment

kszucs commented Feb 22, 2019 • edited Loading

xmnlab commented Feb 22, 2019

xmnlab commented Feb 22, 2019

xmnlab commented Feb 23, 2019

xmnlab commented Feb 23, 2019

kszucs commented Feb 23, 2019 • edited Loading

xmnlab commented Feb 23, 2019

kszucs commented Feb 23, 2019

xmnlab commented Feb 23, 2019

xmnlab commented Feb 23, 2019

kszucs Feb 24, 2019 • edited Loading

Choose a reason for hiding this comment

xmnlab Feb 24, 2019

Choose a reason for hiding this comment

xmnlab commented Feb 25, 2019

xmnlab commented Mar 2, 2019

xmnlab commented Mar 2, 2019

xmnlab commented Mar 6, 2019

kszucs commented Mar 6, 2019

xmnlab commented Mar 6, 2019

xmnlab commented Mar 6, 2019 • edited Loading

xmnlab commented Mar 6, 2019

kszucs commented Mar 6, 2019

xmnlab commented Mar 6, 2019

kszucs commented Mar 6, 2019

xmnlab commented Mar 6, 2019

xmnlab commented Dec 18, 2018 •

edited

Loading

codecov bot commented Dec 18, 2018 •

edited

Loading

kszucs commented Feb 22, 2019 •

edited

Loading

kszucs commented Feb 23, 2019 •

edited

Loading

kszucs Feb 24, 2019 •

edited

Loading

xmnlab commented Mar 6, 2019 •

edited

Loading