Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MapD] Added Geospatial functions #1678

Closed
wants to merge 8 commits into from

Conversation

Projects
None yet
2 participants
@xmnlab
Copy link
Collaborator

commented Dec 18, 2018

This PR solves #1665 and solves #1707

Add Geo Spatial functions on the main structure and define these functions inside MapD backend.

References:

Depends on #1666 ( PR 1666 was used as base for the current PR)

Geospatial functions

  • Geometry/Geography Constructors
    • ST_GeomFromText(WKT) - using literals
    • ST_GeogFromText(WKT) - using literals
  • Geometry Editors
    • ST_Transform (Returns a new geometry with its coordinates transformed to a different spatial reference system.)
    • ST_SetSRID (Sets the SRID on a geometry to a particular integer value.)
  • Geometry Accessors
    • ST_X (Return the X coordinate of the point, or NULL if not available. Input must be a point.)
    • ST_Y (Return the Y coordinate of the point, or NULL if not available. Input must be a point.)
    • ST_XMin (Returns Y minima of a bounding box 2d or 3d or a geometry.)
    • ST_XMax (Returns X maxima of a bounding box 2d or 3d or a geometry.)
    • ST_YMin (Returns Y minima of a bounding box 2d or 3d or a geometry.)
    • ST_YMax (Returns Y maxima of a bounding box 2d or 3d or a geometry.)
    • ST_StartPoint (Returns the first point of a LINESTRING geometry as a POINT or NULL if the input parameter is not a LINESTRING.)
    • ST_EndPoint (Returns the last point of a LINESTRING geometry as a POINT or NULL if the input parameter is not a LINESTRING.)
    • ST_PointN (Return the Nth point in a single linestring in the geometry. Negative values are counted backwards from the end of the LineString, so that -1 is the last point. Returns NULL if there is no linestring in the geometry.)
    • ST_NPoints (Return the number of points in a geometry. Works for all geometries.)
    • ST_NRings (If the geometry is a polygon or multi-polygon returns the number of rings. It counts the outer rings as well.)
    • ST_SRID (Returns the spatial reference identifier for the ST_Geometry)
  • Spatial Relationships and Measurements
    • ST_Distance
    • ST_Contains
    • ST_Area
    • ST_Perimeter
    • ST_Length
    • ST_MaxDistance
  • Extra
    • CastToGeography TODO: will be added in a new PR.
@codecov

This comment has been minimized.

Copy link

commented Dec 18, 2018

Codecov Report

Merging #1678 into master will decrease coverage by 2.51%.
The diff coverage is 94.06%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1678      +/-   ##
==========================================
- Coverage      90%   87.49%   -2.52%     
==========================================
  Files         186      188       +2     
  Lines       27486    27701     +215     
  Branches     2344     2351       +7     
==========================================
- Hits        24739    24236     -503     
- Misses       2338     3055     +717     
- Partials      409      410       +1
Impacted Files Coverage Δ
ibis/tests/all/test_geospatial.py 100% <100%> (ø)
ibis/mapd/operations.py 73.59% <100%> (+0.99%) ⬆️
ibis/expr/tests/test_geospatial.py 100% <100%> (ø)
ibis/expr/rules.py 99.4% <100%> (ø) ⬆️
ibis/expr/operations.py 95.73% <100%> (+0.12%) ⬆️
ibis/mapd/client.py 51.02% <100%> (+0.16%) ⬆️
ibis/tests/backends.py 90.82% <100%> (+1.05%) ⬆️
ibis/tests/util.py 100% <100%> (ø) ⬆️
ibis/expr/tests/conftest.py 94.11% <100%> (+0.78%) ⬆️
ibis/expr/api.py 93.53% <100%> (+0.06%) ⬆️
... and 24 more
@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Dec 19, 2018

@kszucs @cpcloud

I added Geo prefix for all new functions and classes. should ST_ be better?

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Dec 31, 2018

@cpcloud @kszucs any feedback?

@kszucs

This comment has been minimized.

Copy link
Member

commented Dec 31, 2018

@xmnlab Please add more tests. Then it'll be easier to see whether the API is convenient enough or still needs some tuning. In general it looks good though.

@kszucs kszucs added the expressions label Dec 31, 2018

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Dec 31, 2018

@kszucs That sounds good, thanks for the feedback. I will work on that today.

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Jan 3, 2019

@cpcloud @kszucs I am adding a new table for geo data (https://github.com/ibis-project/ibis/blob/c4572177b45420343b2580396460e9e5ca8f66fd/ci/ibis-testing-data/geo.csv)

it seems download function tries to get data files from https://storage.googleapis.com/ibis-testing-data, is it right ? how can I save geo.csv there?

is there another way to do that and avoid using a "external" storage? maybe creating a new repo just for data on ibis-project.

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Jan 9, 2019

@cpcloud @kszucs any thoughts about the previous comment (#1678 (comment)) ?

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Jan 14, 2019

@cpcloud @kszucs

it seems download function tries to get data files from https://storage.googleapis.com/ibis-testing-data, is it right ? how can I save geo.csv there?

is there another way to do that and avoid using a "external" storage? maybe creating a new repo just for data on ibis-project.

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Jan 21, 2019

@cpcloud @kszucs any idea about this?

@kszucs

This comment has been minimized.

Copy link
Member

commented Jan 31, 2019

@xmnlab please rebase

@xmnlab xmnlab force-pushed the Quansight:geospatial_functions branch from 05da7b2 to f1271c0 Feb 6, 2019

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 6, 2019

hey @kszucs @cpcloud any feedback about including geo.csv on tests?

# get data for geospatial support tests
geo_url = (
'https://raw.githubusercontent.com/Quansight/ibis/'
'geospatial_functions/ci/ibis-testing-data/geo.csv'

This comment has been minimized.

Copy link
@kszucs

kszucs Feb 22, 2019

Member

Please create a PR to https://github.com/ibis-project/testing-data with the geo.csv added and change the download script to fetch https://api.github.com/repos/ibis-project/testing-data/tarball/master

This comment has been minimized.

Copy link
@xmnlab

xmnlab Feb 22, 2019

Author Collaborator

@kszucs that sounds pretty good .. I am going to do that now 👍

This comment has been minimized.

Copy link
@xmnlab

xmnlab Feb 22, 2019

Author Collaborator

This comment has been minimized.

Copy link
@xmnlab

xmnlab Feb 22, 2019

Author Collaborator

I will now change download function

This comment has been minimized.

Copy link
@kszucs

kszucs Feb 22, 2019

Member

Merged :)

@kszucs

This comment has been minimized.

Copy link
Member

commented Feb 22, 2019

@xmnlab Please rebase. Is this PR still work in progress?

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 22, 2019

@kszucs OK I am going to rebase that now. thanks!
It is missing just 3 new methods. I was working to to that .. but my mapd on my environment was not running good with geospatial ... I cleaned my docker environment here and it seems it is working now .. I am going to add these new functions and their functions now :) thanks!

@xmnlab xmnlab force-pushed the Quansight:geospatial_functions branch from 3e0fa42 to 72a8990 Feb 22, 2019

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 22, 2019

code rebased.

@xmnlab xmnlab force-pushed the Quansight:geospatial_functions branch from 560386c to 47abb55 Feb 23, 2019

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 23, 2019

hey @kszucs I think I will work on these 3 functions in a new PR. I will remove the on the title [wip] and I am working to fix the error on CI related to the testing data folder.

@xmnlab xmnlab changed the title [MapD] [WIP] Added Geospatial functions [MapD] Added Geospatial functions Feb 23, 2019

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 23, 2019

@kszucs is it possible to remove the jobs on circle ci related to python35?

@kszucs

This comment has been minimized.

Copy link
Member

commented Feb 23, 2019

@xmnlab I think those jobs are stuck, because this PR was created before their removals.

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 23, 2019

oh I see .. thanks for the explanation @kszucs

I think it is done for a review 👍

@kszucs

This comment has been minimized.

Copy link
Member

commented Feb 23, 2019

First of all, We're going to need more tests. Please improve the coverage.

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 23, 2019

@kszucs sounds good. I am working on that now! thanks!

@xmnlab xmnlab force-pushed the Quansight:geospatial_functions branch from 4ce593d to 41be8dc Feb 23, 2019

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 23, 2019

hey @kszucs I added tests for each function .. I just need to fix 3 tests/functions (tomorrow) ... if you have any tip about improving this test code coverage let me know. thanks!



@tu.skipifnot_backend(MapD)
def test_geo_op(backend, geo):

This comment has been minimized.

Copy link
@kszucs

kszucs Feb 24, 2019

Member

@xmnlab Could You decompose and/or pytest.parametrize it to multiple test functions?

This comment has been minimized.

Copy link
@xmnlab

xmnlab Feb 24, 2019

Author Collaborator

sounds good @kszucs . thanks for the feedback. I am working on that.

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 25, 2019

@kszucs it seems codecov now is green :) I think it is ok for a new review :)
I am not so happy if some tests .. maybe it could be improved .. but I think we can do it in a new PR.
let me know your thoughts 👍

@kszucs

This comment has been minimized.

Copy link
Member

commented Mar 1, 2019

@xmnlab added a couple of comments. I'll wait for your response, than merge. Nice PR BTW :)

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 2, 2019

@kszucs thanks for the feedback! I am going to work on that now :)

@xmnlab xmnlab force-pushed the Quansight:geospatial_functions branch 4 times, most recently from 6ec0c65 to d2ac169 Mar 2, 2019

@xmnlab xmnlab force-pushed the Quansight:geospatial_functions branch from d2ac169 to 4c3969c Mar 2, 2019

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 2, 2019

@kszucs I made the changes you suggested.

let me know any thought about the last changes :)

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 6, 2019

@kszucs I am back! thansk for the commit! do you already know about AttributeError: module 'pyarrow' has no attribute 'compat' ?

@kszucs

This comment has been minimized.

Copy link
Member

commented Mar 6, 2019

@xmnlab I gave You the wrong github url, sorry for that :)
I'm looking at the pyarrow import issue.

kszucs added some commits Mar 6, 2019

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 6, 2019

it seems that impyla v0.14.2.2 (latest release, 28 days ago) doesn't use run_async:

docker-compose run -e LOGLEVEL=DEBUG ibis ci/load-data.sh
Traceback (most recent call last):
  File "ci/datamgr.py", line 17, in <module>
    import ibis
  File "/ibis/ibis/__init__.py", line 32, in <module>
    import ibis.impala.api as impala  # noqa: F401
  File "/ibis/ibis/impala/api.py", line 1, in <module>
    from ibis.impala.client import (ImpalaConnection,  # noqa: F401
  File "/ibis/ibis/impala/client.py", line 28, in <module>
    from ibis.impala.compat import impyla, ImpylaError, HS2Error
  File "/ibis/ibis/impala/compat.py", line 17, in <module>
    import impala.dbapi as impyla  # noqa
  File "/opt/conda/lib/python3.7/site-packages/impala/dbapi.py", line 28, in <module>
    import impala.hiveserver2 as hs2
  File "/opt/conda/lib/python3.7/site-packages/impala/hiveserver2.py", line 340
    async=True)

if is not possible to ask for a new release there, maybe would be good to keep the url for the commit as you have made before.

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 6, 2019

after this PR be merged I will create an issue about the improvement of documentation about geo spatial data types and functions with explanation and examples.

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 6, 2019

not sure if this is relevant for this PR but pymapd v0.8.0 should be available in some minutes (conda-forge/pymapd-feedstock#30)

xmnlab and others added some commits Mar 6, 2019

@kszucs

This comment has been minimized.

Copy link
Member

commented Mar 6, 2019

Cool, thanks @xmnlab!

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 6, 2019

@kszucs do you think it is ready to be merged?

@kszucs

kszucs approved these changes Mar 6, 2019

@kszucs

This comment has been minimized.

Copy link
Member

commented Mar 6, 2019

It is green so yes! Thanks for the hard work @xmnlab!

@kszucs kszucs closed this in 8842547 Mar 6, 2019

@xmnlab

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 6, 2019

thank you so much @kszucs for all the help!

@xmnlab xmnlab deleted the Quansight:geospatial_functions branch Mar 6, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.