FEAT: Adding more window operations to OmniSciDB #1976

xmnlab · 2019-09-22T21:49:42Z

In this PR, added the follow window operations:

DenseRank
PercentRank (that is translated to cume_dist)
RowNumber
MinRank
Count

xmnlab · 2019-11-26T22:13:23Z

Currently CI is raising some errors:

py36 building:

>                   raise Exception("Java gateway process exited before sending its port number")
E                   Exception: Java gateway process exited before sending its port number

/opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyspark/java_gateway.py:108: Exception
---------------------------- Captured stderr setup -----------------------------
/usr/bin/env: ‘bash’: No such file or directory
__________________ ERROR at setup of test_array_length_scalar __________________
[gw0] linux -- Python 3.6.7 /opt/conda/envs/ibis-env/bin/python

    @pytest.fixture(scope='session')
    def client():
        from pyspark.sql import SparkSession
        import pyspark.sql.functions as F
    
>       session = SparkSession.builder.getOrCreate()

ibis/pyspark/tests/conftest.py:11: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyspark/sql/session.py:173: in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
/opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyspark/context.py:367: in getOrCreate
    SparkContext(conf=conf or SparkConf())
/opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyspark/context.py:133: in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
/opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyspark/context.py:316: in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
/opt/conda/envs/ibis-env/lib/python3.6/site-packages/pyspark/java_gateway.py:46: in launch_gateway
    return _launch_gateway(conf)

>           raise Exception((retcode, cmd))
E           Exception: (127, 'tar zc 80c3e427b01d4471a63a57e035a599f7 > 80c3e427b01d4471a63a57e035a599f7.tar.gz')

ibis/tests/test_filesystems.py:364: Exception
----------------------------- Captured stderr call -----------------------------
/bin/sh: 1: tar: not found

doc building

�[0;32m/opt/conda/envs/ibis-env/lib/python3.6/site-packages/IPython/utils/_process_posix.py�[0m in �[0;36msh�[0;34m(self)�[0m
�[1;32m     63�[0m             �[0mself�[0m�[0;34m.�[0m�[0m_sh�[0m �[0;34m=�[0m �[0mpexpect�[0m�[0;34m.�[0m�[0mwhich�[0m�[0;34m(�[0m�[0;34m'sh'�[0m�[0;34m)�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[1;32m     64�[0m             �[0;32mif�[0m �[0mself�[0m�[0;34m.�[0m�[0m_sh�[0m �[0;32mis�[0m �[0;32mNone�[0m�[0;34m:�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[0;32m---> 65�[0;31m                 �[0;32mraise�[0m �[0mOSError�[0m�[0;34m(�[0m�[0;34m'"sh" shell not found'�[0m�[0;34m)�[0m�[0;34m�[0m�[0;34m�[0m�[0m
�[0m�[1;32m     66�[0m �[0;34m�[0m�[0m
�[1;32m     67�[0m         �[0;32mreturn�[0m �[0mself�[0m�[0;34m.�[0m�[0m_sh�[0m�[0;34m�[0m�[0;34m�[0m�[0m

�[0;31mOSError�[0m: "sh" shell not found
OSError: "sh" shell not found


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/sphinx/cmd/build.py", line 284, in build_main
    app.build(args.force_all, filenames)
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/sphinx/application.py", line 337, in build
    self.builder.build_update()
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 326, in build_update
    len(to_build))
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 339, in build
    updated_docnames = set(self.read())
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 445, in read
    self._read_serial(docnames)
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 467, in _read_serial
    self.read_doc(docname)
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 511, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/sphinx/io.py", line 323, in read_doc
    pub.publish()
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/docutils/core.py", line 217, in publish
    self.settings)
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/sphinx/io.py", line 116, in read
    self.parse()
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/opt/conda/envs/ibis-env/lib/python3.6/site-packages/nbsphinx.py", line 865, in parse
    raise NotebookError('\n'.join(lines))
nbsphinx.NotebookError: CellExecutionError in notebooks/tutorial/5-IO-Create-Insert-External-Data.ipynb:
------------------
!rm -rf parquet_dir/
hdfs.get('/__ibis/ibis-testing-data/parquet/functional_alltypes', 'parquet_dir')
------------------

xmnlab · 2019-12-12T14:39:53Z

@jreback any more thought about this PR? let me know and if it is OK .. I will rebase to fix the conflict on release.rst

xmnlab · 2020-01-07T16:47:29Z

@jreback any more thought about this PR? let me know and if it is OK .. I will rebase to fix the conflict on release.rst

xmnlab · 2020-01-20T21:29:28Z

hey @jreback ! a gentle reminder about this PR :)

ibis/tests/all/test_window.py

jreback · 2020-01-20T22:13:47Z

ibis/tests/all/test_window.py

+            result.extend(sub_result)
+            if diff > 0:
+                diff -= 1
+        return pd.Series(result, index=x.index)


why are you evaluating directly in the api like this?

are the ntile ops not defined?

these should be dispatched generically, not done like this.

pandas doesn't have ntile operation.
the most similar operation isqcut but it seems there are some differences.

what is your recommendation in this case?

jreback · 2020-01-22T10:35:40Z

ibis/tests/all/test_window.py

+    """
+    # internal ntile function
+    def _ntile(x: pandas.Series, bucket: int):
+        n = x.shape[0]


why are you actually trying to execute operations inside api definitions?

this is an anti pattern in ibis

further these are actually definitions of pandas ops and not omnisci db native ops

so am puzzled what you are attempting to accomplish

the way to do this is to define these as pandas ops
the api then should return these ops and ibis will dispatch to them

I just created that because AFAIK currently there is no way to test ibis ntile operation ...
so that would be just a temporary way to test ibis ntile operations
just for clarification, by pandas ops do you mean 1) ibis pandas ops or really 2) pandas ops?

if it is 2) .. as it could take an extra time, probably I would prefer to remove ntile here and implement that in a different PR.

I am not sure you are following. This completely breaks the ibis style & philosophy. I am really not sure what your end goal is Does omniscidb actually have these functions? if so certainly you can add them that backend, but this shouldn't have anything to do with pandas.

yes .. maybe I am not following you ... what do you mean by api definitions? because I added that just to test_window.py it shouldn't be available into the API.

I can remove this code with no problem .. my question is .. how can I test omniscidb ntile operation? normally we just compare the result from the backend to the result from pure pandas operation.

I have no ideal what these operations actually do. Some documentation and examples would go along way. You need to write a much more explict test.

sounds fair @jreback thanks for the feedback ... I will probably move ntile to a new PR and I will do it for sure. thanks!

xmnlab · 2020-01-28T19:07:12Z

@jreback I removed the ntile operation from this PR. it is done for a new review. thanks!

jreback · 2020-01-29T18:51:36Z

@xmnlab pls rebase. this looks fine. you missed my point above. I don't think is is necessary to add any operations to pandas proper itself (and will likely be rejected). But on the ibis pandas back-end I think you could simply implement these and would provide the good testing comparison.

xmnlab · 2020-01-29T22:05:56Z

rebase done, thanks @jreback !

I just opened that issue there to see if there is any interesting from the pandas community for that implementation .. and also check if there are other ways to get the desired result :)

about your suggestion about to implement that on pandas backend .. that is reasonable, I will need to create a separated test for that ... because on tests/all it compares the result between each backend and pandas (pure) .. and in this new case it will check the results between each backend and pandas backend (let me know if I am missing anything ...)

again, thanks for the review and suggestions!

jreback · 2020-01-29T22:33:31Z

this is ok thanks @xmnlab

xmnlab mentioned this pull request Sep 22, 2019

Ibis: Current Window Function Support Quansight/omnisci#33

Closed

13 tasks

xmnlab changed the title ~~ENH: Adding more window operations to OmniSciDB~~ FEAT: Adding more window operations to OmniSciDB Oct 2, 2019

xmnlab force-pushed the Add-more-window-functions branch from ae0472a to 79c9a49 Compare November 18, 2019 19:36

xmnlab force-pushed the Add-more-window-functions branch from 9c72721 to db07745 Compare November 26, 2019 20:13

xmnlab force-pushed the Add-more-window-functions branch from db07745 to 4055e04 Compare December 6, 2019 02:53

xmnlab marked this pull request as ready for review December 6, 2019 14:49

xmnlab requested review from jreback and scottcode December 6, 2019 14:49

xmnlab assigned jreback Dec 12, 2019

jreback added omnisci window functions Issues or PRs related to window functions labels Jan 20, 2020

jreback added this to the Next Feature Release milestone Jan 20, 2020

jreback suggested changes Jan 20, 2020

View reviewed changes

xmnlab force-pushed the Add-more-window-functions branch 3 times, most recently from 7971a1a to 9871e6f Compare January 21, 2020 19:38

jreback suggested changes Jan 22, 2020

View reviewed changes

jreback removed this from the Next Feature Release milestone Jan 24, 2020

xmnlab requested a review from jreback January 28, 2020 01:40

xmnlab force-pushed the Add-more-window-functions branch from 9871e6f to a7a7eec Compare January 28, 2020 17:52

jreback added this to the Next Feature Release milestone Jan 29, 2020

ENH: Adding more window operations to OmniSciDB

ed0ecd0

xmnlab force-pushed the Add-more-window-functions branch from a7a7eec to ed0ecd0 Compare January 29, 2020 21:48

jreback approved these changes Jan 29, 2020

View reviewed changes

jreback merged commit 95e17c3 into ibis-project:master Jan 29, 2020

xmnlab deleted the Add-more-window-functions branch January 29, 2020 22:46

xmnlab mentioned this pull request Jan 29, 2020

[IBIS] Add NTile window operation Quansight/omnisci#88

Open

xmnlab mentioned this pull request Mar 6, 2020

OmnisciDB ops.IdenticalTo, ops.NTile are working via impala implementation #2112

Closed

xmnlab mentioned this pull request Mar 19, 2020

FEAT: Add Pandas and OmniSciDB NTile operation support #2146

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT: Adding more window operations to OmniSciDB #1976

FEAT: Adding more window operations to OmniSciDB #1976

xmnlab commented Sep 22, 2019 •

edited

Loading

xmnlab commented Nov 26, 2019

xmnlab commented Dec 12, 2019

xmnlab commented Jan 7, 2020

xmnlab commented Jan 20, 2020

jreback Jan 20, 2020

xmnlab Jan 21, 2020

jreback Jan 22, 2020

xmnlab Jan 22, 2020

jreback Jan 24, 2020

xmnlab Jan 24, 2020

jreback Jan 28, 2020

xmnlab Jan 28, 2020

xmnlab commented Jan 28, 2020

jreback commented Jan 29, 2020

xmnlab commented Jan 29, 2020

jreback commented Jan 29, 2020

FEAT: Adding more window operations to OmniSciDB #1976

FEAT: Adding more window operations to OmniSciDB #1976

Conversation

xmnlab commented Sep 22, 2019 • edited Loading

xmnlab commented Nov 26, 2019

xmnlab commented Dec 12, 2019

xmnlab commented Jan 7, 2020

xmnlab commented Jan 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmnlab commented Jan 28, 2020

jreback commented Jan 29, 2020

xmnlab commented Jan 29, 2020

jreback commented Jan 29, 2020

xmnlab commented Sep 22, 2019 •

edited

Loading