ENH: window operations for pyspark backend #1945
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #1945 +/- ##
==========================================
- Coverage 87.68% 86.13% -1.55%
==========================================
Files 93 93
Lines 16971 17068 +97
Branches 2145 2157 +12
==========================================
- Hits 14881 14702 -179
- Misses 1681 1961 +280
+ Partials 409 405 -4
|
toryhaavik
reviewed
Sep 6, 2019
ibis/pyspark/compiler.py
Outdated
|
|
||
|
|
||
| @compiles(ops.Lead) | ||
| def compile_lead(t, expr, scope, *, window, **kwargs): |
There was a problem hiding this comment.
since this and Lag are so similar, can we factor them into a single function that takes the pyspark function as a parameter?
ibis/tests/all/test_window.py
Outdated
| @@ -3,7 +3,8 @@ | |||
|
|
|||
| import ibis | |||
| import ibis.common.exceptions as com | |||
| from ibis.tests.backends import Csv, OmniSciDB, Pandas, Parquet | |||
| from ibis.tests.backends import Csv, Impala, OmniSciDB, Pandas, Parquet, \ | |||
There was a problem hiding this comment.
i think we generally prefer wrapping these in parentheses rather than \
added 4 commits
September 6, 2019 16:38
…ommit but don't pass with non pyspark backends
|
Rebased on top of master. |
xmnlab
reviewed
Sep 7, 2019
ibis/pyspark/compiler.py
Outdated
| return compile_aggregator(t, expr, scope, F.max, context) | ||
|
|
||
| def fn(col): | ||
| if "window" in kwargs: |
There was a problem hiding this comment.
I think the default for string used in ibis is single quotes. probably we should keep all string using single quotes.
toryhaavik
approved these changes
Sep 9, 2019
costrouc
pushed a commit
to costrouc/ibis
that referenced
this pull request
Oct 10, 2019
Implemented window operations for PySpark backend to pass all tests in `ibis/tests/all/test_window.py`: - `WindowOp` - `Lag` - `Lead` - `MinRank` - `DenseRank` - `NTile` - `FirstValue` - `LastValue` - `RowNumber` - `CumulativeSum` - `CumulativeMean` - `CumulativeMin` - `CumulativeMax` Also enhanced select aggregation operations (e.g. `Any`, `NotAny`, `All`, `NotAll`, `Count`, `Max`, `Min`, `Mean`, `Sum`) to be interoperable with windows. Author: Hyonjee <hyonjee.joo@twosigma.com> Closes ibis-project#1945 from hjoo/pyspark-window and squashes the following commits: 54a3405 [Hyonjee] change double quotes to single quotes in pyspark compiler.py 7cb2291 [Hyonjee] add helper methods for pyspark shift and cumulative window ops, refactor import line 5c82b19 [Hyonjee] remove extra test_window lead/lag tests that were added in previous commit but don't pass with non pyspark backends 7ac2dd8 [Hyonjee] skip unsupported window tests for OmniSciDB 419c309 [Hyonjee] fix test_window() in ibis/pyspark/tests/test_basic.py and remove xfail 7bd6585 [Hyonjee] window operations for pyspark backend
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implemented window operations for PySpark backend to pass all tests in
ibis/tests/all/test_window.py:WindowOpLagLeadMinRankDenseRankNTileFirstValueLastValueRowNumberCumulativeSumCumulativeMeanCumulativeMinCumulativeMaxAlso enhanced select aggregation operations (e.g.
Any,NotAny,All,NotAll,Count,Max,Min,Mean,Sum) to be interoperable with windows.