Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: filter for PySpark backend #1943

Merged
merged 3 commits into from
Sep 9, 2019
Merged

Conversation

hjoo
Copy link
Contributor

@hjoo hjoo commented Aug 30, 2019

Introduces predicate handling in ops.Selection for filter() functionality.

Also implemented a few missing comparison operations:

  • And
  • Or
  • NotEquals
  • Less
  • LessEquals

@codecov
Copy link

codecov bot commented Aug 30, 2019

Codecov Report

Merging #1943 into master will decrease coverage by 1.63%.
The diff coverage is 81.81%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1943      +/-   ##
==========================================
- Coverage   87.77%   86.13%   -1.64%     
==========================================
  Files          93       93              
  Lines       17068    17088      +20     
  Branches     2157     2159       +2     
==========================================
- Hits        14981    14719     -262     
- Misses       1679     1964     +285     
+ Partials      408      405       -3
Impacted Files Coverage Δ
ibis/pyspark/compiler.py 95.82% <81.81%> (-0.23%) ⬇️
ibis/bigquery/client.py 41.1% <0%> (-53.39%) ⬇️
ibis/bigquery/compiler.py 59.92% <0%> (-37.5%) ⬇️
ibis/bigquery/udf/api.py 80.48% <0%> (-14.64%) ⬇️
ibis/impala/compiler.py 91.23% <0%> (-5.2%) ⬇️
ibis/pandas/dispatch.py 95.45% <0%> (-4.55%) ⬇️
ibis/pandas/client.py 85.54% <0%> (-3.47%) ⬇️
ibis/bigquery/api.py 63.33% <0%> (-3.34%) ⬇️
ibis/expr/schema.py 90.1% <0%> (-1.1%) ⬇️
ibis/sql/compiler.py 94.26% <0%> (-0.39%) ⬇️
... and 3 more

Copy link
Contributor

@toryhaavik toryhaavik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a request to file a TODO issue

ibis/pyspark/compiler.py Outdated Show resolved Hide resolved
@hjoo
Copy link
Contributor Author

hjoo commented Sep 6, 2019

Most recent push is a rebase on top of master.

@toryhaavik
Copy link
Contributor

This LGTM, @hjoo please just rebase on top of master, and i'll merge on green

@hjoo
Copy link
Contributor Author

hjoo commented Sep 9, 2019

Rebase complete

Copy link
Contributor

@toryhaavik toryhaavik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @hjoo!

@toryhaavik toryhaavik merged commit efea4b3 into ibis-project:master Sep 9, 2019
costrouc pushed a commit to costrouc/ibis that referenced this pull request Oct 10, 2019
Introduces predicate handling in `ops.Selection` for `filter()`
functionality.    Also implemented a few missing comparison
operations:    - `And`  - `Or`  - `NotEquals`  - `Less`  -
`LessEquals`
Author: Hyonjee <hyonjee.joo@twosigma.com>

Closes ibis-project#1943 from hjoo/pyspark-filter and squashes the following commits:

42c2a3c [Hyonjee] parametrize pyspark test_filter
0480921 [Hyonjee] added ibis issue for missing sort_keys handling for pyspark selection op
28e6ad6 [Hyonjee] pyspark backend filter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants