Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT: Add support for case/when in PySpark backend #2610

Merged
merged 2 commits into from
Feb 4, 2021

Conversation

icexelloss
Copy link
Contributor

@icexelloss icexelloss commented Feb 3, 2021

What's the change

  • Added support for case/when clause in PySpark backend
  • Fixed a NPE issue with substr in PySpark backend
  • Switch the implementation of substr in PySpark backend from UDF to built-in function and raise errors for unsupported case (when start and length is not static)

How is this tested

Add test under test_string.py for substr
Add test under test_generic.py for case/when

@icexelloss icexelloss changed the title FEAT: Add support for case/where in PySpark backend FEAT: Add support for case/when in PySpark backend Feb 3, 2021
@icexelloss icexelloss force-pushed the pyspark-substr-null branch 8 times, most recently from 4c7d0cf to fc6239a Compare February 3, 2021 22:36
@jreback jreback added pyspark The Apache PySpark backend backends - spark labels Feb 3, 2021
@jreback jreback added this to the Next release milestone Feb 3, 2021
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls add a release note as well

if isinstance(start, pyspark.sql.Column) or isinstance(
length, pyspark.sql.Column
):
raise NotImplementedError(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this case tested?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is covered by this:

https://github.com/ibis-project/ibis/blob/master/ibis/backends/tests/test_string.py#L216

When catching NotImplemenntedError, the test will be xfailed.

@jreback jreback merged commit a4f0aa0 into ibis-project:master Feb 4, 2021
gerrymanoim added a commit to gerrymanoim/ibis that referenced this pull request Feb 9, 2021
Looks like it was accidentally dropped in ibis-project#2610
gerrymanoim added a commit to gerrymanoim/ibis that referenced this pull request Feb 10, 2021
Looks like it was accidentally dropped in ibis-project#2610
jreback pushed a commit that referenced this pull request Feb 10, 2021
Looks like it was accidentally dropped in #2610
@cpcloud cpcloud removed this from the Next release milestone Jan 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pyspark The Apache PySpark backend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants