Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: Fix issues with context adjustment for filter with PySpark backend #2693

Merged
merged 4 commits into from
Mar 23, 2021
Merged

BUG: Fix issues with context adjustment for filter with PySpark backend #2693

merged 4 commits into from
Mar 23, 2021

Conversation

icexelloss
Copy link
Contributor

@icexelloss icexelloss commented Mar 19, 2021 •
edited
Loading

Mixture of a few minor issues with context adjustment for pyspark:

Main change:

  • Fix Selection node that is filter only so that we pass the original timecontext correctly.

Minor changes:

  • Localize tz-naive time context with spark.sql.session.localTimezone. This is consistent with how does Spark treats timestamps in the DataFrame.
  • Only trim if the context and adjusted_context are different
  • (Minor) Clean up test_timecontext.py
  • (Minor) Fix error in backends/pyspark/testconf.py (verified manually)

Tests

Add new test in test_timecontext.py

@icexelloss icexelloss added pyspark The Apache PySpark backend context adjustment labels Mar 19, 2021
@icexelloss icexelloss requested a review from jreback March 19, 2021 15:09
@icexelloss
Copy link
Contributor Author

cc @LeeTZ @jreback

jreback
jreback approved these changes Mar 22, 2021
View reviewed changes
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. can you add a whatsnew note.

@jreback jreback added this to the Next release milestone Mar 22, 2021
@jreback
Copy link
Contributor

jreback commented Mar 22, 2021

@icexelloss can you merge master one more time

@icexelloss icexelloss force-pushed the pyspark-backend-context-adjustment-filter branch from 22d873d to 66060f8 Compare March 22, 2021 21:09
@icexelloss icexelloss changed the title BUG: Fix issues with context adjustment for filter BUG: Fix issues with context adjustment for filter with PySpark backennd Mar 22, 2021
@icexelloss icexelloss changed the title BUG: Fix issues with context adjustment for filter with PySpark backennd BUG: Fix issues with context adjustment for filter with PySpark backend Mar 22, 2021
@jreback jreback merged commit b13ff82 into ibis-project:master Mar 23, 2021
@jreback
Copy link
Contributor

jreback commented Mar 23, 2021

thanks @icexelloss

@cpcloud cpcloud removed this from the Next release milestone Jan 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jreback jreback jreback approved these changes

Assignees
No one assigned
Labels
pyspark The Apache PySpark backend
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@icexelloss @jreback @cpcloud