ENH: Add support for date/time operations in PySpark backend #1974
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
icexelloss
reviewed
Sep 23, 2019
icexelloss
reviewed
Sep 23, 2019
icexelloss
reviewed
Sep 23, 2019
icexelloss
reviewed
Sep 23, 2019
icexelloss
reviewed
Sep 23, 2019
icexelloss
reviewed
Sep 23, 2019
icexelloss
reviewed
Sep 23, 2019
icexelloss
reviewed
Sep 23, 2019
icexelloss
reviewed
Sep 23, 2019
icexelloss
suggested changes
Sep 23, 2019
hjoo
force-pushed
the
pyspark-temporal
branch
from
7a12a1d
to
4465f9a
Compare
icexelloss
reviewed
Sep 23, 2019
|
@hjoo One small comments here #1974 (comment) otherwise LGTM |
|
@toryhaavik @DiegoAlbertoTorres can you look over and help merge? |
toryhaavik
approved these changes
Sep 25, 2019
added 6 commits
September 25, 2019 16:10
…andas dt functions
hjoo
force-pushed
the
pyspark-temporal
branch
from
4534824
to
8ee0578
Compare
costrouc
pushed a commit
to costrouc/ibis
that referenced
this pull request
Oct 10, 2019
Implemented date/time operations for PySpark backend to pass all supported tests in `ibis/tests/all/test_temporal.py`. Features that are supported: - `Date` - `Timestamp` - Date and time extraction - Date and timestamp truncation - Timestamp now - String to timestamp - Timestamp to formatted string - `strftime` - Day of week index - Day of week name - Date add, subtract - Timestamp add, subtract - Interval add, subtract Additional operations implemented to pass `test_temporal.py`: - `cast` - `limit` - `array` index Features that are not supported: - `ms`, `us`, `ns` time units - unsupported in native PySpark sql functions - non-literal `interval` (e.g. casting an integer column to an interval column) - this is because there is no timedelta type in PySpark - Date and timestamp diff - again, no support for time delta type Author: Hyonjee <hyonjee.joo@twosigma.com> Closes ibis-project#1974 from hjoo/pyspark-temporal and squashes the following commits: 8ee0578 [Hyonjee] xfail failing param mutate test for pyspark backend 142daae [Hyonjee] minor refactor in pyspark compile_limit 2dc7e62 [Hyonjee] pyspark backend: rename extract_x_from_datetime to extract_component_from_datetime 42ddc34 [Hyonjee] switch pyspark client day_of_week_index and day_of_week_name to use pandas dt functions f6bcff1 [Hyonjee] temporal operations cleanup, basic test for casting c163ff0 [Hyonjee] PySpark backend temporal operations
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implemented date/time operations for PySpark backend to pass all supported tests in
ibis/tests/all/test_temporal.py.Features that are supported:
DateTimestampstrftimeAdditional operations implemented to pass
test_temporal.py:castlimitarrayindexFeatures that are not supported:
ms,us,nstime units - unsupported in native PySpark sql functionsinterval(e.g. casting an integer column to an interval column) - this is because there is no timedelta type in PySpark