[SPARK-43570][SPARK-43571][PYTHON][TESTS] Enable DateOpsTests.[test_rsub|test_sub] for pandas 2.0.0. #41824
Conversation
panbingkun
changed the title
| def test_sub(self): | ||
| self.assertRaises(TypeError, lambda: self.psser - "x") | ||
| self.assertRaises(TypeError, lambda: self.psser - 1) | ||
| self.assert_eq( | ||
| (self.pser - self.some_date).dt.days, | ||
| pd.to_timedelta(self.pser - self.some_date).dt.days, | ||
| self.psser - self.some_date, |
There was a problem hiding this comment.
Actually, I think we should also make the behaviour the same with pandas 2.0.0 at pandas API on Spark too. I forgot how we concluded this. Right @itholic ? and should upgrade migration guide too?
There was a problem hiding this comment.
Yes, instead of just fixing it to work for testing, we need to modify the behavior to match pandas. However, we have decided to put this task on hold and address it in the next major release, Spark 4.0. For more details, please refer to the comment from SPARK-44101.
IMHO, I'd recommend temporarily holding off on any tasks related to pandas 2.0.0 until starting to prepare Spark 4.0 release to avoid potential mistakes.
There was a problem hiding this comment.
OK, Let's to put this task on hold , I will convert it to draft.
There was a problem hiding this comment.
Sure. Thanks for your cooperation! :-)
|
Oh, I just realized that this is already fixed from #42533. But seems like the approach is a bit different, so please feel free to proceed this as a follow-up if you think we should need to address the current way? |
I'm good with fixing it in the current way, let me close it now. |
What changes were proposed in this pull request?
The pr aims to enable DateOpsTests.test_rsub & DateOpsTests.test_sub for pandas 2.0.0.
Why are the changes needed?
Improve UT coverage.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
-Pass GA
-Manually test.