FEAT-#5936: support pandas 2.0.2 #5995

anmyachev · 2023-04-12T22:23:20Z

What do these changes do?

first commit message and PR title follow format outlined here

NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.
passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
signed commit with git commit -s
Resolves Support pandas 2.x #5936
tests added and passing
module layout described at docs/development/architecture.rst is up-to-date

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

…pandas-2.0.0

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

anmyachev · 2023-06-05T22:53:26Z

@vnlitvinov ready for review

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

modin/pandas/dataframe.py

vnlitvinov · 2023-06-06T07:22:57Z

@anmyachev there are a few open questions left from previous review, but most of them are minor.
I think the only major blocker for me is my last comment, about updating QC in-place.

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

dchigarev

looks good from my side

thank you for this massive work!

modin/pandas/series.py

modin/test/interchange/dataframe_protocol/hdk/test_protocol.py

…pandas-2.0.0

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

Co-authored-by: Dmitry Chigarev <dmitry.chigarev@intel.com>

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

anmyachev · 2023-06-06T18:06:57Z

@dchigarev ready for review

YarShev · 2023-06-06T19:43:01Z

@vnlitvinov, any comments?

mvashishtha · 2023-06-07T18:53:53Z

@vnlitvinov, any comments?

@YarShev @dchigarev I see that @vnlitvinov comment about _update_inplace is still open

https://github.com/modin-project/modin/pull/5995/files/7edd9bdd8bcaa16466f666949d5314d460f610ea..6d7e8d260bf08a7bb304c258e902c237736aa043#r1219040039

anmyachev · 2023-06-07T19:01:25Z

@vnlitvinov, any comments?

@YarShev @dchigarev I see that @vnlitvinov comment about _update_inplace is still open

https://github.com/modin-project/modin/pull/5995/files/7edd9bdd8bcaa16466f666949d5314d460f610ea..6d7e8d260bf08a7bb304c258e902c237736aa043#r1219040039

@mvashishtha I fixed it but forgot to reply to the comment

vnlitvinov · 2023-06-08T08:17:47Z

modin/pandas/test/test_io.py

        ReadSqlEngine.put(read_sql_engine)
        if ReadSqlEngine.get() == "Connectorx":
            modin_df = pd.read_sql(sql=query, con=conn)
        else:
            modin_df = pd.read_sql(
                sql=query, con=ModinDatabaseConnection("sqlalchemy", conn)
            )
+        ReadSqlEngine.put(old_sql_engine)


this should be done in try..finally to be absolutely sure we return the value back... not critical in tests I suppose, so mostly a note for the future patterns

vnlitvinov · 2023-06-08T08:18:25Z

modin/pandas/test/test_io.py

+        def comparator(df1, df2):
+            df_equals(df1, df2)
+            df_equals(df1.dtypes, df2.dtypes)


feels like it should be extracted to a common place, this comparator is written in a few places already; could go in some tests refactoring PR I suppose

vnlitvinov · 2023-06-08T08:19:37Z

@anmyachev thanks for the tremendous amount of effort into making this PR by the way!

anmyachev added 30 commits April 12, 2023 21:36

FEAT-modin-project#5936: support pandas 2.0.0

6887fb0

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

remove Int64Index, UInt64Index, Float64Index

c0c9cf2

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

remove pandas.datetime

93b7f66

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

remove convert_to_index_sliceable

eca0422

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

remove 'ensure_clean_dir'

929edcc

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

Merge branch 'master' of https://github.com/modin-project/modin into …

025d14a

…pandas-2.0.0

disable 'xarray' until it adds support for pandas 2.0.0

417bf0c

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

remove 'mad' [2]

b9a83b5

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

add 'dtype_backend' parameter for all functions and remove is_monotonic

af9b32e

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

remove 'tshift' and 'iteritems'

5626de0

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

remove 'slice_shift'

9ee440c

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

remove 'append'

3cb0007

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

add new parameter: 'axis' for 'add_suffix', 'add_prefix'

2b0fbfc

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

use copy=None instead of copy=True

510b907

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

remove deprecated parameters in 'between_time'

3cddf29

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'drop_duplicates'

bc5e97f

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'dropna'

0d5672e

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'factorize'

f586ad1

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'groupby'

be041e8

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'kurt'

bdead97

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'mask'

78217ef

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'mean' and 'max'

f0af110

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'min', 'median'

7daad2d

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'resample', 'rank'

1a37565

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'sem', 'skew'

3b40481

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'var', 'sum', 'std'

79c4b63

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'to_json'

24f6f5c

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'all', 'any'

db67661

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'describe', 'clip', 'count'

886a7cf

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update 'resample', 'expanding'

2cc84a8

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

update comments

276b383

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

try to avoid old logic for processing 'numeric_only' param

6d7e8d2

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

vnlitvinov reviewed Jun 6, 2023

View reviewed changes

modin/pandas/dataframe.py Outdated Show resolved Hide resolved

address review comments

8121867

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

dchigarev previously approved these changes Jun 6, 2023

View reviewed changes

modin/pandas/series.py Show resolved Hide resolved

modin/test/interchange/dataframe_protocol/hdk/test_protocol.py Outdated Show resolved Hide resolved

anmyachev added 2 commits June 6, 2023 16:40

Merge branch 'master' of https://github.com/modin-project/modin into …

c068d61

…pandas-2.0.0

try to exclude 'datetime' type for hdk tests

4222292

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

anmyachev dismissed dchigarev’s stale review via 4222292 June 6, 2023 14:41

anmyachev and others added 2 commits June 6, 2023 16:43

Update modin/pandas/series.py

7b8d207

Co-authored-by: Dmitry Chigarev <dmitry.chigarev@intel.com>

fix

c1e7cb8

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

dchigarev previously approved these changes Jun 6, 2023

View reviewed changes

YarShev previously approved these changes Jun 6, 2023

View reviewed changes

anmyachev changed the title ~~FEAT-#5936: support pandas 2.0.0~~ FEAT-#5936: support pandas 2.0.2 Jun 6, 2023

Merge branch 'master' into pandas-2.0.0

3e7e55a

anmyachev dismissed stale reviews from YarShev and dchigarev via 3e7e55a June 7, 2023 11:53

dchigarev approved these changes Jun 7, 2023

View reviewed changes

YarShev approved these changes Jun 7, 2023

View reviewed changes

mvashishtha merged commit 1d8a179 into modin-project:master Jun 7, 2023
49 of 50 checks passed

anmyachev deleted the pandas-2.0.0 branch June 7, 2023 19:23

vnlitvinov reviewed Jun 8, 2023

View reviewed changes

vnlitvinov mentioned this pull request Jun 13, 2023

Adapt pandas Copy-on-Write functionality for modin #6254

Open

anmyachev mentioned this pull request Jul 2, 2023

Unskip tests with Future version pandas #3142

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT-#5936: support pandas 2.0.2 #5995

FEAT-#5936: support pandas 2.0.2 #5995

anmyachev commented Apr 12, 2023 •

edited

anmyachev commented Jun 5, 2023

vnlitvinov commented Jun 6, 2023

dchigarev left a comment

anmyachev commented Jun 6, 2023

YarShev commented Jun 6, 2023

mvashishtha commented Jun 7, 2023

anmyachev commented Jun 7, 2023

vnlitvinov Jun 8, 2023

vnlitvinov Jun 8, 2023

vnlitvinov commented Jun 8, 2023

FEAT-#5936: support pandas 2.0.2 #5995

FEAT-#5936: support pandas 2.0.2 #5995

Conversation

anmyachev commented Apr 12, 2023 • edited

What do these changes do?

anmyachev commented Jun 5, 2023

vnlitvinov commented Jun 6, 2023

dchigarev left a comment

Choose a reason for hiding this comment

anmyachev commented Jun 6, 2023

YarShev commented Jun 6, 2023

mvashishtha commented Jun 7, 2023

anmyachev commented Jun 7, 2023

vnlitvinov Jun 8, 2023

Choose a reason for hiding this comment

vnlitvinov Jun 8, 2023

Choose a reason for hiding this comment

vnlitvinov commented Jun 8, 2023

anmyachev commented Apr 12, 2023 •

edited