Upgrade pandas supported range to >=2.0, <2.3 #4079

shchur · 2024-04-10T12:34:07Z

Issue #, if available: #3989

Description of changes:

Raise pandas upper bound to 2.2 (inclusive)
Update time series pandas-related functionality & tests to work with the new pandas offset aliases (breaking change in pandas 2.2)
- To ensure compatibility with both pandas 2.0/2.1 (old aliases) and 2.2 (new aliases), we adopt the following strategy:
  - All aliases are mapped to the new version (e.g., we always use YE instead of A in our code).
  - For libraries such as GluonTS that don't support pandas 2.2, we replace all aliases with dummy values (W) that are compatible with both old/new style, and take care of all the frequency-related logic in AutoGluon (e.g., time features, lag selection)
  - We cap GluonTS dependency to >=0.14.0,<0.14.4 to ensure that pandas 2.2 can be installed
- Most changes in this PR are related to the tests, ensuring that they can run with both 2.0, 2.1 and 2.2.
Fix warning in CategoryFeatureGenerator

Testing:

Our platform tests currently cover pandas 2.0 (used in all Py3.8 tests) and pandas 2.2 (all remaining tests). We also did a run with pandas 2.1 (https://github.com/autogluon/autogluon/actions/runs/8636868243), so we should be safe to merge.

To do:

@Innixma Verify status of Tabular support
@zhiqiangdon Verify status of AutoMM support
@shchur Verify status of TimeSeries support

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

shchur · 2024-04-10T12:34:31Z

/platform_tests ref=pandas-2.2
Platform Tests Output

yinweisu · 2024-04-10T12:56:11Z

Previous CI Run	Current CI Run

yinweisu · 2024-04-10T14:45:16Z

Previous CI Run	Current CI Run

github-actions · 2024-04-10T17:12:32Z

Job PR-4079-aff7f69 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-4079/aff7f69/index.html

shchur · 2024-04-10T19:39:19Z

/platform_tests ref=pandas-2.2
Platform Tests Output

yinweisu · 2024-04-10T19:48:17Z

Previous CI Run	Current CI Run

tonyhoo · 2024-04-10T22:02:23Z

/platform_tests ref=pandas-2.2

github-actions · 2024-04-10T22:19:25Z

Job PR-4079-fbbb99b is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-4079/fbbb99b/index.html

shchur · 2024-04-11T06:53:48Z

/platform_tests ref=pandas-2.2
Platform Tests Output

yinweisu · 2024-04-11T07:02:00Z

Previous CI Run	Current CI Run
idna==3.6	idna==3.7
idna==3.6	idna==3.7

github-actions · 2024-04-11T09:32:45Z

Job PR-4079-b91cab8 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-4079/b91cab8/index.html

shchur · 2024-04-11T10:16:51Z

features/src/autogluon/features/generators/drop_duplicates.py

@@ -132,6 +132,7 @@ def _drop_duplicate_features_categorical(cls, X: DataFrame, keep: Union[str, boo
            # Converts [5, 'a', np.nan, 5] to [0, 1, 2, 0], these would be considered duplicates since they carry the same information.

            # Have to convert to object dtype because category dtype for unknown reasons will refuse to replace NaNs.
+            # TODO: Fix FutureWarning
            X_cur = X[features_to_check].astype("object").replace(mapping_features_val_dict_cur).astype(np.int64)


@Innixma this line raises FutureWarning:

/local/home/shchuro/workspace/autogluon/features/src/autogluon/features/generators/drop_duplicates.py:135: FutureWarning: Downcasting behavior in `r eplace` is deprecated and will be removed in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`. To opt- in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)` X_cur = X[features_to_check].astype("object").replace(mapping_features_val_dict_cur).astype(np.int64)

Do you have any recommendations on how to fix it while keeping the correct behavior in pandas 2.0/2.1/2.2?

will investigate and send PR

yinweisu · 2024-04-11T10:26:05Z

Previous CI Run	Current CI Run
gluonts==0.14.3	gluonts==0.14.4
pandas==2.2.2	pandas==2.1.4
gluonts==0.14.3	gluonts==0.14.4
pandas==2.2.2	pandas==2.1.4

canerturkmen

LGTM! Thanks!

canerturkmen · 2024-04-11T11:21:02Z

timeseries/src/autogluon/timeseries/utils/datetime/base.py

-    "min": "T",
-    "ms": "L",
-    "us": "U",
+    # sub-daily


github-actions · 2024-04-11T12:57:13Z

Job PR-4079-898abc4 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-4079/898abc4/index.html

zhiqiangdon

May not have enough time to test this before 1.1 release.

Innixma

LGTM! Will do follow-up PRs to addres FutureWarnings

shchur added 13 commits April 10, 2024 19:39

Pandas 2.2 upgrade for TS WIP

5bdbe51

Fix linter

e149ff4

Add GitHub workflow for TS/pandas 2.1

9ef9117

Fix lint

72c77da

Fix freq occurrences

51e4207

Fix freq strings in tests

45bb754

Update CI workflow

0b6b41d

Fix tests for pandas 2.2

242d2fa

Remove extra tests

d8079e6

Remove pandas 2.1 tests

8187700

Remove pytest imports

135f40b

Fix tests for pandas-2.1

524e703

Fix test for pandas 2.1

fbbb99b

shchur force-pushed the pandas-2.2 branch from aff7f69 to fbbb99b Compare April 10, 2024 19:39

Cap GluonTS<0.14.4 to allow pandas 2.2

b91cab8

shchur changed the title ~~[WIP] Upgrade pandas supported range to >=2.0, <2.3~~ Upgrade pandas supported range to >=2.0, <2.3 Apr 11, 2024

shchur requested review from canerturkmen, Innixma and zhiqiangdon April 11, 2024 09:48

shchur requested a review from tonyhoo April 11, 2024 09:48

Silence warnings

7d21a2f

shchur commented Apr 11, 2024

View reviewed changes

Fix comment

898abc4

canerturkmen reviewed Apr 11, 2024

View reviewed changes

timeseries/src/autogluon/timeseries/utils/datetime/base.py

"min": "T",

"ms": "L",

"us": "U",

# sub-daily

Copy link

Contributor

canerturkmen Apr 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow...

zhiqiangdon reviewed Apr 11, 2024

View reviewed changes

Innixma approved these changes Apr 11, 2024

View reviewed changes

Innixma merged commit 66e49bc into master Apr 11, 2024
43 of 56 checks passed

shchur deleted the pandas-2.2 branch April 11, 2024 18:05

This was referenced Apr 11, 2024

1.1 Release Tracker #4027

Closed

Support Pandas 2.2 #3989

Closed

[DO NOT MERGE] Upgrade pandas to 2.2.x #4044

Closed

LennartPurucker pushed a commit to LennartPurucker/autogluon that referenced this pull request Jun 1, 2024

Upgrade pandas supported range to >=2.0, <2.3 (autogluon#4079)

fcb2555

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade pandas supported range to >=2.0, <2.3 #4079

Upgrade pandas supported range to >=2.0, <2.3 #4079

shchur commented Apr 10, 2024 •

edited

shchur commented Apr 10, 2024 •

edited by yinweisu

yinweisu commented Apr 10, 2024

yinweisu commented Apr 10, 2024

github-actions bot commented Apr 10, 2024

shchur commented Apr 10, 2024 •

edited by yinweisu

yinweisu commented Apr 10, 2024

tonyhoo commented Apr 10, 2024

github-actions bot commented Apr 10, 2024

shchur commented Apr 11, 2024 •

edited by yinweisu

yinweisu commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

shchur Apr 11, 2024 •

edited

Innixma Apr 11, 2024

yinweisu commented Apr 11, 2024

canerturkmen left a comment

canerturkmen Apr 11, 2024

github-actions bot commented Apr 11, 2024

zhiqiangdon left a comment

Innixma left a comment

Upgrade pandas supported range to >=2.0, <2.3 #4079

Upgrade pandas supported range to >=2.0, <2.3 #4079

Conversation

shchur commented Apr 10, 2024 • edited

shchur commented Apr 10, 2024 • edited by yinweisu

yinweisu commented Apr 10, 2024

yinweisu commented Apr 10, 2024

github-actions bot commented Apr 10, 2024

shchur commented Apr 10, 2024 • edited by yinweisu

yinweisu commented Apr 10, 2024

tonyhoo commented Apr 10, 2024

github-actions bot commented Apr 10, 2024

shchur commented Apr 11, 2024 • edited by yinweisu

yinweisu commented Apr 11, 2024

github-actions bot commented Apr 11, 2024

shchur Apr 11, 2024 • edited

Choose a reason for hiding this comment

Innixma Apr 11, 2024

Choose a reason for hiding this comment

yinweisu commented Apr 11, 2024

canerturkmen left a comment

Choose a reason for hiding this comment

canerturkmen Apr 11, 2024

Choose a reason for hiding this comment

github-actions bot commented Apr 11, 2024

zhiqiangdon left a comment

Choose a reason for hiding this comment

Innixma left a comment

Choose a reason for hiding this comment

shchur commented Apr 10, 2024 •

edited

shchur commented Apr 10, 2024 •

edited by yinweisu

shchur commented Apr 10, 2024 •

edited by yinweisu

shchur commented Apr 11, 2024 •

edited by yinweisu

shchur Apr 11, 2024 •

edited