CI: Test Python 3.12 #53743

lithomas1 · 2023-06-20T18:29:47Z

xref ENH: Python 3.12 support #53665 (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

lithomas1 · 2023-06-20T22:26:08Z

For those interested, we have
208 failed, 172066 passed, 18996 skipped, 982 xfailed, 84 xpassed, 56 warnings in 2448.38s (0:40:48) =

Not bad, given most are parametrized, but definitely much more than last year 😭 .

…-py312

mroeschke · 2023-06-29T22:36:04Z

I pushed up a fix to use timezone.utc. I think datetime.UTC is > PY311

lithomas1 · 2023-07-11T21:52:37Z

Thanks for updating this, and sorry for losing track of this PR.

I have the Docker image for Python 3.12 set up on my laptop, so I'll try to get some debugging in tomorrow for the Index failures, unless you beat me to it again :).

lithomas1 · 2023-07-11T23:52:14Z

Looks like the major change is

slice objects are now hashable, allowing them to be used as dict keys and set items. (Contributed by Will Bradshaw, Furkan Onder, and Raymond Hettinger in gh-101264.)

lithomas1 · 2023-07-14T04:39:02Z

Down to 4 tests

FAILED pandas/tests/computation/test_eval.py::TestEval::test_true_false_logic - DeprecationWarning: Bitwise inversion '~' on bool is deprecated. This returns the bitwise inversion of the underlying int object and is usually not what you expect from negating a bool. Use the 'not' operator for boolean negation or ~int(x) if you really want the bitwise inversion of the underlying int.
FAILED pandas/tests/frame/indexing/test_where.py::TestDataFrameIndexingWhere::test_where_invalid - DeprecationWarning: Bitwise inversion '~' on bool is deprecated. This returns the bitwise inversion of the underlying int object and is usually not what you expect from negating a bool. Use the 'not' operator for boolean negation or ~int(x) if you really want the bitwise inversion of the underlying int.
FAILED pandas/tests/series/indexing/test_getitem.py::test_getitem_generator - AssertionError: Series are different

Series length are different
[left]:  0, Index([], dtype='object')
[right]: 14, Index(['1bZFplgxJt', 'dgf6MOzmcc', 'nXA7LxB4pD', '7xialahw4M', 'eHSMzF5vTv',
       'ohno0T7Zu6', 'sK2uiqyfCh', 'ccmUonUyaI', 'KNYdKnS7IO', 'jQ9UjUVpex',
       'NGazj6cYBr', 'XHqmZevyyF', 'SQXHWlGTY5', 'ZmyLvG2mLO'],
      dtype='object')
FAILED pandas/tests/tseries/offsets/test_year.py::test_add_out_of_pydatetime_range

I think I regressed one of the indexing tests in my fixes.

pandas/core/indexing.py

lithomas1 · 2023-07-14T23:27:01Z

This should be ready now.

cc @jbrockmendel @phofl for indexing changes

Not sure who to ping for eval/sql changes.

jbrockmendel · 2023-07-17T15:36:32Z

pandas/core/indexes/datetimelike.py

@@ -165,7 +168,7 @@ def __contains__(self, key: Any) -> bool:
        hash(key)
        try:
            self.get_loc(key)
-        except (KeyError, TypeError, ValueError):
+        except (KeyError, TypeError, ValueError, InvalidIndexError):


what cases get here?

Probably this one
https://github.com/pandas-dev/pandas/pull/53743/files#diff-c34a28314fc8cb12f0d2aa710f1c15f06cdfe3e48f03e658f01f99a43d4f5d09R3793-R3797

pandas/core/computation/expr.py

jbrockmendel · 2023-07-17T15:39:17Z

pandas/core/series.py

-        if is_hashable(key):
+        # Convert generator to list before going through hashable part
+        # (We will iterate through the generator there to check for slices)
+        if is_iterator(key):


why does changing the order matter? the current order is pretty fine-tuned for perf

We need to check for slices in iterators by iterating through them, since they aren't allowed as hash keys.
(It doesn't make sense to have slices as keys in an Index and it breaks way too many things).
This exhausts the generator.

IIUC, this should only lower perf for generators?
(at least the is_iterator docstring only mentions that it will only return True for generators not list and co.)

pandas/tests/computation/test_eval.py

.github/workflows/unit-tests.yml

Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

.circleci/config.yml

mroeschke

One comment and a small test failure otherwise looks good

jbrockmendel · 2023-09-14T22:18:58Z

.github/workflows/unit-tests.yml

-        os: [ubuntu-22.04, macOS-latest, windows-latest]
+        # TODO: Disable macOS for now, Github Actions bug where python is not
+        # symlinked correctly to 3.12
+        # xref https://github.com/actions/setup-python/issues/701


the linked issue is now closed; can this be reenabled?

Matt enabled this again a while back, I think.

jbrockmendel · 2023-09-14T22:22:10Z

pandas/core/series.py

+        if is_iterator(key):
+            key = list(key)
+
+        if is_hashable(key) and not isinstance(key, slice):


very late review comment: would it make sense to make is_hashable_non_slice or something? it wouldn't surprise me if many places that use is_hashable current assume non-slice but didn't get updated by this PR

Yeah sure, I just updated the places that broke in the tests.

Is there a case where we would actually want is_hashable for a slice to equal True, though?
(I was thinking it might be cleaner to make is_hashable always return False for slices. This would be a breaking API change, though).

i think having is_hashable be anything other than a try/except around hash would cause problems. im suggesting a new function to de-duplicate the hashable-but-not-slice checks this introduces

Made an issue #55152.

I don't have the same time that I had in summer to work on pandas, but I'll try to have a look (no promises, though).

CI: Test Python 3.12

ff98491

lithomas1 added Build Library building on various platforms CI Continuous Integration labels Jun 20, 2023

lithomas1 requested a review from mroeschke as a code owner June 20, 2023 18:29

lithomas1 marked this pull request as draft June 20, 2023 18:29

lithomas1 removed the request for review from mroeschke June 20, 2023 19:22

lithomas1 added 5 commits June 20, 2023 12:23

Update unit-tests.yml

fcbe1ed

Update config.yml

dd4a488

update

c809bcd

Merge branch 'main' into test-py312

cf7e272

fix condition

8c7aea2

lithomas1 and others added 6 commits June 25, 2023 13:16

Merge branch 'main' of https://github.com/pandas-dev/pandas into test…

dc88fa8

…-py312

fix some tests

d2b3868

Remove wheel building for Python 3.12

5814bcc

fix more

7be7ea9

Use timezone.utc

e9c0ed4

Merge remote-tracking branch 'upstream/main' into test-py312

f735b8c

mroeschke added 2 commits July 7, 2023 07:23

Merge branch 'main' into test-py312

80d31e1

Address typing, utcfromtimestamp

b8f351d

lithomas1 added 3 commits July 11, 2023 17:28

fix some slice changes

7cbfd7c

fix all indexing bugs?

e507d45

fix import

90dbdeb

go for green

5e421f3

mroeschke reviewed Jul 14, 2023

View reviewed changes

pandas/core/indexing.py Outdated Show resolved Hide resolved

disable macos for now, fix other tests

0e04fd2

finally fix?

b1182ac

lithomas1 marked this pull request as ready for review July 14, 2023 23:24

lithomas1 added this to the 2.1 milestone Jul 14, 2023

Merge branch 'main' into test-py312

ddae8f8

jbrockmendel reviewed Jul 17, 2023

View reviewed changes

pandas/core/computation/expr.py Outdated Show resolved Hide resolved

jbrockmendel reviewed Jul 17, 2023

View reviewed changes

Update expr.py

64e12a9

lithomas1 mentioned this pull request Jul 19, 2023

Fix deprecation warning on Python 3.12 #54188

Closed

3 tasks

Merge branch 'main' into test-py312

bc64b47

mroeschke reviewed Jul 24, 2023

View reviewed changes

pandas/tests/computation/test_eval.py Outdated Show resolved Hide resolved

mroeschke reviewed Jul 24, 2023

View reviewed changes

.github/workflows/unit-tests.yml Show resolved Hide resolved

lithomas1 and others added 3 commits July 24, 2023 14:05

Update pandas/tests/computation/test_eval.py

2c1755e

Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

Update test_eval.py

5dfc14b

Update test_eval.py

a1bd210

lithomas1 requested a review from mroeschke July 25, 2023 17:14

mroeschke reviewed Jul 25, 2023

View reviewed changes

.circleci/config.yml Outdated Show resolved Hide resolved

mroeschke reviewed Jul 25, 2023

View reviewed changes

lithomas1 added 2 commits July 25, 2023 13:16

fixes

298f31b

formatting

88a0cb8

mroeschke approved these changes Jul 25, 2023

View reviewed changes

lithomas1 merged commit c9a8f95 into pandas-dev:main Jul 25, 2023
69 of 70 checks passed

lithomas1 deleted the test-py312 branch July 25, 2023 23:36

lesteve mentioned this pull request Aug 7, 2023

Upload Python 3.12 wheel to scientific-python-nightly-wheels? #54447

Closed

phofl mentioned this pull request Sep 3, 2023

BUG: Regression how datetime64[ns] is saved in sqlite table #54877

Closed

3 tasks

jbrockmendel reviewed Sep 14, 2023

View reviewed changes

lithomas1 mentioned this pull request Sep 15, 2023

CLN: Add is_hashable_non_slice #55152

Open

jorisvandenbossche mentioned this pull request Oct 25, 2023

BUG: after pandas 2.1.0 sqlite isnt able to auto parse dates based on schema #55554

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Test Python 3.12 #53743

CI: Test Python 3.12 #53743

lithomas1 commented Jun 20, 2023 •

edited

lithomas1 commented Jun 20, 2023

mroeschke commented Jun 29, 2023

lithomas1 commented Jul 11, 2023

lithomas1 commented Jul 11, 2023 •

edited

lithomas1 commented Jul 14, 2023

lithomas1 commented Jul 14, 2023

jbrockmendel Jul 17, 2023

lithomas1 Jul 17, 2023

jbrockmendel Jul 17, 2023

lithomas1 Jul 17, 2023

mroeschke left a comment

jbrockmendel Sep 14, 2023

lithomas1 Sep 15, 2023

jbrockmendel Sep 14, 2023

lithomas1 Sep 15, 2023

jbrockmendel Sep 15, 2023

lithomas1 Sep 15, 2023

CI: Test Python 3.12 #53743

CI: Test Python 3.12 #53743

Conversation

lithomas1 commented Jun 20, 2023 • edited

lithomas1 commented Jun 20, 2023

mroeschke commented Jun 29, 2023

lithomas1 commented Jul 11, 2023

lithomas1 commented Jul 11, 2023 • edited

lithomas1 commented Jul 14, 2023

lithomas1 commented Jul 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mroeschke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lithomas1 commented Jun 20, 2023 •

edited

lithomas1 commented Jul 11, 2023 •

edited