GH-37880: [CI][Python][Packaging] Add support for Python 3.12 #37901

raulcd · 2023-09-27T11:28:14Z

Rationale for this change

Python 3.12 will be released in the next couple of weeks. We should add the wheels for pyarrow on our 14.0.0 release.

What changes are included in this PR?

This PR adds jobs to build pyarrow wheels for Python 3.12.

Are these changes tested?

They will be tested via archery tasks

Are there any user-facing changes?

No but users will be able to use pyarrow with Python 3.12

Closes: [Python][Packaging] Support Python 3.12 and upload wheels to PyPI #37880

raulcd · 2023-09-27T11:28:58Z

@github-actions crossbow submit cp312

github-actions · 2023-09-27T11:32:02Z

Revision: 907f66d

Submitted crossbow builds: ursacomputing/crossbow @ actions-6d09e69209

Task	Status
wheel-macos-big-sur-cp312-arm64
wheel-macos-mojave-cp312-amd64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-windows-cp312-amd64

raulcd · 2023-09-27T12:59:10Z

@github-actions crossbow submit wheel-macos-mojave-cp312-amd64

github-actions · 2023-09-27T13:01:46Z

Revision: 5cee4a8

Submitted crossbow builds: ursacomputing/crossbow @ actions-2decf3cc0d

Task	Status
wheel-macos-mojave-cp312-amd64

raulcd · 2023-09-27T13:24:32Z

@github-actions crossbow submit wheel-manylinux-2-28-cp312-amd64

github-actions · 2023-09-27T13:27:09Z

Revision: 54d5d34

Submitted crossbow builds: ursacomputing/crossbow @ actions-ca5ce95faa

Task	Status
wheel-manylinux-2-28-cp312-amd64

raulcd · 2023-09-27T13:32:40Z

The macOS issue seems to fail due to this (https://github.com/ursacomputing/crossbow/actions/runs/6326301003/job/17179575146#step:13:173):
pypa/setuptools#3661
I'll investigate further, here the error log:

  File "/Users/runner/work/crossbow/crossbow/test-amd64-env/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 994, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-bw0gv97b/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'

raulcd · 2023-09-27T13:38:09Z

@github-actions crossbow submit wheel-macos-mojave-cp312-amd64

github-actions · 2023-09-27T13:40:44Z

Revision: 349f362

Submitted crossbow builds: ursacomputing/crossbow @ actions-f607b07df1

Task	Status
wheel-macos-mojave-cp312-amd64

raulcd · 2023-09-27T14:50:17Z

I am facing some issues when updating setuptools with numpy:

        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-2rp89sp0/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
          import setuptools.version
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-2rp89sp0/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
          import pkg_resources
        File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-2rp89sp0/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
          register_finder(pkgutil.ImpImporter, find_on_path)
                          ^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
      [end of output]

This seems related to: pypa/pip#12179

raulcd · 2023-09-27T14:50:42Z

@github-actions crossbow submit wheel-macos-mojave-cp312-amd64

github-actions · 2023-09-27T14:53:28Z

Revision: 3e3046c

Submitted crossbow builds: ursacomputing/crossbow @ actions-635103f8ad

Task	Status
wheel-macos-mojave-cp312-amd64

raulcd · 2023-09-27T15:26:15Z

@github-actions crossbow submit wheel-manylinux-2-28-cp312-amd64

github-actions · 2023-09-27T15:29:43Z

Revision: 125ac34

Submitted crossbow builds: ursacomputing/crossbow @ actions-70da4f8be9

Task	Status
wheel-manylinux-2-28-cp312-amd64

raulcd · 2023-09-27T16:27:51Z

@github-actions crossbow submit wheel-macos-mojave-cp312-amd64

github-actions · 2023-09-27T16:30:51Z

Revision: dbd90cc

Submitted crossbow builds: ursacomputing/crossbow @ actions-93131c2a12

Task	Status
wheel-macos-mojave-cp312-amd64

raulcd · 2023-09-27T17:54:43Z

There seems to be an issue compiling grpcio with Python 3.12:
grpc/grpc#33063
It seems it will be released soon as v1.59.0-pre1 seems to contain a fix for it: https://github.com/grpc/grpc/releases/tag/v1.59.0-pre1
I am facing the same issue on the macos wheels here: https://github.com/ursacomputing/crossbow/actions/runs/6328784518/job/17187734569#step:13:10233
and the manylinux one here: https://github.com/ursacomputing/crossbow/actions/runs/6328121689/job/17185543580#step:10:12731
I might disable google testbench temporarily to run the rest of the tests.

raulcd · 2023-09-27T17:59:26Z

@github-actions crossbow submit wheel-macos-mojave-cp312-amd64

github-actions · 2023-09-27T18:02:09Z

Revision: e92c96b

Submitted crossbow builds: ursacomputing/crossbow @ actions-b8b091adfb

Task	Status
wheel-macos-mojave-cp312-amd64

raulcd · 2023-09-28T08:30:32Z

@github-actions crossbow submit cp312

github-actions · 2023-09-28T08:33:10Z

Revision: e92c96b

Submitted crossbow builds: ursacomputing/crossbow @ actions-fa45089ccb

Task	Status
wheel-macos-big-sur-cp312-arm64
wheel-macos-mojave-cp312-amd64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-windows-cp312-amd64

raulcd · 2023-10-03T07:37:36Z

@github-actions crossbow submit cp312

github-actions · 2023-10-03T07:40:11Z

Revision: b1d98ea

Submitted crossbow builds: ursacomputing/crossbow @ actions-817b67d67c

Task	Status
wheel-macos-big-sur-cp312-arm64
wheel-macos-mojave-cp312-amd64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-windows-cp312-amd64

raulcd · 2023-10-03T10:34:42Z

@github-actions crossbow submit wheel-macos-big-sur-cp312-arm64

pitrou · 2023-10-10T16:44:08Z

@github-actions crossbow submit -g wheel

github-actions · 2023-10-10T16:47:30Z

Revision: e1369db

Submitted crossbow builds: ursacomputing/crossbow @ actions-ba06ba3c11

Task	Status
wheel-clean
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-mojave-cp310-amd64
wheel-macos-mojave-cp311-amd64
wheel-macos-mojave-cp312-amd64
wheel-macos-mojave-cp38-amd64
wheel-macos-mojave-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2023-10-10T17:17:46Z

@github-actions crossbow submit wheel-macos-mojave-cp310-amd64

github-actions · 2023-10-10T17:20:20Z

Revision: 27a979c

Submitted crossbow builds: ursacomputing/crossbow @ actions-1822e92828

Task	Status
wheel-macos-mojave-cp310-amd64

raulcd · 2023-10-11T09:03:42Z

@pitrou @jorisvandenbossche any other concern here? I would like to merge it.

pitrou · 2023-10-11T09:05:13Z

@github-actions crossbow submit -g wheel -g python

pitrou · 2023-10-11T09:05:28Z

Let's make sure the 3.10 version change didn't break anything else?

github-actions · 2023-10-11T09:08:45Z

Revision: 27a979c

Submitted crossbow builds: ursacomputing/crossbow @ actions-1c7e96dd5f

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3
test-fedora-35-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3
wheel-clean
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-mojave-cp310-amd64
wheel-macos-mojave-cp311-amd64
wheel-macos-mojave-cp312-amd64
wheel-macos-mojave-cp38-amd64
wheel-macos-mojave-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

pitrou · 2023-10-11T09:37:43Z

There are a lot of fsspec/s3fs errors that are unrelated to this PR.

test-conda-python-3.12 also fails installing dependencies from conda-forge, because gdb seems incompatible with Python 3.12. This can probably be fixed later.

pitrou · 2023-10-11T11:06:07Z

I think the CI results are ok otherwise, so we can merge.

jorisvandenbossche · 2023-10-11T11:19:37Z

A merge with main might be sufficient to get rid of the failures? We saw the same yesterday in the encryption PR, and after a merge they went away.
But indeed, if the triggered wheels builds are passing, that's probably also good enough to merge this directly.

jorisvandenbossche · 2023-10-11T11:25:54Z

Restarted one of the failing wheel builds, as that seemed a temporary download failure.

raulcd

👍 I am merging this! Thanks

### Rationale for this change Python 3.12 will be released in the next couple of weeks. We should add the wheels for pyarrow on our 14.0.0 release. ### What changes are included in this PR? This PR adds jobs to build pyarrow wheels for Python 3.12. ### Are these changes tested? They will be tested via archery tasks ### Are there any user-facing changes? No but users will be able to use pyarrow with Python 3.12 * Closes: #37880 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>

…pache#37901) ### Rationale for this change Python 3.12 will be released in the next couple of weeks. We should add the wheels for pyarrow on our 14.0.0 release. ### What changes are included in this PR? This PR adds jobs to build pyarrow wheels for Python 3.12. ### Are these changes tested? They will be tested via archery tasks ### Are there any user-facing changes? No but users will be able to use pyarrow with Python 3.12 * Closes: apache#37880 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>

conbench-apache-arrow · 2023-10-14T16:04:13Z

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 7695935.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.

…pache#37901) ### Rationale for this change Python 3.12 will be released in the next couple of weeks. We should add the wheels for pyarrow on our 14.0.0 release. ### What changes are included in this PR? This PR adds jobs to build pyarrow wheels for Python 3.12. ### Are these changes tested? They will be tested via archery tasks ### Are there any user-facing changes? No but users will be able to use pyarrow with Python 3.12 * Closes: apache#37880 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>

github-actions bot added Component: Python awaiting committer review Awaiting committer review labels Sep 27, 2023

This comment was marked as outdated.

Sign in to view

terencehonles mentioned this pull request Oct 2, 2023

GH-37880: [CI][Python][Packaging] Add support for Python 3.12 #37977

Closed

raulcd force-pushed the GH-37880 branch from e92c96b to b1d98ea Compare October 3, 2023 07:37

raulcd added 2 commits October 10, 2023 14:43

Do not update pip to do not mess up with other jobs

e1203ff

Do not update setuptools if not necessary

790d950

raulcd mentioned this pull request Oct 10, 2023

[CI][Python] Use pipx to install GCS testbench in order to decouple version of Python under test #38183

Open

Install archery in command

e1369db

Use Python 3.10.11 version that contains the Macos .pkg

27a979c

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 11, 2023

raulcd commented Oct 11, 2023

View reviewed changes

raulcd merged commit 7695935 into apache:main Oct 11, 2023
58 checks passed

raulcd removed the awaiting changes Awaiting changes label Oct 11, 2023

github-actions bot added the awaiting changes Awaiting changes label Oct 11, 2023

thehamish555 mentioned this pull request Oct 12, 2023

Can't install streamlit for Python 3.12 streamlit/streamlit#7506

Closed

4 tasks

GH-37880: [CI][Python][Packaging] Add support for Python 3.12 #37901

GH-37880: [CI][Python][Packaging] Add support for Python 3.12 #37901

Conversation

raulcd commented Sep 27, 2023 • edited by github-actions bot

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

raulcd commented Sep 27, 2023

github-actions bot commented Sep 27, 2023

raulcd commented Sep 27, 2023

github-actions bot commented Sep 27, 2023

This comment was marked as outdated.

This comment was marked as outdated.

raulcd commented Sep 27, 2023

github-actions bot commented Sep 27, 2023

raulcd commented Sep 27, 2023

raulcd commented Sep 27, 2023

github-actions bot commented Sep 27, 2023

raulcd commented Sep 27, 2023

raulcd commented Sep 27, 2023

github-actions bot commented Sep 27, 2023

raulcd commented Sep 27, 2023

github-actions bot commented Sep 27, 2023

raulcd commented Sep 27, 2023

github-actions bot commented Sep 27, 2023

raulcd commented Sep 27, 2023 • edited

raulcd commented Sep 27, 2023

github-actions bot commented Sep 27, 2023

raulcd commented Sep 28, 2023

github-actions bot commented Sep 28, 2023

raulcd commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

raulcd commented Oct 3, 2023

pitrou commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

raulcd commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

raulcd commented Oct 11, 2023

pitrou commented Oct 11, 2023

pitrou commented Oct 11, 2023

github-actions bot commented Oct 11, 2023

pitrou commented Oct 11, 2023

pitrou commented Oct 11, 2023

jorisvandenbossche commented Oct 11, 2023

jorisvandenbossche commented Oct 11, 2023

raulcd left a comment

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Oct 14, 2023

raulcd commented Sep 27, 2023 •

edited by github-actions bot

raulcd commented Sep 27, 2023 •

edited