GH-36730: [Python] Add support for Cython 3.0.0 #37097

danepitkin · 2023-08-09T20:30:23Z

Rationale for this change

Cython 3.0.0 is the latest release. PyArrow should work with Cython 3.0.0. Cython 3 is not enabled in this diff.

What changes are included in this PR?

Don't use vector[XXX]&&
Add a declaration for postincrement
- See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#c-postincrement-postdecrement-operator
Ignore C4551 warning (function call missing argument list) with MSVC
- See also: [BUG] code emitted for Python 3.10 causes warning C4551 in MSVC cython/cython#4445
Add missing const to CLocation's static methods.
Don't use StopIteration to stop generator
- See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-3-syntax-semantics
non-extern cdef functions will now propagate python exceptions automatically unless explicitly labeled noexcept
Function binding in cython is now enabled by default. Class methods that are used as wrappers for pickling should be converted to staticmethods.
Numpydocs now validates more Cython 3 objects than Cython <3
- Enum types are now being validated, and some unhelpful validation checks on Enums are now ignored
Added a cython <3 nightly CI job

Note:

Cython 3.0.0, 3.0.1, 3.0.2 has an issue when compiling with debug mode [BUG] invalid tag name with iterators in --gdb mode cython/cython#5552

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

Closes: [CI][Python] Cython 3.0 seems to make our build of pyarrow fail #36730

github-actions · 2023-08-09T20:30:49Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

python/pyarrow/tests/parquet/test_dataset.py

danepitkin · 2023-08-11T19:28:49Z

Potentially related cloudpickle issue? cloudpipe/cloudpickle#506

Edit: Nope. Cython 3.0.0 changed the default compiler directive "binding" from False to True. Setting back to false fixes the cloudpickle test.

kou · 2023-08-11T20:49:40Z

I've copied #36745 PR description.
We should update the PR description before we merge this PR.

danepitkin · 2023-08-14T22:14:40Z

@github-actions crossbow submit python

github-actions · 2023-08-14T22:17:50Z

Revision: c8e6a19

Submitted crossbow builds: ursacomputing/crossbow @ actions-ba5c455e80

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
python-sdist
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.4.1
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.4.1
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3
test-fedora-35-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3
verify-rc-source-python-linux-almalinux-8-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-20.04-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64

danepitkin · 2023-08-15T14:11:23Z

test-cuda-python and test-ubuntu-20.04-python-3 are failing due to this cython issue cython/cython#5552

github-actions · 2023-09-15T16:55:58Z

Revision: 6f32fcf

Submitted crossbow builds: ursacomputing/crossbow @ actions-165015566a

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.4.1
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.4.1
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3
test-fedora-35-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3

danepitkin · 2023-09-15T17:04:00Z

I've separated out the enablement of Cython 3 into a separate diff: #37743

Now, we can merge Cython 3 support and then enable Cython 3 once Cython 3.0.3 is released.

danepitkin · 2023-09-15T22:55:42Z

IMO this can be merged now!

pitrou · 2023-09-18T13:26:54Z

python/pyarrow/_dataset.pyx

-        """A Dataset created from a list of paths on a particular filesystem.
+    @staticmethod
+    def from_paths(paths, schema=None, format=None, filesystem=None,
+                   partitions=None, root_partition=None):


Hmm... classmethod doesn't work correctly on Cython 3 anymore? Is it a known issue? Or is this change actually not needed?

classmethod does work, I can change it back. I tried switching to staticmethod when numpydocs were failing, but it turns out it was the comment that was causing numpydoc parsing errors. I thought staticmethod was a slight improvement since cls wasn't actually used in the classmethod.

AlenkaF

Thank you for working on this Dane!! Great work 👍

AlenkaF · 2023-09-21T08:51:21Z

@github-actions crossbow submit -g python

github-actions · 2023-09-21T08:53:02Z

parse() missing 1 required positional argument: 'config'
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/6259391389

AlenkaF · 2023-09-21T09:01:07Z

@github-actions crossbow submit python

github-actions · 2023-09-21T09:02:33Z

parse() missing 1 required positional argument: 'config'
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/6259489413

kou · 2023-09-21T09:07:57Z

We need to fix #37803 to use @github-actions crossbow submit .... :<

AlenkaF · 2023-09-21T09:11:13Z

We need to fix #37803 to use @github-actions crossbow submit .... :<

Thanks! Got it now =)

AlenkaF · 2023-09-21T09:32:57Z

@github-actions crossbow submit -g python

github-actions · 2023-09-21T09:36:56Z

Revision: 3c4f581

Submitted crossbow builds: ursacomputing/crossbow @ actions-d63bfc88b3

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.4.1
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.4.1
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3
test-fedora-35-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3

AlenkaF · 2023-09-21T15:05:48Z

None of the failures look related, wil merge. Thanks again Dane!

conbench-apache-arrow · 2023-09-21T22:34:12Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit e83c23b.

There was 1 benchmark result indicating a performance regression:

Commit Run on ursa-i9-9960x at 2023-09-21 15:41:12Z
- dataframe-to-table (R) with dataset=type_floats, language=R

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

### Rationale for this change Cython 3.0.0 is the latest release. PyArrow should work with Cython 3.0.0. **Cython 3 is not enabled in this diff.** ### What changes are included in this PR? * Don't use `vector[XXX]&&` * Add a declaration for `postincrement` * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#c-postincrement-postdecrement-operator * Ignore `C4551` warning (function call missing argument list) with MSVC * See also: cython/cython#4445 * Add missing `const` to `CLocation`'s static methods. * Don't use `StopIteration` to stop generator * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-3-syntax-semantics * non-extern `cdef` functions will now propagate python exceptions automatically unless explicitly labeled `noexcept` * Function binding in cython is now enabled by default. Class methods that are used as wrappers for pickling should be converted to staticmethods. * Numpydocs now validates more Cython 3 objects than Cython <3 * Enum types are now being validated, and some unhelpful validation checks on Enums are now ignored * Added a cython <3 nightly CI job Note: * Cython 3.0.0, 3.0.1, 3.0.2 has an issue when compiling with debug mode cython/cython#5552 ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * Closes: apache#36730 Lead-authored-by: Dane Pitkin <dane@voltrondata.com> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com>

### Rationale for this change The Cython 3.0.0 upgrade apache#37097 is triggering numpydoc errors for these missing docstrings. ### What changes are included in this PR? * Docstrings added to Cython functions that omitted them ### Are these changes tested? Yes, locally. ### Are there any user-facing changes? User-facing documentation is added. * Closes: apache#37217 Lead-authored-by: Dane Pitkin <dane@voltrondata.com> Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com> Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

### Rationale for this change Cython 3.0.0 is the latest release. PyArrow should work with Cython 3.0.0. **Cython 3 is not enabled in this diff.** ### What changes are included in this PR? * Don't use `vector[XXX]&&` * Add a declaration for `postincrement` * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#c-postincrement-postdecrement-operator * Ignore `C4551` warning (function call missing argument list) with MSVC * See also: cython/cython#4445 * Add missing `const` to `CLocation`'s static methods. * Don't use `StopIteration` to stop generator * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-3-syntax-semantics * non-extern `cdef` functions will now propagate python exceptions automatically unless explicitly labeled `noexcept` * Function binding in cython is now enabled by default. Class methods that are used as wrappers for pickling should be converted to staticmethods. * Numpydocs now validates more Cython 3 objects than Cython <3 * Enum types are now being validated, and some unhelpful validation checks on Enums are now ignored * Added a cython <3 nightly CI job Note: * Cython 3.0.0, 3.0.1, 3.0.2 has an issue when compiling with debug mode cython/cython#5552 ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * Closes: apache#36730 Lead-authored-by: Dane Pitkin <dane@voltrondata.com> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com> Signed-off-by: AlenkaF <frim.alenka@gmail.com>

danepitkin requested review from assignUser, kou, raulcd and lidavidm as code owners August 9, 2023 20:30

github-actions bot added Component: Python Component: FlightRPC Component: Documentation awaiting review Awaiting review and removed Component: Documentation labels Aug 9, 2023

danepitkin removed request for kou, lidavidm, raulcd and assignUser August 9, 2023 20:56

danepitkin force-pushed the danepitkin/cython30 branch from 88db07c to 14c4ce9 Compare August 9, 2023 21:19

danepitkin changed the title ~~WIP: [Python] Add support for Cython 3.0.0~~ Work In Progress: [Python] Add support for Cython 3.0.0 Aug 9, 2023

danepitkin force-pushed the danepitkin/cython30 branch from 14c4ce9 to 62f3ab6 Compare August 9, 2023 21:42

danepitkin mentioned this pull request Aug 10, 2023

GH-36730: [Python] Add support for Cython 3.0.0 #36745

Closed

danepitkin commented Aug 11, 2023

View reviewed changes

python/pyarrow/tests/parquet/test_dataset.py Outdated Show resolved Hide resolved

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 11, 2023

kou changed the title ~~Work In Progress: [Python] Add support for Cython 3.0.0~~ GH-36730: Work In Progress: [Python] Add support for Cython 3.0.0 Aug 11, 2023

danepitkin force-pushed the danepitkin/cython30 branch from 59e70f6 to c8e6a19 Compare August 14, 2023 22:10

danepitkin changed the title ~~GH-36730: Work In Progress: [Python] Add support for Cython 3.0.0~~ GH-36730: [Python] Add support for Cython 3.0.0 Aug 15, 2023

danepitkin requested a review from jorisvandenbossche August 15, 2023 14:37

This was referenced Sep 15, 2023

GH-37742: [Python] Enable Cython 3 #37743

Merged

[CI][Python] Cython 3.0 seems to make our build of pyarrow fail #36730

Closed

danepitkin marked this pull request as ready for review September 15, 2023 18:05

pitrou reviewed Sep 18, 2023

View reviewed changes

danepitkin added 3 commits September 18, 2023 13:34

Add todo, revert back to classmethod

ec5754c

Revert formatting change

eee7573

Revert typo

3c4f581

AlenkaF approved these changes Sep 19, 2023

View reviewed changes

danepitkin added the Priority: Blocker Marks a blocker for the release label Sep 19, 2023

AlenkaF merged commit e83c23b into apache:main Sep 21, 2023
53 of 58 checks passed

AlenkaF removed the awaiting change review Awaiting change review label Sep 21, 2023

assignUser mentioned this pull request Oct 31, 2023

Support in pyarrow for Cython > 3 #38527

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-36730: [Python] Add support for Cython 3.0.0 #37097

GH-36730: [Python] Add support for Cython 3.0.0 #37097

danepitkin commented Aug 9, 2023 •

edited

github-actions bot commented Aug 9, 2023

danepitkin commented Aug 11, 2023 •

edited

kou commented Aug 11, 2023

danepitkin commented Aug 14, 2023

github-actions bot commented Aug 14, 2023

danepitkin commented Aug 15, 2023

github-actions bot commented Sep 15, 2023

danepitkin commented Sep 15, 2023

danepitkin commented Sep 15, 2023

pitrou Sep 18, 2023

danepitkin Sep 18, 2023

AlenkaF left a comment

AlenkaF commented Sep 21, 2023

github-actions bot commented Sep 21, 2023

AlenkaF commented Sep 21, 2023

github-actions bot commented Sep 21, 2023

kou commented Sep 21, 2023

AlenkaF commented Sep 21, 2023 •

edited

AlenkaF commented Sep 21, 2023

github-actions bot commented Sep 21, 2023

AlenkaF commented Sep 21, 2023

conbench-apache-arrow bot commented Sep 21, 2023

GH-36730: [Python] Add support for Cython 3.0.0 #37097

GH-36730: [Python] Add support for Cython 3.0.0 #37097

Conversation

danepitkin commented Aug 9, 2023 • edited

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Aug 9, 2023

danepitkin commented Aug 11, 2023 • edited

kou commented Aug 11, 2023

danepitkin commented Aug 14, 2023

github-actions bot commented Aug 14, 2023

danepitkin commented Aug 15, 2023

github-actions bot commented Sep 15, 2023

danepitkin commented Sep 15, 2023

danepitkin commented Sep 15, 2023

pitrou Sep 18, 2023

Choose a reason for hiding this comment

danepitkin Sep 18, 2023

Choose a reason for hiding this comment

AlenkaF left a comment

Choose a reason for hiding this comment

AlenkaF commented Sep 21, 2023

github-actions bot commented Sep 21, 2023

AlenkaF commented Sep 21, 2023

github-actions bot commented Sep 21, 2023

kou commented Sep 21, 2023

AlenkaF commented Sep 21, 2023 • edited

AlenkaF commented Sep 21, 2023

github-actions bot commented Sep 21, 2023

AlenkaF commented Sep 21, 2023

conbench-apache-arrow bot commented Sep 21, 2023

danepitkin commented Aug 9, 2023 •

edited

danepitkin commented Aug 11, 2023 •

edited

AlenkaF commented Sep 21, 2023 •

edited