ARROW-18054: [Python][CI] Enable Cython tests on windows wheels #13552

raulcd · 2022-07-08T16:15:33Z

No description provided.

raulcd · 2022-07-08T16:15:50Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-07-08T16:16:51Z

Revision: ba8c2fb1d07534eefd32ab5e01831f0eacf37fa1

Submitted crossbow builds: ursacomputing/crossbow @ actions-34a58c728d

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-07-08T21:00:20Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-07-08T21:01:35Z

Revision: ea011f119f60124e1594609b6cfa1749b6513fc8

Submitted crossbow builds: ursacomputing/crossbow @ actions-e97d2d0170

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

pitrou · 2022-10-05T16:15:36Z

@raulcd Would you like to rebase from master and try again?

raulcd · 2022-10-06T09:04:30Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-10-06T09:06:03Z

Revision: 615ceec9e41d84a144131c9b9648cbe0b4753a01

Submitted crossbow builds: ursacomputing/crossbow @ actions-b0f1331674

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-10-06T10:47:26Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-10-06T10:48:55Z

Revision: 126d6a23a410e82731770baec08d4c9ca07b62d1

Submitted crossbow builds: ursacomputing/crossbow @ actions-3ecf3c6c3a

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-10-06T13:43:42Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-10-06T13:53:40Z

Revision: ffa9326882d9615af7610c57316e8cfe0c7540ee

Submitted crossbow builds: ursacomputing/crossbow @ actions-ee5fcec155

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-10-06T14:18:24Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-10-06T14:19:59Z

Revision: 8a0554b8509cadf38461d13f8b8c025991cfb820

Submitted crossbow builds: ursacomputing/crossbow @ actions-94bb2917f4

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-10-07T09:35:15Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-10-07T09:37:04Z

Revision: 49b13998271a4cce6ae42a18207d7b18b7d55713

Submitted crossbow builds: ursacomputing/crossbow @ actions-44417c02af

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-10-07T10:55:32Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-10-07T10:57:08Z

Revision: 34a9915fc57bdf5c7a407114449173b452c67b19

Submitted crossbow builds: ursacomputing/crossbow @ actions-c6b9cae0e0

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-10-07T13:15:45Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-10-07T13:17:26Z

Revision: ade8e6e19bf7bf6d6fbd5264380bda4ab14a8e71

Submitted crossbow builds: ursacomputing/crossbow @ actions-bba6f3f4e4

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-10-07T15:28:12Z

@github-actions crossbow submit wheel-windows-cp310-amd64

github-actions · 2022-10-07T15:29:51Z

Revision: 080549c05c9f07acf00addab5059da55b618cc68

Submitted crossbow builds: ursacomputing/crossbow @ actions-ffbd56ba01

Task	Status
wheel-windows-cp310-amd64

raulcd · 2022-10-13T10:53:19Z

@github-actions crossbow submit wheel-windows-cp310-amd64

github-actions · 2022-10-13T11:38:45Z

Revision: 9bfd2c52108400e85f6d2442d7520793f427cb72

Submitted crossbow builds: ursacomputing/crossbow @ actions-ed4aeb0591

Task	Status
wheel-windows-cp310-amd64

raulcd · 2022-10-13T13:12:39Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-10-13T17:40:08Z

Revision: 63304cfbcf31aaff97a1e6d4310dc4f46aadec39

Submitted crossbow builds: ursacomputing/crossbow @ actions-40e49746b6

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-10-14T09:13:41Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-10-14T10:09:44Z

Revision: d5fe64c

Submitted crossbow builds: ursacomputing/crossbow @ actions-01cce0a550

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-10-14T13:15:31Z

@jorisvandenbossche @AlenkaF cython tests on windows wheels for different python versions are working and enabled with this PR now

github-actions · 2022-10-14T13:46:40Z

https://issues.apache.org/jira/browse/ARROW-18054

AlenkaF · 2022-10-17T06:39:35Z

Thank you for the ping and for the change in test_cython.py! LGTM +1

I did notice we are testing PyArrow C++ separately in python_test_sh which is not needed anymore after #14117.
Created an issue for that: https://issues.apache.org/jira/browse/ARROW-18074

jorisvandenbossche

The changes to test_cython.py look good to me!

python/pyarrow/tests/test_cython.py

pitrou · 2022-10-18T15:57:11Z

ci/docker/python-wheel-windows-test-vs2017.dockerfile

+# NOTE: a more recent base image (tried with 2.12.1) comes with python 3.9.7
+# and the msi installers are failing to remove pip and tcl/tk "products" making
+# the subsequent choco python installation step failing for installing python
+# version 3.9.* due to existing python version


For the record, are there other upstream images that we could use?

We could create our own images with Visual C++ based from the official Microsoft ones FROM mcr.microsoft.com/windows/servercore:ltsc2019 example here: https://learn.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2017/install/build-tools-container?view=vs-2017 but I wanted to use the same one we have on the python-wheels for consistency: https://github.com/apache/arrow/blob/master/ci/docker/python-wheel-windows-vs2017.dockerfile#L23
There was someone working on updating the Visual C++ version to 2019, maybe I could try to create new Dockerfile images removing the abrarov/msvc-2017 base image and create new ones based on the Microsoft ones mcr.microsoft.com/windows/servercore?
I would prefer to do that as a different ticket if you are ok with that.

There was someone working on updating the Visual C++ version to 2019

That was me, but it was precisely stalled because of the "uninstall then install Python" problem.

I would prefer to do that as a different ticket if you are ok with that.

That's fine with me!

python/pyarrow/tests/test_cython.py

… present

raulcd · 2022-10-19T08:49:07Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-10-19T10:21:42Z

Revision: 0c05f7f

Submitted crossbow builds: ursacomputing/crossbow @ actions-9b76be2fa9

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-10-19T13:09:10Z

@github-actions crossbow submit wheel-windows*

github-actions · 2022-10-19T13:15:04Z

Revision: 3ff78d0

Submitted crossbow builds: ursacomputing/crossbow @ actions-7d4dffa6bd

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2022-10-21T10:49:15Z

@pitrou let me know if there is anything else you think should be done here. Also, would you prefer me to create a new ticket for the Dockerfiles or should I pick it up as part of https://issues.apache.org/jira/browse/ARROW-17592, the one around migrating to VS 2019.

jorisvandenbossche · 2022-10-25T07:35:26Z

This is ready to go?

pitrou · 2022-10-25T07:38:16Z

Also, would you prefer me to create a new ticket for the Dockerfiles or should I pick it up as part of https://issues.apache.org/jira/browse/ARROW-17592, the one around migrating to VS 2019.

You can pick it up as part of ARROW-17592 IMHO.

pitrou · 2022-10-25T07:39:06Z

This is ready to go?

Yes, let's merge this.

ursabot · 2022-10-27T18:22:26Z

Benchmark runs are scheduled for baseline = 1ae666c and contender = bbc9247. bbc9247 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Finished ⬇️0.27% ⬆️0.0%] ursa-i9-9960x
[Failed] ursa-thinkcentre-m75q
Buildkite builds:
[Failed] bbc9247d ec2-t3-xlarge-us-east-2
[Failed] bbc9247d test-mac-arm
[Finished] bbc9247d ursa-i9-9960x
[Failed] bbc9247d ursa-thinkcentre-m75q
[Failed] 1ae666c3 ec2-t3-xlarge-us-east-2
[Failed] 1ae666c3 test-mac-arm
[Finished] 1ae666c3 ursa-i9-9960x
[Failed] 1ae666c3 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>

raulcd force-pushed the enable-cython-on-wheels branch from ba8c2fb to ea011f1 Compare July 8, 2022 21:00

raulcd force-pushed the enable-cython-on-wheels branch from ea011f1 to 615ceec Compare October 6, 2022 09:02

github-actions bot added the Component: Python label Oct 6, 2022

raulcd force-pushed the enable-cython-on-wheels branch from 080549c to 9bfd2c5 Compare October 13, 2022 10:52

Enable Cython tests on windows wheels

2bca6bd

raulcd added 4 commits October 14, 2022 11:05

Add some debugging to understand where the failures are coming from

785078f

Add os.add_dll_directory for cython test on acse of python > 3.8

469961c

Fix non-windows tests

422831d

Clean up some debugging

d5fe64c

raulcd force-pushed the enable-cython-on-wheels branch from 63304cf to d5fe64c Compare October 14, 2022 09:09

raulcd changed the title ~~Enable Cython tests on windows wheels~~ ARROW-18054: [Python][CI] Enable Cython tests on windows wheels Oct 14, 2022

raulcd marked this pull request as ready for review October 14, 2022 13:13

jorisvandenbossche approved these changes Oct 18, 2022

View reviewed changes

jorisvandenbossche mentioned this pull request Oct 18, 2022

ARROW-17892: [CI] Use Python 3.10 in AppVeyor build #14307

Merged

pitrou reviewed Oct 18, 2022

View reviewed changes

python/pyarrow/tests/test_cython.py Outdated Show resolved Hide resolved

pitrou reviewed Oct 18, 2022

View reviewed changes

jorisvandenbossche reviewed Oct 18, 2022

View reviewed changes

python/pyarrow/tests/test_cython.py Outdated Show resolved Hide resolved

Fix not setting LD_LIBRARY_PATH on windows if os.add_dll_directory is…

0c05f7f

… present

Only update PATH/LD_LIBRARY_PATH if it is necessary

3ff78d0

pitrou merged commit bbc9247 into apache:master Oct 25, 2022

kou pushed a commit that referenced this pull request Nov 15, 2022

ARROW-18054: [Python][CI] Enable Cython tests on windows wheels (#13552)

5980976

Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>

ARROW-18054: [Python][CI] Enable Cython tests on windows wheels #13552

ARROW-18054: [Python][CI] Enable Cython tests on windows wheels #13552

Uh oh!

Conversation

raulcd commented Jul 8, 2022

Uh oh!

raulcd commented Jul 8, 2022

Uh oh!

github-actions bot commented Jul 8, 2022

Uh oh!

raulcd commented Jul 8, 2022

Uh oh!

github-actions bot commented Jul 8, 2022

Uh oh!

pitrou commented Oct 5, 2022

Uh oh!

raulcd commented Oct 6, 2022

Uh oh!

github-actions bot commented Oct 6, 2022

Uh oh!

raulcd commented Oct 6, 2022

Uh oh!

github-actions bot commented Oct 6, 2022

Uh oh!

raulcd commented Oct 6, 2022

Uh oh!

github-actions bot commented Oct 6, 2022

Uh oh!

raulcd commented Oct 6, 2022

Uh oh!

github-actions bot commented Oct 6, 2022

Uh oh!

raulcd commented Oct 7, 2022

Uh oh!

github-actions bot commented Oct 7, 2022

Uh oh!

raulcd commented Oct 7, 2022

Uh oh!

github-actions bot commented Oct 7, 2022

Uh oh!

raulcd commented Oct 7, 2022

Uh oh!

github-actions bot commented Oct 7, 2022

Uh oh!

raulcd commented Oct 7, 2022

Uh oh!

github-actions bot commented Oct 7, 2022

Uh oh!

raulcd commented Oct 13, 2022

Uh oh!

github-actions bot commented Oct 13, 2022

Uh oh!

raulcd commented Oct 13, 2022

Uh oh!

github-actions bot commented Oct 13, 2022

Uh oh!

raulcd commented Oct 14, 2022

Uh oh!

github-actions bot commented Oct 14, 2022

Uh oh!

raulcd commented Oct 14, 2022

Uh oh!

github-actions bot commented Oct 14, 2022

Uh oh!

AlenkaF commented Oct 17, 2022

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pitrou Oct 18, 2022

Choose a reason for hiding this comment

Uh oh!

raulcd Oct 19, 2022

Choose a reason for hiding this comment

Uh oh!

pitrou Oct 19, 2022

Choose a reason for hiding this comment

Uh oh!

pitrou Oct 19, 2022