GH-36752: [Python] Remove AWS SDK bundling when building wheels #36925

raulcd · 2023-07-28T09:34:00Z

Rationale for this change

In #12227 we decided to use a bundled version of the AWS SDK when compiling Python wheels, in order to downgrade the AWS SDK version.

Now that we have fixed S3 finalization issues (#36442), it should be ok to rely on the vcpkg-installed version of the AWS SDK again.

What changes are included in this PR?

Remove use of bundled AWS SDK and use S3 vcpkg feature for requirements.

Are these changes tested?

On CI and via crossbow

Are there any user-facing changes?

No

Closes: [Python] Remove AWS SDK bundling when building wheels #36752

raulcd · 2023-07-28T09:34:24Z

@github-actions crossbow submit -g wheel

github-actions · 2023-07-28T09:34:31Z

⚠️ GitHub issue #36752 has been automatically assigned in GitHub to PR creator.

raulcd · 2023-07-28T09:36:08Z

@github-actions crossbow submit java-jars

github-actions · 2023-07-28T09:37:28Z

Revision: 887cb1d

Submitted crossbow builds: ursacomputing/crossbow @ actions-fdcf65c07d

Task	Status
wheel-clean
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-mojave-cp310-amd64
wheel-macos-mojave-cp311-amd64
wheel-macos-mojave-cp38-amd64
wheel-macos-mojave-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

github-actions · 2023-07-28T09:38:32Z

Revision: 887cb1d

Submitted crossbow builds: ursacomputing/crossbow @ actions-1b69832c2b

Task	Status
java-jars

raulcd · 2023-08-01T12:47:15Z

@github-actions crossbow submit java-jars

github-actions · 2023-08-01T12:49:47Z

Revision: 241e89c

Submitted crossbow builds: ursacomputing/crossbow @ actions-9d816ef1c2

Task	Status
java-jars

kou · 2023-08-01T21:17:29Z

Can we change https://github.com/apache/arrow/blob/main/ci/scripts/python_wheel_windows_build.bat too?

raulcd · 2023-08-01T21:48:11Z

Can we change https://github.com/apache/arrow/blob/main/ci/scripts/python_wheel_windows_build.bat too?

Thanks @kou, I didn't see that one :)

raulcd · 2023-08-01T21:48:23Z

@github-actions crossbow submit -g wheel

github-actions · 2023-08-01T21:51:22Z

Revision: 1fb7e2c

Submitted crossbow builds: ursacomputing/crossbow @ actions-33ffe17146

Task	Status
wheel-clean
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-mojave-cp310-amd64
wheel-macos-mojave-cp311-amd64
wheel-macos-mojave-cp38-amd64
wheel-macos-mojave-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd · 2023-08-02T06:51:23Z

@github-actions crossbow submit wheel-windows-cp310-amd64

github-actions · 2023-08-02T06:54:08Z

Revision: c04d057

Submitted crossbow builds: ursacomputing/crossbow @ actions-c3320dac1a

Task	Status
wheel-windows-cp310-amd64

raulcd · 2023-08-02T10:28:50Z

The Windows wheel failure is now unrelated, see the above issue.
The build is successful with aws-sdk-cpp from VCPKG, it is failing due to a couple of tests that are failing on main too.

kou

+1

.github/workflows/java_jni.yml

conbench-apache-arrow · 2023-08-11T00:37:09Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 9b3bf08.

There were 3 benchmark results indicating a performance regression:

Commit Run on ursa-i9-9960x at 2023-08-10 22:02:56Z
- file-read (R) with compression=uncompressed, dataset=fanniemae_2016Q4, file_type=feather, language=R, output_type=table
- dataframe-to-table (R) with dataset=type_floats, language=R
and 1 more (see the report linked below)

The full Conbench report has more details.

…apache#36925) ### Rationale for this change In apache#12227 we decided to use a bundled version of the AWS SDK when compiling Python wheels, in order to downgrade the AWS SDK version. Now that we have fixed S3 finalization issues (apache#36442), it should be ok to rely on the vcpkg-installed version of the AWS SDK again. ### What changes are included in this PR? Remove use of bundled AWS SDK and use S3 vcpkg feature for requirements. ### Are these changes tested? On CI and via crossbow ### Are there any user-facing changes? No * Closes: apache#36752 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

raulcd requested review from assignUser and kou as code owners July 28, 2023 09:34

github-actions bot added the awaiting committer review Awaiting committer review label Jul 28, 2023

raulcd added 3 commits August 1, 2023 14:41

apacheGH-36752: [Python] Remove AWS SDK bundling when building wheels

2a9be02

Temporary increase timeout to allow vcpkg to build

d6a45dd

Try to give the build an extremely amount of time

241e89c

raulcd force-pushed the GH-36752 branch from 176d5ac to 241e89c Compare August 1, 2023 12:44

Remove AWSSDK_SOURCE=BUNDLED from Windows wheels

1fb7e2c

Update Windows wheel revision to rebuild with VCPKG S3 feature enabled

c04d057

raulcd mentioned this pull request Aug 2, 2023

[Python][Packaging] Windows wheels fail due to Cannot locate timezone 'UTC': Timezone database not found at "C:\Users\ContainerAdministrator\Downloads\tzdata" #36991

Closed

kou approved these changes Aug 2, 2023

View reviewed changes

.github/workflows/java_jni.yml Outdated Show resolved Hide resolved

github-actions bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Aug 2, 2023

Revert timeout increase to build Image

24bb448

github-actions bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Aug 3, 2023

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 3, 2023

kou merged commit 9b3bf08 into apache:main Aug 3, 2023
51 of 53 checks passed

kou removed the awaiting change review Awaiting change review label Aug 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-36752: [Python] Remove AWS SDK bundling when building wheels #36925

GH-36752: [Python] Remove AWS SDK bundling when building wheels #36925

raulcd commented Jul 28, 2023 •

edited by github-actions bot

Loading

raulcd commented Jul 28, 2023

github-actions bot commented Jul 28, 2023

raulcd commented Jul 28, 2023

github-actions bot commented Jul 28, 2023

github-actions bot commented Jul 28, 2023

raulcd commented Aug 1, 2023

github-actions bot commented Aug 1, 2023

kou commented Aug 1, 2023

raulcd commented Aug 1, 2023

raulcd commented Aug 1, 2023

github-actions bot commented Aug 1, 2023

raulcd commented Aug 2, 2023

github-actions bot commented Aug 2, 2023

raulcd commented Aug 2, 2023

kou left a comment

conbench-apache-arrow bot commented Aug 11, 2023

GH-36752: [Python] Remove AWS SDK bundling when building wheels #36925

GH-36752: [Python] Remove AWS SDK bundling when building wheels #36925

Conversation

raulcd commented Jul 28, 2023 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

raulcd commented Jul 28, 2023

github-actions bot commented Jul 28, 2023

raulcd commented Jul 28, 2023

github-actions bot commented Jul 28, 2023

github-actions bot commented Jul 28, 2023

raulcd commented Aug 1, 2023

github-actions bot commented Aug 1, 2023

kou commented Aug 1, 2023

raulcd commented Aug 1, 2023

raulcd commented Aug 1, 2023

github-actions bot commented Aug 1, 2023

raulcd commented Aug 2, 2023

github-actions bot commented Aug 2, 2023

raulcd commented Aug 2, 2023

kou left a comment

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Aug 11, 2023

raulcd commented Jul 28, 2023 •

edited by github-actions bot

Loading