GH-39621: [CI][Packaging] Update vcpkg to 2023.11.20 release #39622

Tom-Newton · 2024-01-15T21:10:19Z

Rationale for this change

Old version of vcpkg is blocking #39352

What changes are included in this PR?

Upgrade vcpkg
Fix ports patches
Upgrade visual studio used in windows wheel builds. VS2019 is now required for the vcpkg abseil build.
Move VCPKG_FORCE_SYSTEM_BINARIES to be set before vcpkg install to fix vcpkg install on linux ARM.
Fix for LLVM 17 which requires that an executable exports "llvm_orc_registerEHFrameSectionWrapper()" and "llvm_orc_unregisterEHFrameSectionWrapper()". This effects the java builds that depend on llvm from vcpkg for gandiva.
Update image used for python wheel builds on windows to 2024-02-05

Are these changes tested?

Does not change any behaviour so should be covered by existing tests

Are there any user-facing changes?

There shouldn't be

Closes: [CI][Packaging] Upgrade vcpkg version used for release builds #39621

Tom-Newton · 2024-01-15T22:41:58Z

The standard tests are all passing so I think at this point I need a review from someone who knows how this works 😅

assignUser · 2024-01-16T00:26:15Z

@github-actions crossbow submit -g vcpkg

kou · 2024-01-16T00:28:19Z

@github-actions crossbow submit -g wheel

github-actions · 2024-01-16T00:28:28Z

Revision: df9d5a2

Submitted crossbow builds: ursacomputing/crossbow @ actions-e1c32d09f5

Task	Status
test-build-vcpkg-win

github-actions · 2024-01-16T00:31:08Z

Revision: df9d5a2

Submitted crossbow builds: ursacomputing/crossbow @ actions-5775383cc3

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-catalina-cp310-amd64
wheel-macos-catalina-cp311-amd64
wheel-macos-catalina-cp312-amd64
wheel-macos-catalina-cp38-amd64
wheel-macos-catalina-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

Tom-Newton · 2024-01-16T08:37:38Z

Thanks for launching all the tests. I will investigate the failures

Tom-Newton · 2024-01-16T08:57:58Z

Summary of failures:
All windows builds error: building abseil:amd64-windows-static-md-release failed with: BUILD_FAILED
e.g. https://github.com/ursacomputing/crossbow/actions/runs/7535390294/job/20511298098
Failures during cmake configure. Exact failure is not obvious because it writes logs to a file that I can't open. Maybe updating to the very latest vcpkg will fix it.

Linux ARM builds Environment variable VCPKG_FORCE_SYSTEM_BINARIES must be set on arm, s390x, ppc64le and riscv platforms. e.g. https://github.com/ursacomputing/crossbow/actions/runs/7535390437/job/20511298713
Fails when installing vcpkg.

1 flake 502 Bad Gateway https://github.com/ursacomputing/crossbow/actions/runs/7535390424/job/20511298624

2 of the mac OS builds Fatal Python error: Aborted during tests at File "/Users/runner/work/crossbow/crossbow/test-amd64-env/lib/python3.11/site-packages/pyarrow/tests/test_dataset.py", line 3753 in test_parquet_dataset_lazy_filtering
e.g. https://github.com/ursacomputing/crossbow/actions/runs/7535390473/job/20511298827
I'm not really sure what this means. Is this what a segfault looks like on mac OS?

raulcd · 2024-01-16T09:02:57Z

The macOS failures are known issues being worked on here: #39562

raulcd · 2024-01-16T10:56:00Z

For the linux ARM failures could you try this:

diff --git a/ci/docker/python-wheel-manylinux.dockerfile b/ci/docker/python-wheel-manylinux.dockerfile
index 0a50d45..3ce3272 100644
--- a/ci/docker/python-wheel-manylinux.dockerfile
+++ b/ci/docker/python-wheel-manylinux.dockerfile
@@ -62,15 +62,16 @@ COPY ci/vcpkg/*.patch \
 COPY ci/scripts/install_vcpkg.sh \
      arrow/ci/scripts/
 ENV VCPKG_ROOT=/opt/vcpkg
-RUN arrow/ci/scripts/install_vcpkg.sh ${VCPKG_ROOT} ${vcpkg}
-ENV PATH="${PATH}:${VCPKG_ROOT}"
-
 ARG build_type=release
 ENV CMAKE_BUILD_TYPE=${build_type} \
     VCPKG_FORCE_SYSTEM_BINARIES=1 \
     VCPKG_OVERLAY_TRIPLETS=/arrow/ci/vcpkg \
     VCPKG_DEFAULT_TRIPLET=${arch_short}-linux-static-${build_type} \
     VCPKG_FEATURE_FLAGS="manifests"
+
+RUN arrow/ci/scripts/install_vcpkg.sh ${VCPKG_ROOT} ${vcpkg}
+ENV PATH="${PATH}:${VCPKG_ROOT}"
+
 COPY ci/vcpkg/vcpkg.json arrow/ci/vcpkg/
 # cannot use the S3 feature here because while aws-sdk-cpp=1.9.160 contains
 # ssl related fixes as well as we can patch the vcpkg portfile to support

Basically setting the required environment variable before installing vcpkg instead of after as we are already doing so.

Tom-Newton · 2024-01-16T19:54:00Z

@github-actions crossbow submit -g wheel

github-actions · 2024-01-16T19:54:31Z

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/7546803265

Tom-Newton · 2024-01-16T19:55:35Z

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/7546803265

😞 It appears greater than contributor status is required.

raulcd · 2024-01-16T20:30:18Z

@github-actions crossbow submit -g wheel

github-actions · 2024-01-16T20:32:56Z

Revision: c10f601

Submitted crossbow builds: ursacomputing/crossbow @ actions-a18c25bf37

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-catalina-cp310-amd64
wheel-macos-catalina-cp311-amd64
wheel-macos-catalina-cp312-amd64
wheel-macos-catalina-cp38-amd64
wheel-macos-catalina-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

Tom-Newton · 2024-01-16T21:02:58Z

I just pushed another commit to use the very latest version of vcpkg. My hope is that this will fix the windows build. (#39622 (comment) is without this update)

raulcd · 2024-01-17T09:47:58Z

@github-actions crossbow submit wheel-windows-*

github-actions · 2024-01-17T09:50:15Z

Revision: 3e3bb83

Submitted crossbow builds: ursacomputing/crossbow @ actions-6216612a43

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

Tom-Newton · 2024-01-17T10:30:21Z

Looks like we are getting the same failures on the windows builds so will need to investigate further. Additionally the 2024-01-11 release seems to have broken the java manylinux build in the normal CI. The was working on the 2023-11-20 release so its quite tempting to go back to that.

Tom-Newton · 2024-01-21T15:48:42Z

I managed to get the windows docker build working locally. The abseil build error is

"This package requires Visual Studio 2019 (MSVC++ 16.0) or higher."

I don't know what implications are for upgrading visual studio but I will give it a try.

It also seems like windows build is broken on main though. There were lots of download failures:

error: Failed to download from mirror set
error: https://repo.msys2.org/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
error: https://www2.futureware.at/~nickoe/msys2-mirror/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
error: https://mirror.yandex.ru/mirrors/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
error: https://mirrors.tuna.tsinghua.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
error: https://mirrors.ustc.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
error: https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: WinHttpSendRequest failed with exit code 12007
error: https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: WinHttpSendRequest failed with exit code 12007
error: https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: WinHttpSendRequest failed with exit code 12007
error: https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: WinHttpSendRequest failed with exit code 12007
error: https://mirror.selfnet.de/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
error: https://mirrors.sjtug.sjtu.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404

Checking some of these URLs in a browser they all seem to variants of not found errors.

kou · 2024-01-22T02:40:42Z

@github-actions crossbow submit wheel-windows-cp312-amd64

github-actions · 2024-01-22T02:43:20Z

Revision: 7ce22d2

Submitted crossbow builds: ursacomputing/crossbow @ actions-776e952a91

Task	Status
wheel-windows-cp312-amd64

Tom-Newton · 2024-02-04T18:48:18Z

I managed to get the java builds working properly locally. Going back to the 2023.11.20 release solves one of the java errors. I'm sure all the normal CI was passing when I first opened this PR for 2023.11.20. I'm hoping it will also resolve the ARM build problems.

Tom-Newton · 2024-02-04T23:34:46Z

ARM build in normal CI seems fine now.

AMD64 Ubuntu 22.04 Java JDK 21 Maven 3.9.5 appears to be flaky. Locally is succeeds and on a previous CI run I think I saw the same error but for JDK 17 instead of 21. I'm not sure if this flakiness was created by this PR - possibly #38719 was not 100% effective. I see a few existing issues around this area but nothing obvious to indicate that the problem still exists on main. It looks like the failure is happening when trying to close the root allocator after all the child allocators have been successfully closed, so I suppose that means the leak was directly on the root allocator.

The R builds appear to be failing for the same reason as on main #39933

kou · 2024-02-05T00:06:42Z

@github-actions crossbow submit java-jars

github-actions · 2024-02-05T00:09:04Z

Revision: 8764067

Submitted crossbow builds: ursacomputing/crossbow @ actions-a21ab902a6

Task	Status
java-jars

Tom-Newton · 2024-02-05T08:45:23Z

@kou please could you re-trigger archery builds. It looks like the most recent run succeeded apart from one which failed with a timeout reading from the cache https://github.com/ursacomputing/crossbow/actions/runs/7777712547/job/21206359154

kou · 2024-02-05T08:53:38Z

Done! (Re-ran)

raulcd · 2024-02-05T10:56:17Z

@github-actions crossbow submit -g wheel

github-actions · 2024-02-05T10:59:17Z

Revision: 8764067

Submitted crossbow builds: ursacomputing/crossbow @ actions-5e4cec918f

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-catalina-cp310-amd64
wheel-macos-catalina-cp311-amd64
wheel-macos-catalina-cp312-amd64
wheel-macos-catalina-cp38-amd64
wheel-macos-catalina-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

raulcd

Thanks a lot for the effort @Tom-Newton !
Just a minor nit but if the latest jobs are successful I think this is ready to get merged. Unless @kou has any other comment.

.env

raulcd · 2024-02-05T11:08:25Z

The manylinux_2_28 failures are unrelated and I've created and issue and a PR to fix them: #39943

Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>

kou

+1

Thanks!

conbench-apache-arrow · 2024-02-06T11:33:34Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 1552510.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

…pache#39622) ### Rationale for this change Old version of vcpkg is blocking apache#39352 ### What changes are included in this PR? - Upgrade vcpkg - Fix ports patches - Upgrade visual studio used in windows wheel builds. VS2019 is now required for the vcpkg `abseil` build. - Move `VCPKG_FORCE_SYSTEM_BINARIES` to be set before vcpkg install to fix vcpkg install on linux ARM. - Fix for LLVM 17 which requires that an executable exports "llvm_orc_registerEHFrameSectionWrapper()" and "llvm_orc_unregisterEHFrameSectionWrapper()". This effects the java builds that depend on llvm from vcpkg for gandiva. - Update image used for python wheel builds on windows to 2024-02-05 ### Are these changes tested? Does not change any behaviour so should be covered by existing tests ### Are there any user-facing changes? There shouldn't be * Closes: apache#39621 Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

github-actions bot added the awaiting review Awaiting review label Jan 15, 2024

Tom-Newton marked this pull request as ready for review January 15, 2024 22:42

Tom-Newton requested review from assignUser, kou and raulcd as code owners January 15, 2024 22:42

kou changed the title ~~GH-39621: Update vcpkg to 2023.11.20 release~~ GH-39621: [CI][Packaging] Update vcpkg to 2023.11.20 release Jan 16, 2024

This comment was marked as outdated.

Sign in to view

Tom-Newton force-pushed the tomnewton/update_vcpkg/GH-39621 branch from 8ea60d1 to 7ce22d2 Compare January 21, 2024 19:27

raulcd reviewed Feb 5, 2024

View reviewed changes

.env Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes awaiting review Awaiting review and removed awaiting review Awaiting review labels Feb 5, 2024

Update .env

9c94e27

Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>

github-actions bot added awaiting review Awaiting review awaiting committer review Awaiting committer review and removed awaiting review Awaiting review awaiting changes Awaiting changes labels Feb 5, 2024

kou approved these changes Feb 6, 2024

View reviewed changes

kou merged commit 1552510 into apache:main Feb 6, 2024
55 of 57 checks passed

kou removed the awaiting committer review Awaiting committer review label Feb 6, 2024

github-actions bot added the awaiting merge Awaiting merge label Feb 6, 2024

lidavidm mentioned this pull request Feb 6, 2024

ci: update vcpkg version to stay with upstream Arrow apache/arrow-adbc#1518

Closed

jorisvandenbossche mentioned this pull request Mar 12, 2024

GH-40376: [Python] Update for NumPy 2.0 ABI change in PyArray_Descr->elsize #40418

Merged

jorisvandenbossche mentioned this pull request Mar 25, 2024

GH-36026: [C++][ORC] Catch all ORC exceptions to avoid crash #40697

Merged

GH-39621: [CI][Packaging] Update vcpkg to 2023.11.20 release #39622

GH-39621: [CI][Packaging] Update vcpkg to 2023.11.20 release #39622

Conversation

Tom-Newton commented Jan 15, 2024 • edited

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Tom-Newton commented Jan 15, 2024 • edited

assignUser commented Jan 16, 2024

kou commented Jan 16, 2024

github-actions bot commented Jan 16, 2024

github-actions bot commented Jan 16, 2024

Tom-Newton commented Jan 16, 2024

Tom-Newton commented Jan 16, 2024 • edited

raulcd commented Jan 16, 2024

raulcd commented Jan 16, 2024

Tom-Newton commented Jan 16, 2024

github-actions bot commented Jan 16, 2024

Tom-Newton commented Jan 16, 2024 • edited

raulcd commented Jan 16, 2024

github-actions bot commented Jan 16, 2024

Tom-Newton commented Jan 16, 2024

This comment was marked as outdated.

This comment was marked as outdated.

raulcd commented Jan 17, 2024

github-actions bot commented Jan 17, 2024

Tom-Newton commented Jan 17, 2024 • edited

Tom-Newton commented Jan 21, 2024 • edited

kou commented Jan 22, 2024

github-actions bot commented Jan 22, 2024

Tom-Newton commented Feb 4, 2024 • edited

Tom-Newton commented Feb 4, 2024 • edited

kou commented Feb 5, 2024

github-actions bot commented Feb 5, 2024

Tom-Newton commented Feb 5, 2024 • edited

kou commented Feb 5, 2024

raulcd commented Feb 5, 2024

github-actions bot commented Feb 5, 2024

raulcd left a comment

Choose a reason for hiding this comment

raulcd commented Feb 5, 2024

kou left a comment

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Feb 6, 2024

Tom-Newton commented Jan 15, 2024 •

edited

Tom-Newton commented Jan 15, 2024 •

edited

Tom-Newton commented Jan 16, 2024 •

edited

Tom-Newton commented Jan 16, 2024 •

edited

Tom-Newton commented Jan 17, 2024 •

edited

Tom-Newton commented Jan 21, 2024 •

edited

Tom-Newton commented Feb 4, 2024 •

edited

Tom-Newton commented Feb 4, 2024 •

edited

Tom-Newton commented Feb 5, 2024 •

edited