GH-43416: [CI] Upgrade our vcpkg version on .env #43417

raulcd · 2024-07-25T09:48:26Z

Rationale for this change

We are using a pretty old version and we are seeing some CI jobs.

What changes are included in this PR?

Upgrade vcpkg version

Are these changes tested?

on CI

Are there any user-facing changes?

No

GitHub Issue: [CI][C++] Vcpkg failures building some wheels #43416

github-actions · 2024-07-25T09:48:50Z

⚠️ GitHub issue #43416 has been automatically assigned in GitHub to PR creator.

raulcd · 2024-07-25T09:54:53Z

@github-actions crossbow submit -g vcpkg wheel java-jars

github-actions · 2024-07-25T09:56:01Z

Unable to match any tasks for `wheel`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/10092011041

raulcd · 2024-07-25T09:57:09Z

@github-actions crossbow submit -g vcpkg -g wheel java-jars

raulcd · 2024-07-25T10:53:25Z

@github-actions crossbow submit python-wheel-manylinux-2-28

github-actions · 2024-07-25T10:54:32Z

Unable to match any tasks for `python-wheel-manylinux-2-28`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/10092813787

raulcd · 2024-07-25T10:59:37Z

@github-actions crossbow submit wheel-manylinux-2-28-cp39-amd64

github-actions · 2024-07-25T11:01:50Z

Revision: a60bc06

Submitted crossbow builds: ursacomputing/crossbow @ actions-243699443d

Task	Status
wheel-manylinux-2-28-cp39-amd64

raulcd · 2024-07-25T14:59:23Z

I am unsure why ORC is failing to build when calling fetchcontent_makeavailable(orc) with:

CMake Error at /tmp/arrow-build/_deps/orc-src/cmake_modules/ThirdpartyToolchain.cmake:467 (elseif):
  given arguments:

    "ORC_PREFER_STATIC_PROTOBUF" "AND"

  Unknown arguments specified
Call Stack (most recent call first):
  /tmp/arrow-build/_deps/orc-src/CMakeLists.txt:166 (INCLUDE)

I can see we define (since #43011):

    set(ORC_PREFER_STATIC_PROTOBUF
        OFF
        CACHE BOOL "" FORCE)

debugging locally I can see we use ORC 2.0.1 and the failure is coming from:

if (ORC_PACKAGE_KIND STREQUAL "conan")
  target_link_libraries (orc_protobuf INTERFACE ${protobuf_LIBRARIES})
elseif (ORC_PREFER_STATIC_PROTOBUF AND ${PROTOBUF_STATIC_LIB})
  target_link_libraries (orc_protobuf INTERFACE ${PROTOBUF_STATIC_LIB})
else ()
  target_link_libraries (orc_protobuf INTERFACE ${PROTOBUF_LIBRARY})
endif()

@kou any idea why fetchcontent building ORC might not see the defined ORC_PREFER_STATIC_PROTOBUF?

raulcd · 2024-07-26T10:02:23Z

@github-actions crossbow submit wheel-manylinux-2-28-cp39-amd64

raulcd · 2024-07-26T10:04:05Z

I am not sure why but applying the patch here: ee89232 is able to build orc

edited commit

github-actions · 2024-07-26T10:04:35Z

Revision: 982a6ce

Submitted crossbow builds: ursacomputing/crossbow @ actions-46a723f68f

Task	Status
wheel-manylinux-2-28-cp39-amd64

raulcd · 2024-07-26T15:24:25Z

I am currently trying to understand why protoc fails on flight with SIGSEGV:

[143/176] Generating Flight.pb.cc, Flight.pb.h, Flight.grpc.pb.cc, Flight.grpc.pb.h
FAILED: src/arrow/flight/Flight.pb.cc src/arrow/flight/Flight.pb.h src/arrow/flight/Flight.grpc.pb.cc src/arrow/flight/Flight.grpc.pb.h /tmp/arrow-build/src/arrow/flight/Flight.pb.cc /tmp/arrow-build/src/arrow/flight/Flight.pb.h /tmp/arrow-build/src/arrow/flight/Flight.grpc.pb.cc /tmp/arrow-build/src/arrow/flight/Flight.grpc.pb.h 
cd /tmp/arrow-build/src/arrow/flight && /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc -I/arrow/cpp/../format --cpp_out=/tmp/arrow-build/src/arrow/flight /arrow/cpp/../format/Flight.proto && /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc -I/arrow/cpp/../format --grpc_out=/tmp/arrow-build/src/arrow/flight --plugin=protoc-gen-grpc=/opt/vcpkg/installed/amd64-linux-static-release/../x64-linux/tools/grpc/grpc_cpp_plugin /arrow/cpp/../format/Flight.proto
--grpc_out: protoc-gen-grpc: Plugin killed by signal 11.

raulcd · 2024-07-26T15:26:03Z

@github-actions crossbow submit wheel-macos-big-sur-cp310-arm64

github-actions · 2024-07-26T15:28:27Z

Revision: 2cedbe6

Submitted crossbow builds: ursacomputing/crossbow @ actions-08db436a21

Task	Status
wheel-macos-big-sur-cp310-arm64

kou · 2024-07-29T00:58:41Z

Hmm. It seems that https://github.com/apache/orc/blob/23044d79a26cf8792bcb1936ab601308b0781962/cmake_modules/ThirdpartyToolchain.cmake#L416-L418 wasn't used. It means that Protobuf_ROOT is an empty string in this case:

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Lines 4524 to 4529 in 187197c

    
           get_target_property(PROTOBUF_INCLUDE_DIR ${ARROW_PROTOBUF_LIBPROTOBUF} 
        
                               INTERFACE_INCLUDE_DIRECTORIES) 
        
           get_filename_component(Protobuf_ROOT "${PROTOBUF_INCLUDE_DIR}" DIRECTORY) 
        
           set(PROTOBUF_HOME 
        
               ${Protobuf_ROOT} 
        
               CACHE STRING "" FORCE)

Could you debug print Protobuf_ROOT value?

BTW, the ORC code should be fixed:

diff --git a/cmake_modules/ThirdpartyToolchain.cmake b/cmake_modules/ThirdpartyToolchain.cmake
index 79324984a..f0808715b 100644
--- a/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cmake_modules/ThirdpartyToolchain.cmake
@@ -466,13 +466,13 @@ if (ORC_PACKAGE_KIND STREQUAL "conan")
 elseif (NOT "${PROTOBUF_HOME}" STREQUAL "")
   find_package (Protobuf REQUIRED)
 
-  if (ORC_PREFER_STATIC_PROTOBUF AND ${PROTOBUF_STATIC_LIB})
+  if (ORC_PREFER_STATIC_PROTOBUF AND PROTOBUF_STATIC_LIB)
     add_resolved_library (orc_protobuf ${PROTOBUF_STATIC_LIB} ${PROTOBUF_INCLUDE_DIR})
   else ()
     add_resolved_library (orc_protobuf ${PROTOBUF_LIBRARY} ${PROTOBUF_INCLUDE_DIR})
   endif ()
 
-  if (ORC_PREFER_STATIC_PROTOBUF AND ${PROTOC_STATIC_LIB})
+  if (ORC_PREFER_STATIC_PROTOBUF AND PROTOC_STATIC_LIB)
     add_resolved_library (orc_protoc ${PROTOC_STATIC_LIB} ${PROTOBUF_INCLUDE_DIR})
   else ()
     add_resolved_library (orc_protoc ${PROTOC_LIBRARY} ${PROTOBUF_INCLUDE_DIR})

@wgtmac Could you fix the ORC side problem (that is not related to this PR's failure)? We should not use ${...} in these if (...)s because it may be "not defined" or "an empty string". In these case, the if (...) becomes if (A AND). It's a syntax error. ORC has similar problem for other libraries such as LZ4: https://github.com/apache/orc/blob/37201cb8a186d2111a5ccf49b213003c4c947859/cmake_modules/ThirdpartyToolchain.cmake#L313

wgtmac · 2024-07-29T04:56:58Z

@kou Sorry for the issue caused by ORC. Let me take a look.

kou · 2024-07-29T06:45:10Z

Thanks!

raulcd · 2024-07-29T08:59:10Z

Hmm. It seems that https://github.com/apache/orc/blob/23044d79a26cf8792bcb1936ab601308b0781962/cmake_modules/ThirdpartyToolchain.cmake#L416-L418 wasn't used. It means that Protobuf_ROOT is an empty string in this case:

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Lines 4524 to 4529 in 187197c

get_target_property(PROTOBUF_INCLUDE_DIR ${ARROW_PROTOBUF_LIBPROTOBUF}

INTERFACE_INCLUDE_DIRECTORIES)

get_filename_component(Protobuf_ROOT "${PROTOBUF_INCLUDE_DIR}" DIRECTORY)

set(PROTOBUF_HOME

${Protobuf_ROOT}

CACHE STRING "" FORCE)

Could you debug print Protobuf_ROOT value?

-- Building Apache ORC from source
-- ## Protobuf_ROOT: /opt/vcpkg/installed/amd64-linux-static-release
-- Build type: RELEASE

@kou you lost me with this comment, are you talking about the ORC issue here or the flight issue with protobuf?

other logs about where protoc is:

-- Found Protobuf: /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc (found version "25.1.0")
-- Providing CMake module for FindProtobufAlt as part of Arrow CMake package
-- Found protoc: /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc

kou · 2024-07-29T09:44:15Z

you lost me with this comment, are you talking about the ORC issue here or the flight issue with protobuf?

The ORC issue here. I don't think that the cpp/cmake_modules/orc.diff is the right approach.
Sorry. I haven't looked at #43417 (comment) yet.

raulcd · 2024-07-29T09:50:54Z

I don't think that the cpp/cmake_modules/orc.diff is the right approach.

ah! ok, that makes sense. Thanks for your help!

kou · 2024-07-30T08:02:00Z

(I hope that I can take a look at the Flight issue tomorrow...)

raulcd · 2024-07-30T08:17:32Z

(I hope that I can take a look at the Flight issue tomorrow...)

Thanks @kou, no worries, I am testing the orc issue and it seems like the FindProtobuf file from ORC isn't used. I'll keep testing and will share findings.

raulcd · 2024-07-30T09:27:23Z

ok, vcpkg uses CONFIG for find_package and does not use FindProtobuf.cmake but ProtobufConfig.cmake or protobuf-config.cmake based on debug:

    /opt/vcpkg/installed/amd64-linux-static-release/share/protobuf/ProtobufConfig.cmake
    /opt/vcpkg/installed/amd64-linux-static-release/share/protobuf/protobuf-config.cmake

This does not populate PROTOBUF_STATIC_LIB nor PROTOC_STATIC_LIB, the following snippet for orc.diff fixes the issue:

diff --git a/cmake_modules/ThirdpartyToolchain.cmake b/cmake_modules/ThirdpartyToolchain.cmake
index aa520ff..7bd75d1 100644
--- a/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cmake_modules/ThirdpartyToolchain.cmake
@@ -416,6 +416,18 @@ endif ()
 if (NOT "${PROTOBUF_HOME}" STREQUAL "" OR ORC_PACKAGE_KIND STREQUAL "conan")
   find_package (Protobuf REQUIRED)
   set(PROTOBUF_VENDORED FALSE)
+  if (Protobuf_FOUND AND NOT PROTOBUF_STATIC_LIB)
+    get_target_property (target_type protobuf::libprotobuf TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set(PROTOBUF_STATIC_LIB protobuf::libprotobuf)
+    endif ()
+  endif()
+  if (Protobuf_FOUND AND NOT PROTOC_STATIC_LIB)
+    get_target_property (target_type protobuf::libprotoc TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set (PROTOC_STATIC_LIB protobuf::libprotoc)
+    endif ()
+  endif()
 else ()
   set(PROTOBUF_PREFIX "${THIRDPARTY_DIR}/protobuf_ep-install")
   set(PROTOBUF_INCLUDE_DIR "${PROTOBUF_PREFIX}/include")

raulcd · 2024-07-30T10:00:18Z

I've tried to debug the protoc issue but I've had no luck, adding it here in case it can help:

[root@17eac57d5aef flight]# gdb --args /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc -I/arrow/cpp/../format --grpc_out=/tmp/arrow-build/src/arrow/flight --plugin=protoc-gen-grpc=/opt/vcpkg/installed/amd64-linux-static-release/../x64-linux/tools/grpc/grpc_cpp_plugin /arrow/cpp/../format/Flight.proto
...
(gdb) set follow-fork-mode child
(gdb) run
Starting program: /opt/vcpkg/installed/x64-linux/tools/protobuf/protoc -I/arrow/cpp/../format --grpc_out=/tmp/arrow-build/src/arrow/flight --plugin=protoc-gen-grpc=/opt/vcpkg/installed/amd64-linux-static-release/../x64-linux/tools/grpc/grpc_cpp_plugin /arrow/cpp/../format/Flight.proto
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[Attaching after Thread 0x799b682acc00 (LWP 642) fork to child process 643]
[New inferior 2 (process 643)]
[Detaching after fork from parent process 642]
[Inferior 1 (process 642) detached]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
process 643 is executing new program: /opt/vcpkg/installed/x64-linux/tools/grpc/grpc_cpp_plugin

Thread 2.1 "grpc_cpp_plugin" received signal SIGSEGV, Segmentation fault.
[Switching to process 643]
0x00007e16ee5638a4 in dl_main (phdr=<optimized out>, phnum=<optimized out>, user_entry=<optimized out>, auxv=<optimized out>) at rtld.c:1832
1832	rtld.c: No such file or directory.

I tried setting the following to docker-compose.yml as suggested on stackoverflow on the Error disabling address space randomization: Operation not permitted to try and remove that warning without luck (I don't think is related, though):

diff --git a/docker-compose.yml b/docker-compose.yml
index cf22324..674fe0a 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1132,6 +1132,10 @@ services:
       - .:/arrow:delegated
       - ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux-2-28-ccache:/ccache:delegated
     command: /arrow/ci/scripts/python_wheel_manylinux_build.sh
+    cap_add:
+      - SYS_PTRACE
+    security_opt:
+      - apparmor:unconfined
 
   python-wheel-manylinux-test-imports:
     image: ${ARCH}/python:${PYTHON}

It seems that just executing grpc_cpp_plugin segfaults:

[root@849e8f917b46 flight]# /opt/vcpkg/installed/x64-linux/tools/grpc/grpc_cpp_plugin
Segmentation fault (core dumped)

edit: added some more info.

kou · 2024-07-31T02:23:26Z

This problem #43417 (comment) may be already solved in upstream: apache/orc#1963
(But not released yet.)

apacheGH-43416: [CI] Upgrade our vcpkg version on .env

c09aef7

github-actions bot added the awaiting committer review Awaiting committer review label Jul 25, 2024

raulcd mentioned this pull request Jul 25, 2024

[C++] Compiling with vcpkg dependencies is broken for latest VS 2022 (17.10) #43327

Open

This comment was marked as outdated.

Sign in to view

Fix patches

a60bc06

github-actions bot added the Component: C++ label Jul 26, 2024

raulcd force-pushed the GH-43416 branch from 982a6ce to 2abb87d Compare July 26, 2024 10:05

Apply orc patch in order to make CMake happy

ee89232

raulcd force-pushed the GH-43416 branch from 2abb87d to ee89232 Compare July 26, 2024 10:18

raulcd added 2 commits July 26, 2024 12:27

Add license to patch file

26af9f9

Make Windows happy with the patch

2cedbe6

Update patch

823a17e

New snippet to fix orc

daf593c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-43416: [CI] Upgrade our vcpkg version on .env #43417

GH-43416: [CI] Upgrade our vcpkg version on .env #43417

raulcd commented Jul 25, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Jul 25, 2024

raulcd commented Jul 25, 2024

github-actions bot commented Jul 25, 2024

raulcd commented Jul 25, 2024

This comment was marked as outdated.

raulcd commented Jul 25, 2024

github-actions bot commented Jul 25, 2024

raulcd commented Jul 25, 2024

github-actions bot commented Jul 25, 2024

raulcd commented Jul 25, 2024

raulcd commented Jul 26, 2024

raulcd commented Jul 26, 2024 •

edited

Loading

github-actions bot commented Jul 26, 2024

raulcd commented Jul 26, 2024

raulcd commented Jul 26, 2024

github-actions bot commented Jul 26, 2024

kou commented Jul 29, 2024

wgtmac commented Jul 29, 2024

kou commented Jul 29, 2024

raulcd commented Jul 29, 2024

kou commented Jul 29, 2024

raulcd commented Jul 29, 2024

kou commented Jul 30, 2024

raulcd commented Jul 30, 2024

raulcd commented Jul 30, 2024 •

edited

Loading

raulcd commented Jul 30, 2024 •

edited

Loading

kou commented Jul 31, 2024

GH-43416: [CI] Upgrade our vcpkg version on .env #43417

Are you sure you want to change the base?

GH-43416: [CI] Upgrade our vcpkg version on .env #43417

Conversation

raulcd commented Jul 25, 2024 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Jul 25, 2024

raulcd commented Jul 25, 2024

github-actions bot commented Jul 25, 2024

raulcd commented Jul 25, 2024

This comment was marked as outdated.

raulcd commented Jul 25, 2024

github-actions bot commented Jul 25, 2024

raulcd commented Jul 25, 2024

github-actions bot commented Jul 25, 2024

raulcd commented Jul 25, 2024

raulcd commented Jul 26, 2024

raulcd commented Jul 26, 2024 • edited Loading

github-actions bot commented Jul 26, 2024

raulcd commented Jul 26, 2024

raulcd commented Jul 26, 2024

github-actions bot commented Jul 26, 2024

kou commented Jul 29, 2024

wgtmac commented Jul 29, 2024

kou commented Jul 29, 2024

raulcd commented Jul 29, 2024

kou commented Jul 29, 2024

raulcd commented Jul 29, 2024

kou commented Jul 30, 2024

raulcd commented Jul 30, 2024

raulcd commented Jul 30, 2024 • edited Loading

raulcd commented Jul 30, 2024 • edited Loading

kou commented Jul 31, 2024

raulcd commented Jul 25, 2024 •

edited by github-actions bot

Loading

raulcd commented Jul 26, 2024 •

edited

Loading

raulcd commented Jul 30, 2024 •

edited

Loading

raulcd commented Jul 30, 2024 •

edited

Loading