Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI][C++][Homebrew] Failed to detect Protobuf by ORC #42149

Closed
kou opened this issue Jun 14, 2024 · 5 comments
Closed

[CI][C++][Homebrew] Failed to detect Protobuf by ORC #42149

kou opened this issue Jun 14, 2024 · 5 comments

Comments

@kou
Copy link
Member

kou commented Jun 14, 2024

Describe the bug, including details regarding any error messages, version, and platform.

ARM64 macOS 14 C++: https://github.com/apache/arrow/actions/runs/9508523884/job/26209983344#step:12:862

[25/1026] Performing configure step for 'orc_ep'
FAILED: orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure 
cd /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build && /opt/homebrew/Cellar/cmake/3.29.5/bin/cmake -P /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-DEBUG.cmake && /opt/homebrew/Cellar/cmake/3.29.5/bin/cmake -E touch /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure
CMake Error at /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-DEBUG.cmake:37 (message):
  Command failed: 1

   '/opt/homebrew/Cellar/cmake/3.29.5/bin/cmake' '-DCMAKE_C_COMPILER=/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc' '-DCMAKE_CXX_COMPILER=/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++' '-DCMAKE_AR=/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar' '-DCMAKE_RANLIB=/Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib' '-DBUILD_SHARED_LIBS=OFF' '-DBUILD_STATIC_LIBS=ON' '-DBUILD_TESTING=OFF' '-DCMAKE_BUILD_TYPE=DEBUG' '-DCMAKE_CXX_FLAGS= -Qunused-arguments -fcolor-diagnostics -fPIC' '-DCMAKE_CXX_FLAGS_DEBUG=-g -Werror -O0 -ggdb -g1  -Wno-error' '-DCMAKE_CXX_FLAGS_MISIZEREL=-Os -DNDEBUG ' '-DCMAKE_CXX_FLAGS_RELEASE=-O3 -DNDEBUG -O2  ' '-DCMAKE_CXX_FLAGS_RELWITHDEBINFO=-O2 -g -DNDEBUG -ggdb  ' '-DCMAKE_CXX_STANDARD=17' '-DCMAKE_C_FLAGS= -Qunused-arguments -fPIC' '-DCMAKE_C_FLAGS_DEBUG=-g -Werror -O0 -ggdb -g1  -Wno-error' '-DCMAKE_C_FLAGS_MISIZEREL=-Os -DNDEBUG ' '-DCMAKE_C_FLAGS_RELEASE=-O3 -DNDEBUG -O2  ' '-DCMAKE_C_FLAGS_RELWITHDEBINFO=-O2 -g -DNDEBUG -ggdb  ' '-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=' '-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=' '-DCMAKE_INSTALL_LIBDIR=lib' '-DCMAKE_OSX_SYSROOT=/Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk' '-DCMAKE_VERBOSE_MAKEFILE=OFF' '-DCMAKE_C_COMPILER_LAUNCHER=/opt/homebrew/bin/ccache' '-DCMAKE_CXX_COMPILER_LAUNCHER=/opt/homebrew/bin/ccache' '-DCMAKE_INSTALL_PREFIX=/Users/runner/work/arrow/arrow/build/cpp/orc_ep-install' '-DSTOP_BUILD_ON_WARNING=OFF' '-DBUILD_LIBHDFSPP=OFF' '-DBUILD_JAVA=OFF' '-DBUILD_TOOLS=OFF' '-DBUILD_CPP_TESTS=OFF' '-DINSTALL_VENDORED_LIBS=OFF' '-DLZ4_HOME=/opt/homebrew/Cellar/lz4/1.9.4' '-DPROTOBUF_EXECUTABLE=/opt/homebrew/bin/protoc-27.0.0' '-DPROTOBUF_HOME=/opt/homebrew' '-DPROTOBUF_INCLUDE_DIR=/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '/opt/homebrew/include' '-DPROTOBUF_LIBRARY=/opt/homebrew/lib/libprotobuf.27.0.0.dylib' '-DPROTOC_LIBRARY=/opt/homebrew/lib/libprotoc.27.0.0.dylib' '-DSNAPPY_HOME=/opt/homebrew' '-DSNAPPY_LIBRARY=/opt/homebrew/lib/libsnappy.1.2.1.dylib' '-DLZ4_LIBRARY=/opt/homebrew/Cellar/lz4/1.9.4/lib/liblz4.dylib' '-DLZ4_STATIC_LIB=/opt/homebrew/Cellar/lz4/1.9.4/lib/liblz4.dylib' '-DLZ4_INCLUDE_DIR=/opt/homebrew/Cellar/lz4/1.9.4/include' '-DSNAPPY_INCLUDE_DIR=/opt/homebrew/include' '-DZSTD_HOME=/opt/homebrew' '-DZSTD_INCLUDE_DIR=/opt/homebrew/include' '-DZSTD_LIBRARY=/opt/homebrew/lib/libzstd.1.5.6.dylib' '-GNinja' '-S' '/Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep' '-B' '/Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build'

  See also

    /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-*.log


-- stdout output is:
-- The C compiler identification is AppleClang 15.0.0.15000040
-- The CXX compiler identification is AppleClang 15.0.0.15000040
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode_15.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type: DEBUG
-- compiler AppleClang version 15.0.0.15000040
-- Disable the metrics collection
-- Performing Test ORC_CXX_HAS_CSTDINT
-- Performing Test ORC_CXX_HAS_CSTDINT - Success
-- Performing Test ORC_CXX_HAS_THREAD_LOCAL
-- Performing Test ORC_CXX_HAS_THREAD_LOCAL - Success
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- SNAPPY_HOME: /opt/homebrew
-- Found the Snappy header: /opt/homebrew/include/snappy.h
-- Found the Snappy library: /opt/homebrew/lib/libsnappy.1.2.1.dylib
-- ZSTD_HOME: /opt/homebrew
-- Found the zstd header: /opt/homebrew/include/zstd.h
-- Found the zstd library: /opt/homebrew/lib/libzstd.1.5.6.dylib
-- Found the zstd static library: /opt/homebrew/lib/libzstd.a
-- LZ4_HOME: /opt/homebrew/Cellar/lz4/1.9.4
-- Found the LZ4 header: /opt/homebrew/Cellar/lz4/1.9.4/include/lz4.h
-- Found the LZ4 library: /opt/homebrew/Cellar/lz4/1.9.4/lib/liblz4.dylib
-- Found the LZ4 static library: /opt/homebrew/Cellar/lz4/1.9.4/lib/liblz4.dylib
-- PROTOBUF_HOME: /opt/homebrew
-- ZLIB_HOME: 
-- Found the ZLIB header: /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/include/zlib.h
-- Found the ZLIB library: /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/lib/libz.a
-- Found the ZLIB static library: /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/lib/libz.a
-- Configuring incomplete, errors occurred!

-- stderr output is:
CMake Warning:
  Ignoring extra path from command line:

   "/opt/homebrew/include"


CMake Warning:
  Ignoring extra path from command line:

   "/opt/homebrew/include"


CMake Error at cmake_modules/FindProtobuf.cmake:115 (message):
  Could not find Protobuf.  Looked in /opt/homebrew.
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:417 (find_package)
  CMakeLists.txt:166 (INCLUDE)



CMake Error at /Users/runner/work/arrow/arrow/build/cpp/orc_ep-prefix/src/orc_ep-stamp/orc_ep-configure-DEBUG.cmake:47 (message):
  Stopping after outputting logs.

Component(s)

C++, Continuous Integration

@raulcd
Copy link
Member

raulcd commented Jun 19, 2024

I've seen this happening also on non ARM64 jobs, example on this PR for macOS 12:
Specifically this job: https://github.com/apache/arrow/actions/runs/9579913000/job/26413390696?pr=42207#step:8:891

@raulcd raulcd added this to the 17.0.0 milestone Jun 19, 2024
@raulcd raulcd added the Priority: Blocker Marks a blocker for the release label Jun 19, 2024
drin added a commit to drin/homebrew-hatchery that referenced this issue Jun 19, 2024
@drin
Copy link
Contributor

drin commented Jun 19, 2024

I have a customized version of the apache-arrow homebrew recipe that turns on substrait (apache-arrow-substrait)

I have been seeing this issue today and I couldn't figure out a workaround, so I'm going to try turning off ORC for myself (which the apache-arrow homebrew recipe currently has anyways).

Mostly commenting here since I arbitrarily linked to this issue (and that would probably seem fairly random without any context).

@kou
Copy link
Member Author

kou commented Jun 22, 2024

@wgtmac I looked at this problem and opened an issue for this: https://issues.apache.org/jira/browse/ORC-1732
Could you take a look at it?

@raulcd
Copy link
Member

raulcd commented Jun 26, 2024

It seems quite a lot of our verification jobs fail due to this issue :

CMake Error at cmake_modules/FindProtobuf.cmake:115 (message):
-- Found the ZLIB header: /var/folders/4z/th_0342d7sv9wl7kb3yfllq80000gn/T/arrow-HEAD.XXXXX.pGOTHd3O/cpp-build/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/include/zlib.h
  Could not find Protobuf.  Looked in /usr/local.
-- Found the ZLIB library: /var/folders/4z/th_0342d7sv9wl7kb3yfllq80000gn/T/arrow-HEAD.XXXXX.pGOTHd3O/cpp-build/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/lib/libz.a
Call Stack (most recent call first):
-- Found the ZLIB static library: /var/folders/4z/th_0342d7sv9wl7kb3yfllq80000gn/T/arrow-HEAD.XXXXX.pGOTHd3O/cpp-build/orc_ep-prefix/src/orc_ep-build/c++/libs/thirdparty/zlib_ep-install/lib/libz.a
  cmake_modules/ThirdpartyToolchain.cmake:417 (find_package)
-- Configuring incomplete, errors occurred!
  CMakeLists.txt:166 (INCLUDE)

verify-rc-source-cpp-macos-amd64
verify-rc-source-cpp-macos-arm64
verify-rc-source-integration-macos-amd64
verify-rc-source-integration-macos-arm64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-ruby-macos-amd64
verify-rc-source-ruby-macos-arm64

kou added a commit to kou/arrow that referenced this issue Jul 3, 2024
kou added a commit to kou/arrow that referenced this issue Jul 3, 2024
kou added a commit to kou/arrow that referenced this issue Jul 4, 2024
kou added a commit to kou/arrow that referenced this issue Jul 5, 2024
kou added a commit to kou/arrow that referenced this issue Jul 5, 2024
kou added a commit to kou/arrow that referenced this issue Jul 6, 2024
raulcd pushed a commit that referenced this issue Jul 8, 2024
### Rationale for this change

This also has a workaround for https://issues.apache.org/jira/browse/ORC-1732 .

### What changes are included in this PR?

ORC 2.0.1 has a dependency detection problem. We can't override the detection with ExternalProject but can override the detection with FetchContent.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: #42149

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
@raulcd
Copy link
Member

raulcd commented Jul 8, 2024

Issue resolved by pull request 43011
#43011

@raulcd raulcd closed this as completed Jul 8, 2024
raulcd pushed a commit that referenced this issue Jul 8, 2024
### Rationale for this change

This also has a workaround for https://issues.apache.org/jira/browse/ORC-1732 .

### What changes are included in this PR?

ORC 2.0.1 has a dependency detection problem. We can't override the detection with ExternalProject but can override the detection with FetchContent.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: #42149

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this issue Jul 9, 2024
### Rationale for this change

This also has a workaround for https://issues.apache.org/jira/browse/ORC-1732 .

### What changes are included in this PR?

ORC 2.0.1 has a dependency detection problem. We can't override the detection with ExternalProject but can override the detection with FetchContent.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#42149

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
dongjoon-hyun pushed a commit to apache/orc that referenced this issue Jul 11, 2024
### What changes were proposed in this pull request?
Detect Protobuf installed by Homebrew on macOS.

### Why are the changes needed?
Deal with the issue discussed [here](apache/arrow#42149)

### How was this patch tested?
Test it locally.

### Was this patch authored or co-authored using generative AI tooling?
NO

Closes #1963 from luffy-zh/ORC-1732.

Lead-authored-by: luffy-zh <zhnice@outlook.com>
Co-authored-by: Hao Zou <34559830+luffy-zh@users.noreply.github.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun pushed a commit to apache/orc that referenced this issue Jul 11, 2024
### What changes were proposed in this pull request?
Detect Protobuf installed by Homebrew on macOS.

### Why are the changes needed?
Deal with the issue discussed [here](apache/arrow#42149)

### How was this patch tested?
Test it locally.

### Was this patch authored or co-authored using generative AI tooling?
NO

Closes #1963 from luffy-zh/ORC-1732.

Lead-authored-by: luffy-zh <zhnice@outlook.com>
Co-authored-by: Hao Zou <34559830+luffy-zh@users.noreply.github.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit a9e0351)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants