Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] Specify minimum required zstd version in cmake #18746

Closed
asfimport opened this issue Jul 19, 2021 · 6 comments
Closed

[C++] Specify minimum required zstd version in cmake #18746

asfimport opened this issue Jul 19, 2021 · 6 comments
Assignees
Labels
Milestone

Comments

@asfimport
Copy link

This causes the build to fail as ZSTD_minCLevel() can't be found https://github.com/ursacomputing/crossbow/runs/3105438383


[ 54%] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/csv/converter.cc.o
cd /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/build/src/arrow && /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow/Library/Homebrew/shims/mac/super/clang++ -DARROW_EXPORTING -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_JEMALLOC -DARROW_JEMALLOC_INCLUDE_DIR="" -DARROW_MIMALLOC -DARROW_WITH_BACKTRACE -DARROW_WITH_LZ4 -DARROW_WITH_RE2 -DARROW_WITH_SNAPPY -DARROW_WITH_TIMING_TESTS -DARROW_WITH_UTF8PROC -DARROW_WITH_ZLIB -DARROW_WITH_ZSTD -DURI_STATIC_BUILD -DUTF8PROC_STATIC -I/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/build/src -I/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/cpp/src -I/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/cpp/src/generated -isystem /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/cpp/thirdparty/flatbuffers/include -isystem /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/build/jemalloc_ep-prefix/src -isystem /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/build/mimalloc_ep/src/mimalloc_ep/lib/mimalloc-1.6/include -isystem /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/build/rapidjson_ep/src/rapidjson_ep-install/include -isystem /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/build/xsimd_ep/src/xsimd_ep-install/include -isystem /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/build/re2_ep-install/include -isystem /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/build/utf8proc_ep-install/include -isystem /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/cpp/thirdparty/hadoop/include -Qunused-arguments -fcolor-diagnostics -O3 -DNDEBUG  -Wall -Wno-unknown-Note-option -Wno-pass-failed -stdlib=libc++ -msse4.2  -DNDEBUG -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -mmacosx-version-min=10.15 -fPIC -std=c++11 -o CMakeFiles/arrow_objlib.dir/csv/converter.cc.o -c /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/cpp/src/arrow/csv/converter.cc
/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp/apache-arrow-20210719-2645-6nb4zw/cpp/src/arrow/util/compression_zstd.cc:231:59: error: use of undeclared identifier 'ZSTD_minCLevel'; did you mean 'ZSTD_maxCLevel'?
  int minimum_compression_level() const override { return ZSTD_minCLevel(); }
                                                          ^~~~~~~~~~~~~~
                                                          ZSTD_maxCLevel
/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow/include/zstd.h:142:25: note: 'ZSTD_maxCLevel' declared here
ZSTDLIB_API int         ZSTD_maxCLevel(void);               /*!< maximum compression level available */
                        ^
1 error generated.
make[2]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/util/compression_zstd.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/all] Error 2
make: *** [all] Error 2==> Formula
Path: /Users/runner/work/crossbow/crossbow/arrow/r/check/arrow.Rcheck/00_pkg_src/arrow/tools/apache-arrow.rb
==> Configuration
HOMEBREW_VERSION: >=1.7.1 (shallow or no git repository)
ORIGIN: (none)
HEAD: (none)
Last commit: never
Core tap ORIGIN: https://github.com/autobrew/homebrew-core
Core tap HEAD: 73b60196102b25b6a91197210cad74ae7833b871
Core tap last commit: 8 weeks ago
HOMEBREW_PREFIX: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow
HOMEBREW_REPOSITORY: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow
HOMEBREW_CELLAR: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow/Cellar
HOMEBREW_CACHE: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/
HOMEBREW_TEMP: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/hbtmp
HOMEBREW_CASK_OPTS: --no-quarantine
HOMEBREW_CLEANUP_PERIODIC_FULL_DAYS: 3650
HOMEBREW_GIT: git
HOMEBREW_NO_ANALYTICS: 1
HOMEBREW_NO_AUTO_UPDATE: 1
CPU: 3-core 64-bit ivybridge
Homebrew Ruby: 2.3.7 => /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow/Library/Homebrew/vendor/portable-ruby/2.3.7/bin/ruby
Clang: 12.0 build 1200
Git: 2.24.3 => /Applications/Xcode_12.4.app/Contents/Developer/usr/bin/git
Curl: 7.64.1 => /usr/bin/curl
Java: 14.0.2, 13.0.2, 12.0.2, 11.0.11, 1.8.0_292
macOS: 10.15.7-x86_64
CLT: 12.4.0.0.1.1610135815
CLT headers: 12.4.0.0.1.1610135815
Xcode: 12.4 => /Applications/Xcode_12.4.app/Contents/Developer
Error: Error: apache-arrow HEAD-91a6ea4 did not build 

Reporter: David Li / @lidavidm
Assignee: Krisztian Szucs / @kszucs

PRs and other links:

Note: This issue was originally created as ARROW-13384. Please see the migration documentation for further details.

@asfimport
Copy link
Author

David Li / @lidavidm:
We're installing zstd 1.3.5:


* installing *source* package ‘arrow’ ...
** using staged installation
*** Downloading apache-arrow
**** Using local manifest for apache-arrow
Mon Jul 19 15:21:39 UTC 2021: Auto-brewing apache-arrow in /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T//build-apache-arrow...
==> Tapping autobrew/core from https://github.com/autobrew/homebrew-core
Tapped 2 commands and 4636 formulae (4,885 files, 12.7MB).
aws-sdk-cpp
lz4
snappy
openssl
thrift
zstd
==> Downloading https://autobrew.github.io/bottles/aws-sdk-cpp-1.7.364.high_sierra.bottle.2.tar.gz
==> Pouring aws-sdk-cpp-1.7.364.high_sierra.bottle.2.tar.gz
==> Skipping post_install step for autobrew...
🍺  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow/Cellar/aws-sdk-cpp/1.7.364: 967 files, 21.8MB
==> Downloading https://bintray-archive.github.io/bottles/lz4-1.8.3.high_sierra.bottle.tar.gz
==> Pouring lz4-1.8.3.high_sierra.bottle.tar.gz
==> Skipping post_install step for autobrew...
🍺  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow/Cellar/lz4/1.8.3: 22 files, 487.9KB
==> Downloading https://bintray-archive.github.io/bottles/snappy-1.1.7_1.high_sierra.bottle.tar.gz
==> Pouring snappy-1.1.7_1.high_sierra.bottle.tar.gz
==> Skipping post_install step for autobrew...
🍺  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow/Cellar/snappy/1.1.7_1: 18 files, 118.1KB
==> Downloading https://bintray-archive.github.io/bottles/openssl-1.0.2p.high_sierra.bottle.tar.gz
==> Pouring openssl-1.0.2p.high_sierra.bottle.tar.gz
==> Skipping post_install step for autobrew...
🍺  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow/Cellar/openssl/1.0.2p: 1,793 files, 12.3MB
==> Downloading https://bintray-archive.github.io/bottles/thrift-0.11.0.high_sierra.bottle.tar.gz
==> Pouring thrift-0.11.0.high_sierra.bottle.tar.gz
==> Skipping post_install step for autobrew...
🍺  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow/Cellar/thrift/0.11.0: 102 files, 7MB
==> Downloading https://bintray-archive.github.io/bottles/zstd-1.3.5.high_sierra.bottle.tar.gz
==> Pouring zstd-1.3.5.high_sierra.bottle.tar.gz
==> Skipping post_install step for autobrew... 

@asfimport
Copy link
Author

David Li / @lidavidm:
Likely it's because GHA uses High Sierra which is not supported by Homebrew anymore and so only an old version is available as a bottle

@asfimport
Copy link
Author

Neal Richardson / @nealrichardson:
IDK that this is GHA's fault actually, but in any case we should be able to get the zstd version updated for autobrew. I'll look into it.

More generally though, is there a minimum version we require? Should that be checked/enforced in cmake?

@asfimport
Copy link
Author

David Li / @lidavidm:
Since ARROW-13091 we require v1.3.6 for minCLevel https://github.com/facebook/zstd/blob/705a62b612151cff06f453bc3452b9e99088a574/CHANGELOG#L312

We should probably enforce that in CMake, yeah.

@asfimport
Copy link
Author

Neal Richardson / @nealrichardson:
Ok, submitted PR to bump the version in the autobrew bottles autobrew/homebrew-core#72 cc @jeroen

I'll make a PR here to add the cmake version check.

@asfimport
Copy link
Author

Kouhei Sutou / @kou:
Issue resolved by pull request 10747
#10747

@asfimport asfimport added this to the 5.0.0 milestone Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants