QPL can no longer be compiled

[rschu1ze]

#48833 broke the build of the QPL library:

...

[11659/11990] Linking CXX executable utils/check-marks/check-marks
FAILED: utils/check-marks/check-marks
: && /usr/bin/clang++ --target=x86_64-linux-gnu --sysroot=/home/nomad/ClickHouse/cmake/linux/../../contrib/sysroot/linux-x86_64/x86_64-linux-gnu/libc --gcc-toolchain=/home/nomad/ClickHouse/cmake/linux/../../contrib/sysroot/linux-x86_64 -std=c++20 -fdiagnostics-color=always -Xclang -fuse-ctor-homing -Wno-enum-constexpr-conversion -fsized-deallocation  -gdwarf-aranges -pipe -mssse3 -msse4.1 -msse4.2 -mpclmul -mpopcnt -mavx2 -mavx512f -mavx512bw -mavx512vl -fasynchronous-unwind-tables -ffile-prefix-map=/home/nomad/ClickHouse=. -falign-functions=32 -mbranches-within-32B-boundaries -fdiagnostics-absolute-paths -fstrict-vtable-pointers -Wall -Wextra -Wframe-larger-than=65536 -Weverything -Wpedantic -Wno-zero-length-array -Wno-c++98-compat-pedantic -Wno-c++98-compat -Wno-c++20-compat -Wno-sign-conversion -Wno-implicit-int-conversion -Wno-implicit-int-float-conversion -Wno-ctad-maybe-unsupported -Wno-disabled-macro-expansion -Wno-documentation-unknown-command -Wno-double-promotion -Wno-exit-time-destructors -Wno-float-equal -Wno-global-constructors -Wno-missing-prototypes -Wno-missing-variable-declarations -Wno-padded -Wno-switch-enum -Wno-undefined-func-template -Wno-unused-template -Wno-vla -Wno-weak-template-vtables -Wno-weak-vtables -Wno-thread-safety-negative -Wno-enum-constexpr-conversion -Wno-unsafe-buffer-usage -O2 -g -DNDEBUG -O3 -g -gdwarf-4  -fno-pie --gcc-toolchain=/home/nomad/ClickHouse/cmake/linux/../../contrib/sysroot/linux-x86_64 --ld-path=/home/nomad/ck-build/ld.lld -rdynamic -Wl,--gdb-index -Wl,--build-id=sha1 -no-pie -Wl,-no-pie    -Xlinker --no-undefined src/CMakeFiles/clickhouse_malloc.dir/Common/malloc.cpp.o utils/check-marks/CMakeFiles/check-marks.dir/main.cpp.o -o utils/check-marks/check-marks  src/libclickhouse_new_delete.a  src/libdbms.a  contrib/boost-cmake/lib_boost_program_options.a  contrib/llvm-project/llvm/lib/libLLVMExecutionEngine.a  contrib/llvm-project/llvm/lib/libLLVMOrcTargetProcess.a  contrib/llvm-project/llvm/lib/libLLVMOrcShared.a  contrib/llvm-project/llvm/lib/libLLVMRuntimeDyld.a  contrib/llvm-project/llvm/lib/libLLVMPasses.a  contrib/llvm-project/llvm/lib/libLLVMCoroutines.a  contrib/llvm-project/llvm/lib/libLLVMObjCARCOpts.a  contrib/llvm-project/llvm/lib/libLLVMipo.a  contrib/llvm-project/llvm/lib/libLLVMFrontendOpenMP.a  contrib/llvm-project/llvm/lib/libLLVMIRReader.a  contrib/llvm-project/llvm/lib/libLLVMAsmParser.a  contrib/llvm-project/llvm/lib/libLLVMLinker.a  contrib/llvm-project/llvm/lib/libLLVMVectorize.a  contrib/llvm-project/llvm/lib/libLLVMX86CodeGen.a  contrib/llvm-project/llvm/lib/libLLVMAsmPrinter.a  contrib/llvm-project/llvm/lib/libLLVMGlobalISel.a  contrib/llvm-project/llvm/lib/libLLVMSelectionDAG.a  contrib/llvm-project/llvm/lib/libLLVMCodeGen.a  contrib/llvm-project/llvm/lib/libLLVMBitWriter.a  contrib/llvm-project/llvm/lib/libLLVMScalarOpts.a  contrib/llvm-project/llvm/lib/libLLVMAggressiveInstCombine.a  contrib/llvm-project/llvm/lib/libLLVMInstCombine.a  contrib/llvm-project/llvm/lib/libLLVMInstrumentation.a  contrib/llvm-project/llvm/lib/libLLVMTransformUtils.a  contrib/llvm-project/llvm/lib/libLLVMTarget.a  contrib/llvm-project/llvm/lib/libLLVMAnalysis.a  contrib/llvm-project/llvm/lib/libLLVMProfileData.a  contrib/llvm-project/llvm/lib/libLLVMSymbolize.a  contrib/llvm-project/llvm/lib/libLLVMDebugInfoDWARF.a  contrib/llvm-project/llvm/lib/libLLVMDebugInfoPDB.a  contrib/llvm-project/llvm/lib/libLLVMObject.a  contrib/llvm-project/llvm/lib/libLLVMBitReader.a  contrib/llvm-project/llvm/lib/libLLVMMCParser.a  contrib/llvm-project/llvm/lib/libLLVMTextAPI.a  contrib/llvm-project/llvm/lib/libLLVMDebugInfoMSF.a  contrib/llvm-project/llvm/lib/libLLVMX86Desc.a  contrib/llvm-project/llvm/lib/libLLVMMCDisassembler.a  contrib/llvm-project/llvm/lib/libLLVMX86Info.a  contrib/llvm-project/llvm/lib/libLLVMMC.a  contrib/llvm-project/llvm/lib/libLLVMDebugInfoCodeView.a  contrib/llvm-project/llvm/lib/libLLVMCFGuard.a  contrib/llvm-project/llvm/lib/libLLVMCore.a  contrib/llvm-project/llvm/lib/libLLVMRemarks.a  contrib/llvm-project/llvm/lib/libLLVMBitstreamReader.a  contrib/llvm-project/llvm/lib/libLLVMBinaryFormat.a  contrib/llvm-project/llvm/lib/libLLVMSupport.a  contrib/llvm-project/llvm/lib/libLLVMDemangle.a  contrib/croaring-cmake/lib_roaring.a  contrib/cppkafka-cmake/lib_cppkafka.a  contrib/librdkafka-cmake/lib_rdkafka.a  contrib/nats-io-cmake/lib_nats_io.a  contrib/cyrus-sasl-cmake/lib_sasl2.a  contrib/nuraft-cmake/lib_nuraft.a  contrib/boost-cmake/lib_boost_coroutine.a  src/Common/Config/libclickhouse_common_config.a  contrib/yaml-cpp-cmake/lib_yaml_cpp.a  src/Common/ZooKeeper/libclickhouse_common_zookeeper.a  src/Dictionaries/Embedded/libclickhouse_dictionaries_embedded.a  src/Parsers/libclickhouse_parsers.a  src/Access/Common/libclickhouse_common_access.a  base/poco/MongoDB/lib_poco_mongodb.a  src/Common/mysqlxx/libmysqlxx.a  src/libclickhouse_common_io.a  contrib/boost-cmake/lib_boost_program_options.a  contrib/jemalloc-cmake/lib_jemalloc.a  contrib/gwpasan-cmake/lib_gwp_asan.a  src/Common/StringUtils/libstring_utils.a  base/widechar_width/libwidechar_width.a  base/base/libcommon.a  contrib/boost-cmake/lib_boost_system.a  base/poco/NetSSL_OpenSSL/lib_poco_net_ssl.a  base/poco/Net/lib_poco_net.a  base/poco/Crypto/lib_poco_crypto.a  base/poco/Util/lib_poco_util.a  base/poco/JSON/lib_poco_json.a  base/poco/JSON/lib_poco_json_pdjson.a  base/poco/XML/lib_poco_xml.a  base/poco/XML/lib_poco_xml_expat.a  contrib/replxx-cmake/lib_replxx.a  contrib/cctz-cmake/lib_cctz.a  -Wl,--whole-archive /home/nomad/ck-build/contrib/cctz-cmake/libtzdata.a -Wl,--no-whole-archive  contrib/fmtlib-cmake/lib_fmt.a  contrib/dragonbox-cmake/lib_dragonbox_to_chars.a  contrib/cityhash102/lib_cityhash.a  contrib/re2-cmake/libre2_st.a  base/poco/Foundation/lib_poco_foundation.a  base/poco/Foundation/lib_poco_foundation_pcre.a  contrib/libcpuid-cmake/lib_cpuid.a  contrib/aws-cmake/lib_aws.a  contrib/liburing-cmake/lib_liburing.a  contrib/minizip-ng-cmake/lib_minizip.a  contrib/xz-cmake/lib_liblzma.a  contrib/bzip2-cmake/lib_bzip2.a  contrib/mariadb-connector-c-cmake/lib_mariadbclient.a  contrib/icu-cmake/lib_icui18n.a  contrib/icu-cmake/lib_icuuc.a  contrib/icu-cmake/lib_icudata.a  contrib/capnproto-cmake/lib_capnpc.a  contrib/capnproto-cmake/lib_capnp.a  contrib/capnproto-cmake/lib_kj.a  contrib/arrow-cmake/lib_parquet.a  contrib/arrow-cmake/lib_arrow.a  contrib/boost-cmake/lib_boost_filesystem.a  contrib/double-conversion-cmake/lib_double-conversion.a  contrib/brotli-cmake/lib_brotli.a  contrib/flatbuffers/libflatbuffers.a  contrib/arrow-cmake/lib_orc.a  contrib/avro-cmake/lib_avrocpp.a  contrib/boost-cmake/lib_boost_iostreams.a  contrib/openldap-cmake/lib_ldap_r.a  contrib/openldap-cmake/lib_lber.a  src/Server/grpc_protos/libclickhouse_grpc_protos.a  contrib/grpc/libgrpc++.a  contrib/grpc/libgrpc.a  contrib/re2-cmake/libre2.a  contrib/c-ares-cmake/lib_c-ares.a  contrib/abseil-cpp/absl/status/libabsl_status.a  contrib/grpc/libaddress_sorting.a  contrib/grpc/libupb.a  contrib/grpc/libgpr.a  contrib/libhdfs3-cmake/lib_hdfs3.a  contrib/protobuf-cmake/lib_libprotobuf.a  contrib/libgsasl-cmake/lib_gsasl.a  contrib/krb5-cmake/lib_krb5.a  contrib/isa-l-cmake/lib_isal.a  contrib/hive-metastore-cmake/lib_hivemetastore.a  contrib/thrift-cmake/lib_thrift.a  contrib/azure-cmake/lib_azure_sdk.a  contrib/libxml2-cmake/lib_libxml2.a  contrib/curl-cmake/lib_curl.a  contrib/s2geometry-cmake/lib_s2.a  contrib/abseil-cpp/absl/container/libabsl_raw_hash_set.a  contrib/abseil-cpp/absl/container/libabsl_hashtablez_sampler.a  contrib/abseil-cpp/absl/strings/libabsl_cord.a  contrib/abseil-cpp/absl/strings/libabsl_cordz_info.a  contrib/abseil-cpp/absl/strings/libabsl_cord_internal.a  contrib/abseil-cpp/absl/strings/libabsl_cordz_functions.a  contrib/abseil-cpp/absl/profiling/libabsl_exponential_biased.a  contrib/abseil-cpp/absl/strings/libabsl_cordz_handle.a  contrib/abseil-cpp/absl/synchronization/libabsl_synchronization.a  contrib/abseil-cpp/absl/debugging/libabsl_stacktrace.a  contrib/abseil-cpp/absl/debugging/libabsl_symbolize.a  contrib/abseil-cpp/absl/debugging/libabsl_debugging_internal.a  contrib/abseil-cpp/absl/debugging/libabsl_demangle_internal.a  contrib/abseil-cpp/absl/synchronization/libabsl_graphcycles_internal.a  contrib/abseil-cpp/absl/time/libabsl_time.a  contrib/abseil-cpp/absl/time/libabsl_civil_time.a  contrib/abseil-cpp/absl/time/libabsl_time_zone.a  contrib/abseil-cpp/absl/base/libabsl_malloc_internal.a  contrib/abseil-cpp/absl/hash/libabsl_hash.a  contrib/abseil-cpp/absl/hash/libabsl_city.a  contrib/abseil-cpp/absl/types/libabsl_bad_optional_access.a  contrib/abseil-cpp/absl/types/libabsl_bad_variant_access.a  contrib/abseil-cpp/absl/hash/libabsl_low_level_hash.a  contrib/abseil-cpp/absl/strings/libabsl_str_format_internal.a  contrib/abseil-cpp/absl/strings/libabsl_strings.a  contrib/abseil-cpp/absl/numeric/libabsl_int128.a  contrib/abseil-cpp/absl/base/libabsl_throw_delegate.a  contrib/abseil-cpp/absl/strings/libabsl_strings_internal.a  contrib/abseil-cpp/absl/base/libabsl_base.a  contrib/abseil-cpp/absl/base/libabsl_spinlock_wait.a  -lrt  contrib/abseil-cpp/absl/base/libabsl_raw_logging_internal.a  contrib/abseil-cpp/absl/base/libabsl_log_severity.a  contrib/amqpcpp-cmake/lib_amqp-cpp.a  contrib/libuv-cmake/lib_uv.a  -lrt  contrib/sqlite-cmake/lib_sqlite.a  contrib/rocksdb-cmake/lib_rocksdb.a  contrib/lz4-cmake/lib_lz4.a  contrib/zstd-cmake/lib_zstd.a  contrib/zlib-ng-cmake/lib_zlib.a  contrib/snappy-cmake/lib_snappy.a  contrib/libpqxx-cmake/lib_libpqxx.a  contrib/libpq-cmake/lib_libpq.a  contrib/boringssl-cmake/lib_ssl.a  contrib/boringssl-cmake/lib_crypto.a  -lpthread  contrib/qpl-cmake/lib_qpl.a  contrib/qpl-cmake/libaccel-config.a  -ldl  contrib/boost-cmake/lib_boost_context.a  contrib/libstemmer-c-cmake/lib_stemmer.a  contrib/wordnet-blast-cmake/lib_wnb.a  contrib/boost-cmake/lib_boost_graph.a  contrib/boost-cmake/lib_boost_regex.a  contrib/lemmagen-c-cmake/lib_lemmagen.a  -Wl,--whole-archive /home/nomad/ck-build/contrib/nlp-data-cmake/libnlp_dictionaries.a -Wl,--no-whole-archive  contrib/ulid-c-cmake/lib_ulid.a  contrib/simdjson-cmake/lib_simdjson.a  contrib/consistent-hashing/lib_consistent_hashing.a  rust/skim/RelWithDebInfo/lib_ch_rust_skim_rust.a  rust/skim/lib_ch_rust_skim_ffi.a  base/glibc-compatibility/libglibc-compatibility.a  base/glibc-compatibility/memcpy/libmemcpy.a  base/glibc-compatibility/libglibc-compatibility.a  base/glibc-compatibility/memcpy/libmemcpy.a  -Wl,--start-group  contrib/libcxx-cmake/libcxx.a  contrib/libcxxabi-cmake/libcxxabi.a  contrib/libunwind-cmake/libunwind.a  -Wl,--end-group  -nodefaultlibs /usr/lib64/clang/16/lib/linux/libclang_rt.builtins-x86_64.a  -lc -lm -lrt -lpthread -ldl && :
ld.lld: error: duplicate symbol: isal_update_histogram_base
>>> defined at huff_codes.c:661 (./contrib/isa-l/igzip/huff_codes.c:661)
>>>            huff_codes.c.o:(isal_update_histogram_base) in archive contrib/isa-l-cmake/lib_isal.a
>>> defined at huff_codes.c:654 (./contrib/qpl/sources/isal/igzip/huff_codes.c:654)
>>>            huff_codes.c.o:(.text+0x0) in archive contrib/qpl-cmake/lib_qpl.a

ld.lld: error: duplicate symbol: isal_create_hufftables
>>> defined at huff_codes.c:1377 (./contrib/isa-l/igzip/huff_codes.c:1377)
>>>            huff_codes.c.o:(isal_create_hufftables) in archive contrib/isa-l-cmake/lib_isal.a
>>> defined at huff_codes.c:1371 (./contrib/qpl/sources/isal/igzip/huff_codes.c:1371)
>>>            huff_codes.c.o:(.text+0x3E0) in archive contrib/qpl-cmake/lib_qpl.a

ld.lld: error: duplicate symbol: isal_create_hufftables_subset
>>> defined at huff_codes.c:1467 (./contrib/isa-l/igzip/huff_codes.c:1467)
>>>            huff_codes.c.o:(isal_create_hufftables_subset) in archive contrib/isa-l-cmake/lib_isal.a
>>> defined at huff_codes.c:1474 (./contrib/qpl/sources/isal/igzip/huff_codes.c:1474)
>>>            huff_codes.c.o:(.text+0x31C0) in archive contrib/qpl-cmake/lib_qpl.a

ld.lld: error: duplicate symbol: create_hufftables_icf
>>> defined at huff_codes.c:1591 (./contrib/isa-l/igzip/huff_codes.c:1591)
>>>            huff_codes.c.o:(create_hufftables_icf) in archive contrib/isa-l-cmake/lib_isal.a
>>> defined at huff_codes.c:1624 (./contrib/qpl/sources/isal/igzip/huff_codes.c:1624)
>>>            huff_codes.c.o:(.text+0x4F60) in archive contrib/qpl-cmake/lib_qpl.a
clang-16.0: error: linker command failed with exit code 1 (use -v to see invocation)

Without QPL, the build succeeds:

cmake -D ENABLE_QPL=ON -D ENABLE_AVX2=ON -D ENABLE_AVX512=ON -D PARALLEL_COMPILE_JOBS=128 -D ENABLE_TESTS=OFF -D PARALLEL_LINK_JOBS=64 ../ClickHouse/ 

The cause of the problem is that QPL comes with its own dump of isa-l (it would be interesting to understand why - just convenience or does it deviate in some ways from upstream isa-l?). ClickHouse now unconditionally links upstream isa-l as per #48833 which leads to duplicate symbols when the QPL codec is enabled.

There is precedence for that (e.g. #43259: cJSON linked by two unrelated submodules), and besides hacks (configuring the library to not export symbols, using PRIVATE linkage in CMake, ...) the only real fix is to build/link isa-l only once. So the best direction IMHO is to make upstream QPL not include a copy of isa-l but have isa-l as a prerequisite. "qpl-cmake/CMakeLists.txt" could then simply do target_link_libraries(_qpl ch_contrib::isal) (see contrib/isa-l-cmake/CMakeLists.txt) to satisfy the isa-l requirements. Note that the issue is not specific to ClickHouse, every program linking QPL and isa-l will run into this problem.

[rschu1ze]

@jinjunzh
@Tiaonmmn

[alexey-milovidov]

By the way, we don't support QPL at all - it is not used for production builds, and it is not built in CI.
It means that the feature can be dropped at any moment in time.

[alexey-milovidov]

@jinjunzh @Tiaonmmn, please send a pull request to support this feature.