Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] Configure bundled AWS SDK to use aws-lc instead of OpenSSL #34157

Closed
js8544 opened this issue Feb 13, 2023 · 2 comments · Fixed by #34159
Closed

[C++] Configure bundled AWS SDK to use aws-lc instead of OpenSSL #34157

js8544 opened this issue Feb 13, 2023 · 2 comments · Fixed by #34159

Comments

@js8544
Copy link
Collaborator

js8544 commented Feb 13, 2023

Describe the enhancement requested

Using OpenSSL causes various issues like #33808 (comment) and #34111. We should try to use aws-lc for libcrypto and libssl. We need to hide them inside s2n-tls to avoid name conflicts with OpenSSL used by other libraries.

Component(s)

C++

@raulcd
Copy link
Member

raulcd commented Feb 14, 2023

Hi, are the failures on some of our nightly jobs like wheel-manylinux2014-cp37-amd64 or wheel-manylinux2014-cp38-arm64 related to this or should we open a different ticket to track those failures?

FAILED: aws_c_cal_ep-prefix/src/aws_c_cal_ep-stamp/aws_c_cal_ep-configure /tmp/arrow-build/aws_c_cal_ep-prefix/src/aws_c_cal_ep-stamp/aws_c_cal_ep-configure 
cd /tmp/arrow-build/aws_c_cal_ep-prefix/src/aws_c_cal_ep-build && /usr/local/bin/cmake -P /tmp/arrow-build/aws_c_cal_ep-prefix/src/aws_c_cal_ep-stamp/aws_c_cal_ep-configure-RELEASE.cmake && /usr/local/bin/cmake -E touch /tmp/arrow-build/aws_c_cal_ep-prefix/src/aws_c_cal_ep-stamp/aws_c_cal_ep-configure
CMake Error at /tmp/arrow-build/aws_c_cal_ep-prefix/src/aws_c_cal_ep-stamp/aws_c_cal_ep-configure-RELEASE.cmake:37 (message):
  Command failed: 1

   '/usr/local/bin/cmake' '-DCMAKE_C_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/cc' '-DCMAKE_CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++' '-DCMAKE_AR=/opt/rh/devtoolset-10/root/usr/bin/ar' '-DCMAKE_RANLIB=/opt/rh/devtoolset-10/root/usr/bin/ranlib' '-DBUILD_SHARED_LIBS=OFF' '-DBUILD_STATIC_LIBS=ON' '-DBUILD_TESTING=OFF' '-DCMAKE_BUILD_TYPE=RELEASE' '-DCMAKE_CXX_FLAGS= -fdiagnostics-color=always -fPIC' '-DCMAKE_CXX_FLAGS_DEBUG=-g -O0 -ggdb -Wno-error' '-DCMAKE_CXX_FLAGS_MISIZEREL=-Os -DNDEBUG' '-DCMAKE_CXX_FLAGS_RELEASE=-O3 -DNDEBUG -O2 -ftree-vectorize' '-DCMAKE_CXX_FLAGS_RELWITHDEBINFO=-O2 -g -DNDEBUG -O2 -ftree-vectorize  -O0 -ggdb' '-DCMAKE_CXX_STANDARD=17' '-DCMAKE_C_FLAGS= -fPIC' '-DCMAKE_C_FLAGS_DEBUG=-g -O0 -ggdb -Wno-error' '-DCMAKE_C_FLAGS_MISIZEREL=-Os -DNDEBUG' '-DCMAKE_C_FLAGS_RELEASE=-O3 -DNDEBUG -O2 -ftree-vectorize' '-DCMAKE_C_FLAGS_RELWITHDEBINFO=-O2 -g -DNDEBUG -O2 -ftree-vectorize  -O0 -ggdb' '-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=' '-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=' '-DCMAKE_INSTALL_LIBDIR=lib' '-DCMAKE_VERBOSE_MAKEFILE=FALSE' '-DCMAKE_C_COMPILER_LAUNCHER=/usr/local/bin/ccache' '-DCMAKE_CXX_COMPILER_LAUNCHER=/usr/local/bin/ccache' '-DCMAKE_BUILD_TYPE=release' '-DENABLE_TESTING=OFF' '-DENABLE_UNITY_BUILD=ON' '-DCMAKE_INSTALL_PREFIX=/tmp/arrow-build/awssdk_ep-install' '-DCMAKE_PREFIX_PATH=/tmp/arrow-build/awssdk_ep-install' '-GNinja' '/tmp/arrow-build/aws_c_cal_ep-prefix/src/aws_c_cal_ep'

  See also

    /tmp/arrow-build/aws_c_cal_ep-prefix/src/aws_c_cal_ep-stamp/aws_c_cal_ep-configure-*.log


-- stdout output is:
-- The C compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rh/devtoolset-10/root/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Configuring incomplete, errors occurred!
See also "/tmp/arrow-build/aws_c_cal_ep-prefix/src/aws_c_cal_ep-build/CMakeFiles/CMakeOutput.log".

-- stderr output is:
CMake Error at /usr/local/share/cmake-3.21/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find crypto (missing: crypto_LIBRARY crypto_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/local/share/cmake-3.21/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  cmake/modules/Findcrypto.cmake:88 (find_package_handle_standard_args)
  CMakeLists.txt:106 (find_package)



CMake Error at /tmp/arrow-build/aws_c_cal_ep-prefix/src/aws_c_cal_ep-stamp/aws_c_cal_ep-configure-RELEASE.cmake:47 (message):
  Stopping after outputting logs.

@js8544
Copy link
Collaborator Author

js8544 commented Feb 14, 2023

@raulcd It's the same issue.

kou added a commit that referenced this issue Feb 15, 2023
…enSSL (#34159)

Using OpenSSL causes various issues like #33808 (comment) and #34111. We should try to use aws-lc for libcrypto and libssl. We need to hide them inside s2n-tls to avoid name conflicts with OpenSSL used by other libraries.

Some notes:
1. Only linux needs s2n-tls and aws-lc.
2. Because aws-c-http requires curl which requires libcrypto from OpenSSL, we need to hide Aws::libcrypto by S2N_INTERN_LIBCRYPTO=ON. 
3. aws-c-cal also uses libcrypto, but it can't hide it. So we need to use OpenSSL instead of aws-lc for it.
4. Mac and windows don't need s2n-tls, but do need some security libs from system, like `-framework Security` and `ncrypt.lib`. 
5. arrow-s3fs-test is re-enabled.
6. Force Java Gandiva to use static protobuf lib.

* Closes: #34157

Lead-authored-by: Jin Shang <shangjin1997@gmail.com>
Co-authored-by: jinshang <jinshang@tencent.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@kou kou added this to the 12.0.0 milestone Feb 15, 2023
gringasalpastor pushed a commit to gringasalpastor/arrow that referenced this issue Feb 17, 2023
… of OpenSSL (apache#34159)

Using OpenSSL causes various issues like apache#33808 (comment) and apache#34111. We should try to use aws-lc for libcrypto and libssl. We need to hide them inside s2n-tls to avoid name conflicts with OpenSSL used by other libraries.

Some notes:
1. Only linux needs s2n-tls and aws-lc.
2. Because aws-c-http requires curl which requires libcrypto from OpenSSL, we need to hide Aws::libcrypto by S2N_INTERN_LIBCRYPTO=ON. 
3. aws-c-cal also uses libcrypto, but it can't hide it. So we need to use OpenSSL instead of aws-lc for it.
4. Mac and windows don't need s2n-tls, but do need some security libs from system, like `-framework Security` and `ncrypt.lib`. 
5. arrow-s3fs-test is re-enabled.
6. Force Java Gandiva to use static protobuf lib.

* Closes: apache#34157

Lead-authored-by: Jin Shang <shangjin1997@gmail.com>
Co-authored-by: jinshang <jinshang@tencent.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
fatemehp pushed a commit to fatemehp/arrow that referenced this issue Feb 24, 2023
… of OpenSSL (apache#34159)

Using OpenSSL causes various issues like apache#33808 (comment) and apache#34111. We should try to use aws-lc for libcrypto and libssl. We need to hide them inside s2n-tls to avoid name conflicts with OpenSSL used by other libraries.

Some notes:
1. Only linux needs s2n-tls and aws-lc.
2. Because aws-c-http requires curl which requires libcrypto from OpenSSL, we need to hide Aws::libcrypto by S2N_INTERN_LIBCRYPTO=ON. 
3. aws-c-cal also uses libcrypto, but it can't hide it. So we need to use OpenSSL instead of aws-lc for it.
4. Mac and windows don't need s2n-tls, but do need some security libs from system, like `-framework Security` and `ncrypt.lib`. 
5. arrow-s3fs-test is re-enabled.
6. Force Java Gandiva to use static protobuf lib.

* Closes: apache#34157

Lead-authored-by: Jin Shang <shangjin1997@gmail.com>
Co-authored-by: jinshang <jinshang@tencent.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment