Aws::Utils::HashingUtils::CalculateMD5() returns empty ByteBuffer on SLES15SP7 ARM64 (aarch64) with OpenSSL 3.2.3

[hqho]

Describe the bug

On SLES15SP7 aarch64 (ARM64) with system OpenSSL 3.2.3 and libcurl 8.14.1, calling Aws::Utils::HashingUtils::CalculateMD5() returns an empty Aws::Utils::ByteBuffer (length = 0). This causes S3 multi-delete requests (which require Content-MD5) to be sent with an empty MD5 header and fail with InvalidDigest: The Content-MD5 you specified was invalid. The same SDK + OpenSSL configuration works on x86_64 SP7; the bug appears specific to ARM64 SP7.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

=== Platform Information ===
Architecture: x86_64
OpenSSL version: OpenSSL 3.2.3 3 Sep 2024

=== MD5 Calculation Test ===
Test payload: test-object-1test-object-2
Payload size: 137 bytes

Calling Aws::Utils::HashingUtils::CalculateMD5()...
MD5 ByteBuffer length: 16 (expected: 16)
SUCCESS: MD5 calculation returned correct length
Base64 MD5: kaK1wnr5JyA5xE9bdzBuiw==
Base64 length: 24 (expected: 24)

Current Behavior

=== Platform Information ===
Architecture: ARM64 (aarch64)
OpenSSL version: OpenSSL 3.2.3 3 Sep 2024

=== MD5 Calculation Test ===
Test payload: test-object-1test-object-2
Payload size: 137 bytes

Calling Aws::Utils::HashingUtils::CalculateMD5()...
MD5 ByteBuffer length: 0 (expected: 16)
ERROR: CalculateMD5() returned EMPTY ByteBuffer!

Reproduction Steps

#include <iostream>
#include <string>
#include <aws/core/Aws.h>
#include <aws/core/utils/HashingUtils.h>
#include <aws/core/utils/memory/stl/AWSString.h>
#include <openssl/opensslv.h>

int main(int argc, char* argv[]) {
    // Initialize AWS SDK
    Aws::SDKOptions options;
    Aws::InitAPI(options);

    // Print platform and OpenSSL info
    std::cout << "=== Platform Information ===" << std::endl;
#ifdef __aarch64__
    std::cout << "Architecture: ARM64 (aarch64)" << std::endl;
#elif defined(__x86_64__)
    std::cout << "Architecture: x86_64" << std::endl;
#else
    std::cout << "Architecture: Unknown" << std::endl;
#endif

    std::cout << "OpenSSL version: " << OPENSSL_VERSION_TEXT << std::endl;
    std::cout << std::endl;

    // Test payload (similar to DeleteObjects XML)
    std::string test_payload = 
        "<?xml version=\"1.0\" encoding=\"UTF-8\"?>"
        "<Delete>"
        "<Object><Key>test-object-1</Key></Object>"
        "<Object><Key>test-object-2</Key></Object>"
        "</Delete>";

    std::cout << "=== MD5 Calculation Test ===" << std::endl;
    std::cout << "Test payload: " << test_payload << std::endl;
    std::cout << "Payload size: " << test_payload.length() << " bytes" << std::endl;
    std::cout << std::endl;

    // Calculate MD5 using AWS SDK
    std::cout << "Calling Aws::Utils::HashingUtils::CalculateMD5()..." << std::endl;
    auto md5_bytebuffer = Aws::Utils::HashingUtils::CalculateMD5(test_payload);
    
    std::cout << "MD5 ByteBuffer length: " << md5_bytebuffer.GetLength() 
              << " (expected: 16)" << std::endl;

    if (md5_bytebuffer.GetLength() == 0) {
        std::cout << "ERROR: CalculateMD5() returned EMPTY ByteBuffer!" << std::endl;
    } else if (md5_bytebuffer.GetLength() == 16) {
        std::cout << "✓ SUCCESS: MD5 calculation returned correct length" << std::endl;
        
        // Calculate Base64 encoding
        auto base64_md5 = Aws::Utils::HashingUtils::Base64Encode(md5_bytebuffer);
        std::cout << "Base64 MD5: " << base64_md5.c_str() << std::endl;
        std::cout << "Base64 length: " << base64_md5.length() << " (expected: 24)" << std::endl;
    } else {
        std::cout << "ERROR: Unexpected MD5 ByteBuffer length!" << std::endl;
    }

    std::cout << std::endl;

    // Cleanup AWS SDK
    Aws::ShutdownAPI(options);

    return (md5_bytebuffer.GetLength() == 16) ? 0 : 1;
}

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### AWS CPP SDK version used

1.11.630

### Compiler and Version used

gcc (SUSE Linux) 14.3.0

### Operating System and version

SLES15SP7

[DmitriyMusatkin]

any chance its running against fips openssl?
what does OpenSSL_version say?

[hqho]

openssl version
OpenSSL 3.2.3 3 Sep 2024 (Library: OpenSSL 3.2.3 3 Sep 2024)

[sbiscigl]

Hey @hqho thanks for reaching out, but i cannot replicate your issue in a standalone environment.

project structure:

~/sdk-usage-workspace ❯❯❯ tree -L 1                                                                                                                                                             
├── CMakeLists.txt
├── Dockerfile
├── main.cpp
└── replicate.sh

CMakeLists.txt

cmake_minimum_required(VERSION 3.13)
project(sdk_usage_workspace)
set(CMAKE_CXX_STANDARD 20)

find_package(AWSSDK REQUIRED COMPONENTS core)
find_package(OpenSSL REQUIRED)

add_executable(${PROJECT_NAME} main.cpp)
target_link_libraries(${PROJECT_NAME} PRIVATE ${AWSSDK_LINK_LIBRARIES} OpenSSL::SSL OpenSSL::Crypto)

Dockerfile

FROM registry.suse.com/bci/bci-base:15.7

RUN zypper install -yt pattern devel_basis
RUN zypper install -y gcc-c++ cmake git libcurl-devel libopenssl-devel

RUN git clone --depth 1 --recurse-submodules https://github.com/aws/aws-sdk-cpp &&\
    mkdir aws-sdk-cpp/build &&\
    pushd aws-sdk-cpp/build &&\
    cmake -DBUILD_ONLY="core" -DAUTORUN_UNIT_TESTS=OFF -DCMAKE_INSTALL_REFIX=/sdk-install .. &&\
    cmake --build . &&\
    cmake --install . &&\
    popd

# Copy over and build
RUN mkdir sdk-example
COPY CMakeLists.txt /sdk-example/CMakeLists.txt
COPY main.cpp /sdk-example/main.cpp
RUN mkdir sdk-example/build &&\
    pushd sdk-example/build &&\
    cmake -DCMAKE_PREFIX_PATH=/sdk-install .. &&\
    cmake --build . &&\
    popd

main.cpp (just yours)

#include <iostream>
#include <string>
#include <aws/core/Aws.h>
#include <aws/core/utils/HashingUtils.h>
#include <aws/core/utils/memory/stl/AWSString.h>
#include <openssl/opensslv.h>

int main(int argc, char* argv[]) {
    // Initialize AWS SDK
    Aws::SDKOptions options;
    Aws::InitAPI(options);

    // Print platform and OpenSSL info
    std::cout << "=== Platform Information ===" << std::endl;
#ifdef __aarch64__
    std::cout << "Architecture: ARM64 (aarch64)" << std::endl;
#elif defined(__x86_64__)
    std::cout << "Architecture: x86_64" << std::endl;
#else
    std::cout << "Architecture: Unknown" << std::endl;
#endif

    std::cout << "OpenSSL version: " << OPENSSL_VERSION_TEXT << std::endl;
    std::cout << std::endl;

    // Test payload (similar to DeleteObjects XML)
    std::string test_payload =
        "<?xml version=\"1.0\" encoding=\"UTF-8\"?>"
        "<Delete>"
        "<Object><Key>test-object-1</Key></Object>"
        "<Object><Key>test-object-2</Key></Object>"
        "</Delete>";

    std::cout << "=== MD5 Calculation Test ===" << std::endl;
    std::cout << "Test payload: " << test_payload << std::endl;
    std::cout << "Payload size: " << test_payload.length() << " bytes" << std::endl;
    std::cout << std::endl;

    // Calculate MD5 using AWS SDK
    std::cout << "Calling Aws::Utils::HashingUtils::CalculateMD5()..." << std::endl;
    auto md5_bytebuffer = Aws::Utils::HashingUtils::CalculateMD5(test_payload);

    std::cout << "MD5 ByteBuffer length: " << md5_bytebuffer.GetLength()
              << " (expected: 16)" << std::endl;

    if (md5_bytebuffer.GetLength() == 0) {
        std::cout << "ERROR: CalculateMD5() returned EMPTY ByteBuffer!" << std::endl;
    } else if (md5_bytebuffer.GetLength() == 16) {
        std::cout << "✓ SUCCESS: MD5 calculation returned correct length" << std::endl;

        // Calculate Base64 encoding
        auto base64_md5 = Aws::Utils::HashingUtils::Base64Encode(md5_bytebuffer);
        std::cout << "Base64 MD5: " << base64_md5.c_str() << std::endl;
        std::cout << "Base64 length: " << base64_md5.length() << " (expected: 24)" << std::endl;
    } else {
        std::cout << "ERROR: Unexpected MD5 ByteBuffer length!" << std::endl;
    }

    std::cout << std::endl;

    // Cleanup AWS SDK
    Aws::ShutdownAPI(options);

    return (md5_bytebuffer.GetLength() == 16) ? 0 : 1;
}

replicate.sh

#!/bin/zsh

set -u

# build image
docker build -t test-image .

# run example
docker run --name test-image test-image /sdk-example/build/sdk_usage_workspace

when ./replicate.sh is run it will output

=== Platform Information ===
Architecture: ARM64 (aarch64)
OpenSSL version: OpenSSL 3.2.3 3 Sep 2024

=== MD5 Calculation Test ===
Test payload: <?xml version="1.0" encoding="UTF-8"?><Delete><Object><Key>test-object-1</Key></Object><Object><Key>test-object-2</Key></Object></Delete>
Payload size: 137 bytes

Calling Aws::Utils::HashingUtils::CalculateMD5()...
MD5 ByteBuffer length: 16 (expected: 16)
✓ SUCCESS: MD5 calculation returned correct length
Base64 MD5: kaK1wnr5JyA5xE9bdzBuiw==
Base64 length: 24 (expected: 24)

Which is what is expected. This leads me to believe its something in your environment. If you can update the dockerfile to mimic your environment would be happy to take a look. But from this standalone environment perspective the SDK is working as intended.

reproducible project:
suse-example.zip

[sbiscigl]

AWS CPP SDK version used
1.11.630

This causes S3 multi-delete requests (which require Content-MD5)

Additionally this version uses CRC64 checksums by default on DeleteObjects. This version will not use MD5 by default on any APIs unless you have the sdk configured to work around it.

[hqho]

I've ran with the dockerfile that was giving and still getting the empty buffer.

 ---> Running in 37fb6ef86ae6
 ---> Removed intermediate container 37fb6ef86ae6
 ---> 296aa7547eaa
Step 6/8 : COPY CMakeLists.txt /sdk-example/CMakeLists.txt
 ---> 614eb6bbf0c7
Step 7/8 : COPY main.cpp /sdk-example/main.cpp
 ---> a12a7e5312a7
Step 8/8 : RUN mkdir sdk-example/build &&    pushd sdk-example/build &&    cmake -DCMAKE_PREFIX_PATH=/sdk-install .. &&    cmake --build . &&    popd
 ---> Running in 6c2340f9b700
/sdk-example/build /
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found AWS SDK for C++, Version: 1.11.676, Install Root:/usr/local, Platform Prefix:, Platform Dependent Libraries: pthread;curl
-- Components specified for AWSSDK: core, application will be depending on libs: aws-cpp-sdk-core
-- Try finding aws-cpp-sdk-core
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found crypto: /usr/lib64/libcrypto.so
-- LibCrypto Include Dir: /usr/include
-- LibCrypto Shared Lib:  /usr/lib64/libcrypto.so
-- LibCrypto Static Lib:  crypto_STATIC_LIBRARY-NOTFOUND
-- Found OpenSSL: /usr/lib64/libcrypto.so (found version "3.2.3")
-- Found aws-cpp-sdk-core
-- Configuring done (2.1s)
-- Generating done (0.0s)
-- Build files have been written to: /sdk-example/build
[ 50%] Building CXX object CMakeFiles/sdk_usage_workspace.dir/main.cpp.o
[100%] Linking CXX executable sdk_usage_workspace
[100%] Built target sdk_usage_workspace
/
 ---> Removed intermediate container 6c2340f9b700
 ---> 8d83e66c6f7e
Successfully built 8d83e66c6f7e
Successfully tagged test-image:latest
=== Platform Information ===
Architecture: ARM64 (aarch64)
OpenSSL version: OpenSSL 3.2.3 3 Sep 2024

=== MD5 Calculation Test ===
Test payload: <?xml version="1.0" encoding="UTF-8"?><Delete><Object><Key>test-object-1</Key></Object><Object><Key>test-object-2</Key></Object></Delete>
Payload size: 137 bytes

Calling Aws::Utils::HashingUtils::CalculateMD5()...
MD5 ByteBuffer length: 0 (expected: 16)
❌ ERROR: CalculateMD5() returned EMPTY ByteBuffer!

out.txt

](url)

[hqho]

AWS CPP SDK version used
1.11.630

This causes S3 multi-delete requests (which require Content-MD5)

Additionally this version uses CRC64 checksums by default on DeleteObjects. This version will not use MD5 by default on any APIs unless you have the sdk configured to work around it.

Yes, we did override CRC64 with MD5 for DeleteObjects by following https://docs.aws.amazon.com/sdk-for-cpp/latest/api/root/html/md_docs_2_m_d5_checksum_fallback.html

[sbiscigl]

Yes, we did override CRC64 with MD5 for DeleteObjects

Why? CRC64 is a better algorithm in most use cases.

I've ran with the dockerfile that was giving and still getting the empty buffer.

I pushed this example, and ran it in a github runner and i still see it working

=== Platform Information ===
Architecture: ARM64 (aarch64)
OpenSSL version: OpenSSL 3.2.3 3 Sep 2024
=== MD5 Calculation Test ===
Test payload: <?xml version="1.0" encoding="UTF-8"?><Delete><Object><Key>test-object-1</Key></Object><Object><Key>test-object-2</Key></Object></Delete>
Payload size: 137 bytes
Calling Aws::Utils::HashingUtils::CalculateMD5()...
MD5 ByteBuffer length: 16 (expected: 16)
✓ SUCCESS: MD5 calculation returned correct length
Base64 MD5: kaK1wnr5JyA5xE9bdzBuiw==
Base64 length: 24 (expected: 24)

I suspect there is something wrong with your process. if i run it in a CI container it will work as expected. I can not debug your personal environment any more than this. There is something wrong in your reproduction steps.