[S3Crt] Unable to do getobject when explicit 'Aws::S3Crt::ClientConfiguration::ca_path' is set to default path

[csi-amolpawar]

Describe the bug

Unable to do getobject when explicit 'Aws::S3Crt::ClientConfiguration::ca_path' is set to default path

Expected Behavior

Get Object should work as expected when ca_path is set explicitly.

When we don't set ca_path explicitly, it work fine.

Current Behavior

Receives the error message GetObject error:TLS (SSL) negotiation failed (aws-c-io: AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE)

Reproduction Steps

The issue is easily reproducible with below code snippet

#include <iostream>
#include <string>
#include <openssl/crypto.h>
#include <aws/core/Aws.h>
#include <aws/core/utils/memory/stl/AWSStringStream.h>
#include <aws/core/utils/logging/CRTLogSystem.h>
#include <aws/s3-crt/S3CrtClient.h>
#include <aws/s3-crt/model/GetObjectRequest.h>

static const char ALLOCATION_TAG[] = "s3-crt-getobject-public";

std::string get_default_openssl_dir()
{
  const std::string OPENSSLDIR_KEY("OPENSSLDIR: ");

  auto ssl_dir = std::string(SSLeay_version(SSLEAY_DIR));
  auto found = ssl_dir.find(OPENSSLDIR_KEY);
  if(found != std::string::npos)
  {
    ssl_dir = ssl_dir.substr(OPENSSLDIR_KEY.size());
    if(auto s = ssl_dir.size(); ssl_dir.at(0) == '"' && ssl_dir.at(s - 1) == '"')
      ssl_dir = ssl_dir.substr(1, s -2);
  }
  return ssl_dir;
}

int main(int argc, char* argv[])
{
  Aws::SDKOptions options;

  Aws::String ca_path = get_default_openssl_dir();

  options.httpOptions.initAndCleanupCurl = false;
  options.cryptoOptions.initAndCleanupOpenSSL = false;
  options.ioOptions.tlsConnectionOptions_create_fn = [=]() {
    Aws::Crt::Io::TlsContextOptions tlsCtxOptions = Aws::Crt::Io::TlsContextOptions::InitDefaultClient();
    tlsCtxOptions.SetVerifyPeer(true);
    Aws::Crt::Io::TlsContext tlsContext(tlsCtxOptions, Aws::Crt::Io::TlsMode::CLIENT);
    return Aws::MakeShared<Aws::Crt::Io::TlsConnectionOptions>(ALLOCATION_TAG, tlsContext.NewConnectionOptions());
  };

  options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
  options.loggingOptions.crt_logger_create_fn = []() {
    return Aws::MakeShared<Aws::Utils::Logging::DefaultCRTLogSystem>(
      ALLOCATION_TAG, Aws::Utils::Logging::LogLevel::Trace);
  };

  Aws::InitAPI(options);
  {    
    Aws::S3Crt::ClientConfiguration config;
    config.region = Aws::Region::US_EAST_1;
    config.caPath = Aws::String(ca_path);
    Aws::S3Crt::S3CrtClient s3CrtClient(config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never);
    Aws::String bucket("my-tests");
    Aws::String objectKey("test/my_object");

    Aws::S3Crt::Model::GetObjectRequest request;
    request.SetBucket(bucket);
    request.SetKey(objectKey);
    
    if(auto outcome = s3CrtClient.GetObject(request); outcome.IsSuccess())
      std::cout << outcome.GetResult().GetBody().rdbuf() << std::endl;
    else
      std::cerr << "GetObject error:" << outcome.GetError().GetMessage() << std::endl;
  }
  Aws::ShutdownAPI(options);
  return 0;
}

Please note that get_default_openssl_dir() is evaluate to /etc/pki/tls

Possible Solution

NA

Additional Information/Context

Build Command:
g++ -std=c++20 -o <output> test_s3_crt_ca_path.cpp -I${AWS_INSTALL_PATH}/include -L${AWS_INSTALL_PATH}/lib64 -lcurl -lssl -lpthread -lcrypto -laws-cpp-sdk-s3-crt -laws-cpp-sdk-core

AWS CPP SDK version used

AWS SDK for C++ 1.11.351

Compiler and Version used

g++ (GCC) 13.2.0

Operating System and version

Red Hat Enterprise Linux 9.4 (Plow)

[jmklix]

Thanks for the detailed repro code and explanation. Looking into what might be causing this to fail.

Here is what I'm seeing in the logs

[ERROR] 2024-06-27 21:25:17.139 http-connection [140737202796288] static: Client connection failed with error 1029 (AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE).
[WARN] 2024-06-27 21:25:17.139 connection-manager [140737202796288] id=0x9dd080: Failed to obtain new connection from http layer, error 1029(TLS (SSL) negotiation failed)
[DEBUG] 2024-06-27 21:25:17.139 connection-manager [140737202796288] id=0x9dd080: Failing excess connection acquisition with error code 1029
[DEBUG] 2024-06-27 21:25:17.139 connection-manager [140737202796288] id=0x9dd080: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=0, ref_count=1
[WARN] 2024-06-27 21:25:17.140 connection-manager [140737202796288] id=0x9dd080: Failed to complete connection acquisition with error_code 1029(TLS (SSL) negotiation failed)
[ERROR] 2024-06-27 21:25:17.140 S3Endpoint [140737202796288] id=0xb0a170: Could not acquire connection due to error code 1029 (TLS (SSL) negotiation failed)

[jmklix]

It looks like config.caPath might not be finding the correct file. For now you should be able to use config.caFile to workaround this:

        Aws::String ca_file = ca_path + "/cert.pem";
        config.caFile = Aws::String(ca_file);

[csi-amolpawar]

@jmklix It works fine when config.ca_file is set. See

[root@bc281d03816c aws]# ./aws_s3_crt_w_ca_file
ca_file=/etc/pki/tls/cert.pem
object_content= 'This is a test file'

Previously, we were setting only ca_path to default system path and was working as expected. See code snippet below

auto tlsCtxOptions = Aws::Crt::Io::TlsContextOptions::InitDefaultClient();
  tlsCtxOptions.SetVerifyPeer(!cfg->get<bool>(CTX_SKIP_VERIFY_SSL));
  tlsCtxOptions.OverrideDefaultTrustStore(ca_path.c_str(), "");
...

auto tlsContext = Aws::Crt::Io::TlsContext(tlsCtxOptions, Aws::Crt::Io::TlsMode::CLIENT);

config.tlsConnectionOptions = Aws::MakeShared<Aws::Crt::Io::TlsConnectionOptions>("libcsi-io-s3-crt", tlsContext.NewConnectionOptions());

and should be expected to work only setting directly in Aws::S3Crt::ClientConfiguration::ca_path

[DmitriyMusatkin]

can you print out contents of your "/etc/pki/tls" folder?

For context, s3 crt uses s2n on linux for tls support (https://github.com/aws/s2n-tls) and s2n just uses libcrypto's X509_STORE_load_locations under the covers to load all the certs. So somehow libcrypto is not able to load certs from that path. Im not super familiar with where RHEL expects to store certs, but initial guess is that certs are in a slightly different folder or process somehow is not able to access them

[csi-amolpawar]

Please see below

@DmitriyMusatkin

[DmitriyMusatkin]

And cert.pem has the root CA that covers s3 certs? Is it default RHEL's pem or is it custom one?

I did a quick test on my AL2 machine and ca_path seems to work fine.

[csi-amolpawar]

With the same CA certs, S3 is working fine.

Aws::InitAPI(options);
{    
   Aws::Client::ClientConfiguration config;
   config.region = Aws::Region::US_EAST_1;
   config.verifySSL = true;
   config.caPath = get_default_openssl_dir();
   std::cout << "using ca_path:" << config.caPath << std::endl;

   Aws::S3::S3Client s3Client(config);
   Aws::String bucket = "my_bucket";
   Aws::String objectKey = "test/my_object";
   Aws::S3::Model::GetObjectRequest request;
   request.SetBucket(bucket);
   request.SetKey(objectKey);
   Aws::S3::Model::GetObjectOutcome outcome = s3Client.GetObject(request);
   if(outcome.IsSuccess())
     std::cout << outcome.GetResult().GetBody().rdbuf() << std::endl
   else
     std::cerr << "GetObject error:" << outcome.GetError().GetMessage() << std::endl;
}
Aws::ShutdownAPI(options);

Also note that this behaviour has been observed since we upgraded to 1.11.313 from 1.11.100.

@DmitriyMusatkin

[csi-amolpawar]

And cert.pem has the root CA that covers s3 certs? Is it default RHEL's pem or is it custom one?

Yes. This is root CA and default RHEL's PEM.

We have custom self signed certs which we can not use.