Default value of installSigPipeHandler must be true

[majetideepak]

Describe the bug

The current default of installSigPipeHandler is set to false (https://github.com/aws/aws-sdk-cpp/blob/main/src/aws-cpp-sdk-core/include/aws/core/Aws.h#L83) and this can lead to hard to track process terminations as described here (https://github.com/aws/aws-sdk-cpp/blob/main/src/aws-cpp-sdk-core/include/aws/core/Aws.h#L96) and observed here (facebookincubator/velox#3800).
There are no log messages or core dumps when this happens and it becomes very hard to track.

Expected Behavior

The default behavior should be to catch the SIGPIPE signal and log a message and prevent the process from termination.

Current Behavior

When using the C++ SDK in a Velox application (Prestissimo), we noticed frequent process terminations without any notification.
We had to run the process in gdb to track the issue which is not always possible.
The backtrace observed is

Thread 69 "IOThreadPool12" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7f595abfd700 (LWP 6998)]
0x00007fb45eb1b559 in write () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007fb45eb1b559 in write () from /lib64/libpthread.so.0
#1  0x00007fb45bd19f05 in sock_write () from /lib64/libcrypto.so.10
#2  0x00007fb45bd181f5 in BIO_write () from /lib64/libcrypto.so.10
#3  0x00007fb45c080ea8 in ssl3_write_pending () from /lib64/libssl.so.10
#4  0x00007fb45c082fe3 in ssl3_dispatch_alert () from /lib64/libssl.so.10
#5  0x00007fb45c07edd6 in ssl3_shutdown () from /lib64/libssl.so.10
#6  0x00007fb45f46e960 in ossl_closeone.isra () from /lib64/libcurl.so.4
#7  0x00007fb45f46e9e2 in ossl_close () from /lib64/libcurl.so.4
#8  0x00007fb45f474fcb in Curl_ssl_close () from /lib64/libcurl.so.4
#9  0x00007fb45f46481b in Curl_disconnect () from /lib64/libcurl.so.4
#10 0x00007fb45f467a60 in Curl_connect () from /lib64/libcurl.so.4
#11 0x00007fb45f448f73 in multi_runsingle () from /lib64/libcurl.so.4
#12 0x00007fb45f449e26 in curl_multi_perform () from /lib64/libcurl.so.4
#13 0x00007fb45f42512e in curl_easy_perform () from /lib64/libcurl.so.4
#14 0x000000000399a869 in Aws::Http::CurlHttpClient::MakeRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const ()
#15 0x0000000003956840 in Aws::Client::AWSClient::AttemptOneRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const ()
#16 0x000000000395ef0a in Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const ()
#17 0x000000000396035f in Aws::Client::AWSClient::MakeRequestWithUnparsedResponse(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const ()
#18 0x00000000038c7e40 in Aws::S3::S3Client::GetObject(Aws::S3::Model::GetObjectRequest const&) const ()
#19 0x00000000037a24bf in facebook::velox::(anonymous namespace)::S3ReadFile::preadInternal (this=0x7f5a00002d90, offset=1242816535, length=8388608, position=0x7f594c259068 "")
    at /home/ec2-user/presto/presto-native-execution/velox/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp:164
#20 0x00000000037a207d in facebook::velox::(anonymous namespace)::S3ReadFile::preadv (this=0x7f5a00002d90, offset=1242816535, buffers=std::vector of length 1, capacity 1 = {...})
    at /home/ec2-user/presto/presto-native-execution/velox/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp:116
#21 0x0000000005d567ff in facebook::velox::dwio::common::ReadFileInputStream::read (this=0x7f59f847dda0, buffers=std::vector of length 1, capacity 1 = {...}, offset=1242816535, 
    logType=facebook::velox::dwio::common::MetricsLog::MetricsType::FILE) at /home/ec2-user/presto/presto-native-execution/velox/velox/dwio/common/InputStream.cpp:183
#22 0x0000000005d17063 in facebook::velox::dwio::common::(anonymous namespace)::DwioCoalescedLoad::<lambda(const std::vector<facebook::velox::cache::CachePin, std::allocator<facebook::velox::cache::CachePin> >&, int32_t, int32_t, uint64_t, const std::vector<folly::Range<char*>, std::allocator<folly::Range<char*> > >&)>::operator()(const std::vector<facebook::velox::cache::CachePin, std::allocator<facebook::velox::cache::CachePin> > &, int32_t, int32_t, uint64_t, const std::vector<folly::Range<char*>, std::allocator<folly::Range<char*> > > &) const (
    __closure=0x7f595abf1d10, offset=1242816535, buffers=std::vector of length 1, capacity 1 = {...})
    at /home/ec2-user/presto/presto-native-execution/velox/velox/dwio/common/CachedBufferedInput.cpp:395
#23 0x0000000005d18de3 in std::_Function_handler<void(const std::vector<facebook::velox::cache::CachePin, std::allocator<facebook::velox::cache::CachePin> >&, int, int, long unsigned int, const std::vector<folly::Range<char*>, std::allocator<folly::Range<char*> > >&), facebook::velox::dwio::common::(anonymous namespace)::DwioCoalescedLoad::loadData(bool)::<lambda(const std::vector<facebook::velox::cache::CachePin, std::allocator<facebook::velox::cache::CachePin> >&, int32_t, int32_t, uint64_t, const std::vector<folly::Range<char*>, std::allocator<folly::Range<char*> > >&)> >::_M_invoke(const std::_Any_data &, const std::vector<facebook::velox::cache::CachePin, std::allocator<facebook::velox::cache::CachePin> > &, int &&, int &&, unsigned long &&, const std::vector<folly::Range<char*>, std::allocator<folly::Range<char*> > > &) (__functor=..., __args#0=std::vector of length 1, capacity 1 = {...}, __args#1=@0x7f595abf1bac: 0, 
    __args#2=@0x7f595abf1ba8: 1, __args#3=@0x7f595abf1ba0: 1242816535, __args#4=std::vector of length 1, capacity 1 = {...})
    at /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/std_function.h:300
#24 0x0000000005e3a086 in std::function<void (std::vector<facebook::velox::cache::CachePin, std::allocator<facebook::velox::cache::CachePin> > const&, int, int, unsigned long, std::vector<folly::Range<char*>, std::allocator<folly::Range<char*> > > const&)>::operator()(std::vector<facebook::velox::cache::CachePin, std::allocator<facebook::velox::cache::CachePin> > const&, int, int, unsigned long, std::vector<folly::Range<char*>, std::allocator<folly::Range<char*> > > const&) const (this=0x7f595abf1d10, __args#0=std::vector of length 1, capacity 1 = {...}, 
    __args#1=0, __args#2=1, __args#3=1242816535, __args#4=std::vector of length 1, capacity 1 = {...}) at /opt/rh/devtoolset-9/root/usr/include/c++/9/bits/std_function.h:688

Reproduction Steps

We need to query data on S3 with Presto with the C++ native worker (Prestissimo).
Steps to install Presto https://github.com/prestodb/presto

Possible Solution

Set the default to true

Additional Information/Context

No response

AWS CPP SDK version used

1.9.379

Compiler and Version used

C++ Compiler Version: 9.3.1

Operating System and version

AmazonLinux2 Linux-5.10.162-141.675.amzn2.x86_64

[SergeyRyabinin]

Hi @majetideepak ,

Thank you a lot for your query.
Unfortunately, I don't think we are able to set a signal sigpipe handler by default. The reason is that the signal handlers are global per application/process and the common expectation is that libraries shall not modify the global state of the application (I know our SDK uses global state variable, but at least they are limited to the SDK library).

However, we are going to describe the issue of OpenSSL+Curl sending the signal, and the option to install a default dummy sigpipe handler more explicitly in our documentation.

Best regards,
Sergey

[majetideepak]

@SergeyRyabinin thanks for the clarification!

[jmklix]

We have updated our documentation here:

Use a global SIGPIPE handler

If you build the SDK for C++ with curl and OpenSSL, you must specify a signal handler. If you don’t use your own custom signal handler, set installSigPipeHandler to true.

Thanks for bringing the original issue to our attention. Please let us know if this or any other parts of our documentation need more clarification.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.