SSL write() gets broken pipe signal SIGPIPE

[kuroburki]

I am using latest version of SAC and rabbitmq-c code for my C++ client on Linux. The broker and clients are configured in SSL and broker is configured in a 2 node cluster. While testing for failover, I noticed my C++ client would terminate if I issued rabbitmqctl stop_app on the master node. The call stack shows the process received SIGPIPE while performing write() within the openssl layer. I looked up the code and can find the non-SSL part ignores this signal at a call level using MSG_NOSIGNAL on send(). Is this not handled similarly for the SSL connection? I have currently ignored the signal at my process level. Further reading suggests it can be ignored at a socket level using setsockopt. Let me know if this was considered for the SSL feature support. I would like to know if this is something which can be handled in the library or at the application level.

EDIT: SO_NOSIGPIPE is a BSD implementation and not supported on Linux. Does this mean I am limited to ignoring signal at process level for now? I have checked internally and this is feasible.

[kuroburki]

Program received signal SIGPIPE, Broken pipe.
0x00007ffff7bcd1cd in write () from /usr/lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff7bcd1cd in write () from /usr/lib64/libpthread.so.0
#1  0x00007ffff76459e6 in sock_write () from /usr/lib64/libcrypto.so.10
#2  0x00007ffff7643a4b in BIO_write () from /usr/lib64/libcrypto.so.10
#3  0x00007ffff797d782 in ssl3_write_pending () from /usr/lib64/libssl.so.10
#4  0x00007ffff797de33 in ssl3_write_bytes () from /usr/lib64/libssl.so.10
#5  0x000000000046cbea in amqp_ssl_socket_send (base=0x7fffec0034a0, buf=0x7ffff7fa9010, len=22, flags=0) at rabbitmq/amqp_openssl.c:98
#6  0x0000000000477ba7 in amqp_socket_send (self=0x7fffec0034a0, buf=0x7ffff7fa9010, len=22, flags=0) at rabbitmq/amqp_socket.c:209
#7  0x0000000000477fd3 in amqp_try_send (state=0x7fffec003b90, buf=0x7ffff7fa9010, len=22, deadline=..., flags=0) at rabbitmq/amqp_socket.c:384
#8  0x000000000046bf40 in amqp_send_frame_inner (state=0x7fffec003b90, frame=0x7ffff41162d0, flags=0) at rabbitmq/amqp_connection.c:560
#9  0x0000000000479253 in amqp_send_method_inner (state=0x7fffec003b90, channel=0, id=655410, decoded=0x7ffff4116410, flags=0) at rabbitmq/amqp_socket.c:1036
#10 0x0000000000479204 in amqp_send_method (state=0x7fffec003b90, channel=0, id=655410, decoded=0x7ffff4116410) at rabbitmq/amqp_socket.c:1024
#11 0x00000000004792f0 in amqp_simple_rpc (state=0x7fffec003b90, channel=0, request_id=655410, expected_reply_ids=0x7ffff4116430, decoded_request_method=0x7ffff4116410) at rabbitmq/amqp_socket.c:1061
#12 0x000000000047aa33 in amqp_connection_close (state=0x7fffec003b90, code=200) at rabbitmq/amqp_api.c:301
#13 0x000000000043644b in AmqpClient::Channel::~Channel (this=0x7fffec0035a0, __in_chrg=<optimized out>) at rabbitmq/Channel.cpp:200
#14 0x000000000042f45c in boost::detail::sp_ms_deleter<AmqpClient::Channel>::destroy (this=0x7fffec003598) at /usr/include/boost/smart_ptr/make_shared_object.hpp:57
#15 0x000000000042e414 in boost::detail::sp_ms_deleter<AmqpClient::Channel>::operator() (this=0x7fffec003598) at /usr/include/boost/smart_ptr/make_shared_object.hpp:87
#16 0x0000000000430bd3 in boost::detail::sp_counted_impl_pd<AmqpClient::Channel*, boost::detail::sp_ms_deleter<AmqpClient::Channel> >::dispose (this=0x7fffec003580)
    at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:153
#17 0x0000000000427d02 in boost::detail::sp_counted_base::release (this=0x7fffec003580) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
#18 0x0000000000427d91 in boost::detail::shared_count::~shared_count (this=0x7ffff4116588, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:371
#19 0x0000000000427f18 in boost::shared_ptr<AmqpClient::Channel>::~shared_ptr (this=0x7ffff4116580, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:328
#20 0x0000000000429bac in boost::shared_ptr<AmqpClient::Channel>::operator=(boost::shared_ptr<AmqpClient::Channel>&&) (this=0x7fffec0009d8,

[alanxz]

Out of the box OpenSSL doesn't provide a good way to do this that works everywhere. If you're on a BSD-based platform then its possible to do a setsockopt with SO_NOSIGPIPE. Linux on the other hand, this is done with a flag MSG_NOSIGNAL passed into the write call; the OpenSSL SSL_write API doesn't provide a way to pass this in.

The workaround on Linux is as you suggest, ignoring the signal using something like signal(SIGPIPE, SIG_IGN) (rabbitmq-c the library should not do this as it its process-wide).

A longer-term solution is to write an OpenSSL BIO that correctly handles this.

[alanxz]

@kuroburki I've added support to rabbitmq-c for ignoring SIGPIPE in an SSL connection on Linux in ab256d1. Could you try it out and see if it works?