PUSH-PULL with curve leads to loss of events in case of a backpressure

[rajeshnair]

Problem

We use JeroMQ 0.5.1 to build a PUSH-PULL pipeline
We have strict requirement to ensure no loss of events and we do a quite a few number of settings

1. We ensure socket.setLinger(0) on both pusher and puller
2. We ensure socket.setTCPKeepAlive(1) on both pusher and puller
3. We ensure socket.setSendTimeout(-1) on pusher to block
4. We set reasonable HWM of 50K with buffer size of 1M

This all has proven to work for Plain connection

But with CurveZMQ, with a very fast pusher and very slow puller , we see ZMQ logical connection being broken when the HWM is reached
This does two things

1. We receive the CurveZMQ MESSAGE command in the puller ZMsg.recvMsg . I presume it indicates new TLS handshake
2. We see loss of message when this handshake happens

If I increase the HWM , the number of missed events reduces which leads me to believe this is a backpressure problem due to fast producer and slow consumer

We are seeking

1. Is my assumption correct that this is a backpressure problem due to fast producer and slow consumer ?
2. I am able to reproduce this with CurveZMQ on a standalone program but unable to reproduce when I remove Curve. So is this simply due to the fact that the events are missed during the time it takes for TLS handshake to complete
3. I looked into the ZMQ.Socket but could not find any method that would give me the current usage of the internal buffer . Am I missing something or is that by design ? A method like this can help application developer to implement a throttling alogorithm at application level
4. Do we have any configuration for Pusher socket to throttle if the receiver buffer is full ?

cc: @trevorbernard , @fbacchella , @minrk , @fredoboulo