RabbitHub httpc error on POST to Callback URL (socket_closed_remotely)

[gfiehler]

We are using the RabbitHub Rabbitmq plugin (https://github.com/brc859844/rabbithub) and experiencing the following issue:

When a queue is pre-populated (say 1000 msgs) then a subscriber is created, or when the publishing volume is sufficiently higher than the subscriber callback rate, each subscriber POST's 99 messages to the callback URL then gets the following error

=ERROR REPORT==== 12-Apr-2016::17:47:51 ===

RabbitHub post error

{rabbithub_consumer,http_post_failure,socket_closed_remotely,{}}

=INFO REPORT==== 12-Apr-2016::17:47:51 ===

RabbitHub stopping consumer, normal

{state,{rabbithub_subscription,{resource,<<"/">>,queue,<<"foo">>},

                           "foo",

                           "http://gregg-VirtualBox:8999/rabbithub/sub1"},

   #Ref<0.0.1.85179>,<<"amq.http.consumer-zLIH5VOb2BdVbtLR_4J4pw">>}

If we re-up the subscription then the same pattern happens again, 99 msgs are POSTed to the callback URL then we get the same error. The error is per subscriber as I setup 2 queues, 1 with 1 subscriber, 1 with 2 subscribers each with a unique callback URL and each callback URL received 99 msg's then the error occurred for each subscriber.

We tried another server with similar situation and the number was as consistent, but always failed between 98-101 messages.

The backend server hosting the callback does not seem to be the issue, we can hit it directly with a load much higher than what RabbitHub is sending with no issues at all.

Based on some forum posts on the socket_closed_remotely error we tried the following rabbitmq.config

{rabbithub, [

    {http_client_options, [

     {max_keep_alive_length, 0},

{max_pipeline_length, 0}, {max_sessions, 0}

    ]}

]}



            but it had no effect on the situation.  Any assistance on how to fix this issue would be greatly appreciated.

Thanks

Gregg

[brc859844]

Gregg,

Probably best to ping me directly at brett.r.cameron@gmail.com.

In terms of the problem, are you by any chance using Apache? Reason I ask is that by default Apache sets MaxKeepAliveRequests to 100, which seems suspiciously close to your 99-101 range.

It certainly looks like the web server is chopping the connection, and it's a fair bet that it's HTTP keepalive-related (after some maximum number of requests from a given client the connection is being closed by the web server). Your idea of setting max_keep_alive_length seems reasonable; however I this parameter pertains to the maximum number of outstanding (in-flight) requests on the same connection as opposed to the maximum number of (completed) requests (within a given period) before the connection is reset. As a test, are you able to set MaxKeepAliveRequests (or whatever is appropriate for the web server in question) to 0 or some other value and see whether this changes the situation? I have seen reports of issues with the Erlang HTTP client and keepalives, but would need to do a bit more digging. Certainly other RabbitHub users have not reported seeing this behavior, but I'm not sure what web server(s) they're using.

For what it's worth, Tony's original code had a simple and effective hand-rolled HTTP client, which established a fresh connection for each request, but I switched to the Erlang client to provide a bit more functionality (like support for HTTPS).

Regards,
Brett

[gfiehler]

Hi Brett,  We have decided to use Rabbitmq and Rabbithub and am now looking at a couple of enhancements for Rabbithub.  I just finished my first try at erlang and am attaching my new rabbithub_web.erl to see if the way I added an endpoint to retrieve a list of subscriptions is ok.
  Happy to make any adjustments to meet your requirements to keep things consistent.  Each of my code additions have a comment %% GF: ... so you can find them easily.
  Our next enhancement we are considering is to support the publishing of a message to a header exchange with message headers.  Wanted to know if you have any thoughts around the best way to pass message headers in a POST.  I was thinking of specially named HTTP headers like RabbitHeader.=value and parse thru them or I guess we could add query parameters in much the same way.  Any thoughts on what would be most appropriate would be appreciated.
  at this point we may not need to support this via Rabbithub since these publishers may just use amqp directly from java, but we haven't finalized that design yet.
 If it is the best approach, I can clone on github and submit a pull request when all is ready, but wanted to get your feedback while we are still early in the process.
Thanks for your support and a great product.
Gregg

On Monday, April 25, 2016 6:43 PM, Brett Cameron <notifications@github.com> wrote:

Closed #5.—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub