Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
PHP Fatal error: Uncaught ErrorException: fwrite(): send of 19 bytes failed with errno=32 #424
Sometimes after send basic_nack
Weirdly im getting exactly the same. Its been fine for ages. @Mirocow are you on AWS?
@michaelklishin the server logs dont really reveal anything useful :(
I spent a little bit of time debugging this yesterday, I think I've figured it out. From what I can determine it looks like I had a blocking process that was taking longer than the heartbeat. RabbitMQ was then closing the channel (I can see this in the Management UI) hence the broken pipe error.
If im correct then I guess there needs to be a more elegant way of handling when the lib misses a heartbeat and the channel is closed.
PHP 7.0.8 (cli) (built: Jun 23 2016 16:30:56) ( NTS )
I check it on Mac OS and Debian OS too.
when i don`t use basic_nack or basic_ack process work perfect
UPD: in that situation we had 2K of new messages each 30min, and it took us between 0.1 and 30 seconds to process and ack/nack a message. We had 60 as a heartbeat. And we often stuck with php exception after processing ~250 messages.
Look in here also: #91
I have the same issue. I did nothing but is started to appear. Tried to rise up timeouts and heartbeat values up to maximum values, tried to set keepalive to true but without success. Server says only this in his log:
My message producer sends 4-6 messages to the server and fails with the exception as in the subject.
means that a socket write on the RabbitMQ end timed out. There isn't much RabbitMQ or this client can do to avoid them.
simply means that a client connection was closed (lines before might indicate if the close was server-initiated but all we have here is a single line).
Missed heartbeats are visibly logged by the server. If a consumer can block I/O activity for longer than a heartbeat timeout, the server will indeed consider the peer unavailable but it will log more than just
In general there are all kinds of things that can result in failed socket writes and not all of them are under client library's control. @adamlc explained one scenario above, which comes down to the fundamental limitation to how this client does I/O and dispatches consumer operations. Plenty of other cases are probably just failed socket writes.
Java and .NET clients ignore all I/O errors when a connection is being closed (because they don't matter any more). If someone has evidence that this client doesn't, please file a new issue with more details.
referenced this issue
Sep 26, 2017
This thread seems to be a honeypot for all
Missed (Client) Heartbeats
First common reason is missed heartbeats detected by RabbitMQ. When this happens, RabbitMQ will add a log entry about it and then close connection, per specification requirements.
If consumer operations take longer than the heartbeat timeout, without substantial changes to how this client does I/O and operation dispatch, increasing the heartbeat is the only workaround I can think of (I'm hardly a PHP expert but pretty familiar with multiple other clients as well as RabbitMQ itself). Disabling heartbeats entirely is possible but is not something I'd recommend, even
An Intermediary Closes "Inactive" TCP Connections
Second common reason: TCP connection is closed by an intermediary (e.g. a proxy or load balancer).
it means that client's TCP connection was closed before AMQP 0-9-1 (this client's) connection was. Sometimes this is harmless and means that apps do not close connections before they terminate. Not very nice but has no functional downsides. Such log entries could also indicate a failing client application process (irrelevant in this thread), or, quite commonly, a proxy closing the connection. Proxies and load balancers have TCP connection inactivity timeouts, mentioned in the Heartbeats guide). They often range from 30s to 5m.
Other Connection Lifecycle Log Entries
Below entries are not necessarily related to failed socket writes but it's worth explaining them
If you see just the following in RabbitMQ logs:
(without any mentions or heartbeats, unexpectedly closed TCP connection, connection errors, or any timed out socket writes on RabbitMQ's end), it means that a client connection was cleanly and successfully closed, and it was the application that initiated it.
means that RabbitMQ attempted to write to a socket but that operation timed out. If you see this
Other Possible Reasons? TCP Connections Can Fail.
In most other cases, a failed socket write is just that, a failed socket write. Network connections can fail or degrade. This client or RabbitMQ cannot avoid that. This is exactly why messaging protocols such as AMQP 0-9-1, STOMP, MQTT have introduced heartbeats, which in this client doesn't serve its purpose very well.
An Alternative with Unfortunate Up Defaults: TCP Keepalives
TCP keepalives can be (and were meant to be, if it wasn't for Linux defaults that were great in 1990s but not any more) used as an alternative.
Hopefully this explains what may be going on here and why this issue cannot be once and for all addressed by this client, although a functioning concurrent heartbeat implementation would help a lot, as it does in other clients.
I was getting the same issue, calling basic_ack throws that fwrite connection error, not for all messages, but for most of them. It starts consuming messages just fine and after some time I start seeing those errors (in a PHP cli long running script). It's been working like a charm for 2 years until I recently updated 2.4 => 2.6/2.7.
The quick workaround (I guess not the ideal solution tho) is @keatis suggestion, setting prefetch_count to 1:
slower but safe.
I solved it this way:
tl; dr :
For the record.. had similar issue, suspecting AWS ELB
Consumer connecting to Service 1 (LoadBalancer) have been getting frequent errors
From what I can tell, heartbeat is not set explicitly in PHP, and should default to 60.
Issue went away when consumer was configured to connect to Service 2 (ClusterIP), instead of connecting to Service 1 (LoadBalancer).
@michaelklishin thanks for very detailed response. It helps a lot.
However I still have one case:
Looking on errno=11 for fwrite -
Do you have any tips/ideas?
In my case the problem was in the type of the $heartbeat parameter of the AMQPStreamConnection. When I sent null instead of 0, I got few kinds of connection corruption error messages. After fixing my code to send int 0, the problem has gone.
I suggest the authors to add type checking of the input parameters of the AMQPStreamConnection constructor.