"Failed to write, and not due to blocking" on Memcached 1.5.3

[blkperl]

I'm trying to debug an issue where connections to memcache timeout after a stream of EAGAIN messages on the recvfrom and sendto system calls.

Temporarily upgrading to the latest version of memcached did not resolve the issue. It can be reproduced on multiple servers across different availability zones and the servers have been replaced and the issue eventually reoccurs on the new servers.

On the client

I have been unable to isolate the issue from Drupal so far as testing with php manually works fine. The initial drush9 command also works after memcached restart but fails on all following occurrences.

timeout 120 strace -s 2048 -Tfe trace=network drush9 php-eval 'true;'
[...]
[pid  1140] sendto(8,   [....] , 8196, MSG_NOSIGNAL|MSG_MORE, NULL, 0) = -1 EAGAIN (Resource temporarily unavailable) <0.000020>

During this time ~13,000 sets and gets are performed from the Drupal bootstrap.

On the with server -vvv flags enabled

Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: <37 GET 674e2bb21b4c41979d663e0d26dd12f9_%3Aconfig%3A-core.base_field_override.node.product_page_block.status
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: > FOUND KEY 674e2bb21b4c41979d663e0d26dd12f9_%3Aconfig%3A-core.base_field_override.node.product_page_block.status
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: >37 Writing bin response:
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: >37   0x81 0x0d 0x00 0x65
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: >37   0x04 0x00 0x00 0x00
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: >37   0x00 0x00 0x04 0x46
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: >37   0x02 0x3e 0x00 0x00
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: >37   0x00 0x00 0x00 0x00
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: >37   0x00 0x00 0x00 0x00
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: 37: going from conn_nread to conn_mwrite
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: Failed to write, and not due to blocking: Connection reset by peer
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: 37: going from conn_mwrite to conn_closing
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: <37 connection closed.
Mar 05 19:59:46 web-12345.redacted.com memcached[32007]: 37: going from conn_closing to conn_closed

Version info

   ["version"]=>
    string(5) "1.5.3"
    ["libevent"]=>
    string(13) "2.0.21-stable"

Client Config

$settings['memcache']['options'][Memcached::OPT_COMPRESSION] = TRUE;
$settings['memcache']['options'][Memcached::OPT_BINARY_PROTOCOL] = TRUE;

[dormando]

Sounds like the client killed the connection for some reason :( Server tries to write but gets peer'ed.

Looks like php-memcached, libmemcached based client. What client version? + What distro/where did the server and client builds come from?

[blkperl]

Client is from Ubuntu repositories. Server is compiled and packaged manually.

• php-memcached 3.1.3
• libmemcached11 1.0.18-4.1
• Ubuntu Xenial with memcached compiled from source (1.5.3)
• PHP 7.2 and PHP 7.1

I also opened php-memcached-dev/php-memcached#429

[dormando]

Well the error is likely on the client, unless the server is returning a malformed response. Hopefully you'll have more luck with the php-memcached project? At some point somewhere the client should be emitting an error code.

[blkperl]

Alright, thanks for narrowing it down. I'll wait for them to respond on php-memcached-dev/php-memcached#429

[glennpratt]

@dormando what does this mean, do you mean the client closed an end of the connection or something else? I'm not seeing close in strace.

Server tries to write but gets peer'ed.

[dormando]

@glennpratt If the server is getting "connection reset by peer" when writing, it means it received a TCP RST (connection was reset). That could happen for a number of reasons (firewalls? client crash? fd recycle?), so I can't really tell you why offhand.

[glennpratt]

Thanks for the feedback, greatly appreciated. I can't say I see any of those things from strace or tcpdump.

Here's a Wireshark summary from both sides:

https://gist.github.com/glennpratt/1973c7e5f423dfe7365c72d89c34ab21

[dormando]

Got some "acked unseen segments" toward the end of the capture. packet loss/rate issues?

[glennpratt]

Memcached Server

ssh server 'netstat -s | grep -i retrans'
    169742 segments retransmited
    TCPLostRetransmit: 1129
    28874 fast retransmits
    1985 forward retransmits
    1254 retransmits in slow start
    156 SACK retransmits failed
    TCPSynRetrans: 5320

Bad Client machine

ssh bad-client 'netstat -s | grep -i retrans'
    1497604 segments retransmited
    TCPLostRetransmit: 16
    3963 fast retransmits
    353 forward retransmits
    286 retransmits in slow start
    23 SACK retransmits failed
    TCPSynRetrans: 5077

Good Client machine

ssh good-client 'netstat -s | grep -i retrans'
    318614 segments retransmited
    TCPLostRetransmit: 2
    2547 fast retransmits
    600 forward retransmits
    293 retransmits in slow start
    75 SACK retransmits failed
    TCPRetransFail: 1
    TCPSynRetrans: 25842

[glennpratt]

Also, here's the wireshark summaries from a good session:

https://gist.github.com/glennpratt/9e29e54504806acdc2fc8feca35558f8

[dormando]

confused; what's a good client machine vs bad client machine? only certain clients get disconnected from even the same servers?

[glennpratt]

The behavior only seems to happen from one particular server (out of three in a cluster) at a time. The behavior reliably returns on new clusters, after replacing servers, etc. The errors will stop for 1 or 2 request cycles after restarting memcached.

It seems possible that the issue may be triggered by the actual content of the cache traffic.

[dormando]

That's certainly possible, which is why I was asking about getting client errors at first. It should be raising some kind of exception internally.

also when I said your wireshark is missing data, I don't mean the machine was retransmitting, I mean your tcpdump wasn't capturing everything so it wasn't fully correlating. It could've missed the RST in there.