Don't close connections on clean protocol-level SERVER_ERROR responses - #36
Merged
Conversation
A SERVER_ERROR reply is a complete, well-formed protocol line that leaves the connection at a clean boundary, so it is safe to return it to the pool instead of closing it. Errors wrap the previously unused ErrServerError and are matchable with errors.Is.
mapno
marked this pull request as ready for review
July 27, 2026 10:15
56quarters
reviewed
Jul 27, 2026
| case bytes.Equal(line, resultNotFound): | ||
| return ErrCacheMiss | ||
| } | ||
| if err := serverErrorFromLine(line); err != nil { |
Collaborator
There was a problem hiding this comment.
Can you create a default: case for this switch like you did for another case above for consistency?
Collaborator
There was a problem hiding this comment.
Doesn't seem like you updated it everywhere?
56quarters
previously approved these changes
Jul 28, 2026
56quarters
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A
SERVER_ERROR <msg>\r\nreply is a complete, well-formed protocol line: the server keeps the connection open and in sync (it even swallows the data block on storage errors). Closing our side causes needless reconnect churn, e.g. onobject too large for cacheor OOM bursts — the worst possible time to add TCP handshake load, since the server is already under memory pressure.Why this is safe: memcached keeps the connection open on these errors
Verified against memcached 1.6.31 source. For the ASCII protocol (the only one this client speaks), every common
SERVER_ERRORis written without_string(), which transitions toconn_new_cmd— it does not close the connection. On storage errors the server additionally swallows the in-flight data block, so both sides stay in sync:SERVER_ERROR object too large for cache/SERVER_ERROR out of memory storing object— replies, then swallows the data block and continues:proto_text.c#L2045-L2063
SERVER_ERROR Out of memory during read(chunk alloc failure mid-read of a set payload) — same swallow pattern:memcached.c#L3240-L3250
SERVER_ERROR out of memory writing get response— get requests carry no data block, so the connection is already in sync; the server just replies and moves on:proto_text.c#L702-L713
out_of_memory()for ASCII clients is justout_string()(memcached.c#L1300-L1313); theconn_closingbehavior lives only in the binary-protocol branch, which this client never uses.Known exception:
SERVER_ERROR out of memory reading request(failed read-buffer realloc on a huge multiget) setsclose_after_write = true(memcached.c#L2494-L2496). If such a connection is pooled, the next request fails with EOF and the connection is then discarded as non-resumable — the same bounded, one-request cost as the pre-existing stale-pooled-connection races (server restarts, server-side idle timeouts) that the client already tolerates. Severe OOMs where memcached can't even allocate a response object close the connection without writing any error line, so the client sees a read error, notSERVER_ERROR, and the connection is dropped as before.Note: error text for these responses changes from
memcache: unexpected response line...tomemcache: server error: <msg>.