Skip to content

Don't close connections on clean protocol-level SERVER_ERROR responses - #36

Merged
mapno merged 4 commits into
masterfrom
mapno/server-error-resumable-conns
Jul 28, 2026
Merged

Don't close connections on clean protocol-level SERVER_ERROR responses#36
mapno merged 4 commits into
masterfrom
mapno/server-error-resumable-conns

Conversation

@mapno

@mapno mapno commented Jul 24, 2026

Copy link
Copy Markdown

Summary

A SERVER_ERROR <msg>\r\n reply is a complete, well-formed protocol line: the server keeps the connection open and in sync (it even swallows the data block on storage errors). Closing our side causes needless reconnect churn, e.g. on object too large for cache or OOM bursts — the worst possible time to add TCP handshake load, since the server is already under memory pressure.

Why this is safe: memcached keeps the connection open on these errors

Verified against memcached 1.6.31 source. For the ASCII protocol (the only one this client speaks), every common SERVER_ERROR is written with out_string(), which transitions to conn_new_cmd — it does not close the connection. On storage errors the server additionally swallows the in-flight data block, so both sides stay in sync:

  • SERVER_ERROR object too large for cache / SERVER_ERROR out of memory storing object — replies, then swallows the data block and continues:
    proto_text.c#L2045-L2063
    out_of_memory(c, "SERVER_ERROR out of memory storing object");
    ...
    /* swallow the data line */
    conn_set_state(c, conn_swallow);
    c->sbytes = vlen;
  • SERVER_ERROR Out of memory during read (chunk alloc failure mid-read of a set payload) — same swallow pattern:
    memcached.c#L3240-L3250
  • SERVER_ERROR out of memory writing get response — get requests carry no data block, so the connection is already in sync; the server just replies and moves on:
    proto_text.c#L702-L713
  • out_of_memory() for ASCII clients is just out_string() (memcached.c#L1300-L1313); the conn_closing behavior lives only in the binary-protocol branch, which this client never uses.

Known exception: SERVER_ERROR out of memory reading request (failed read-buffer realloc on a huge multiget) sets close_after_write = true (memcached.c#L2494-L2496). If such a connection is pooled, the next request fails with EOF and the connection is then discarded as non-resumable — the same bounded, one-request cost as the pre-existing stale-pooled-connection races (server restarts, server-side idle timeouts) that the client already tolerates. Severe OOMs where memcached can't even allocate a response object close the connection without writing any error line, so the client sees a read error, not SERVER_ERROR, and the connection is dropped as before.

Note: error text for these responses changes from memcache: unexpected response line... to memcache: server error: <msg>.

mapno added 2 commits July 24, 2026 14:43
A SERVER_ERROR reply is a complete, well-formed protocol line that leaves
the connection at a clean boundary, so it is safe to return it to the
pool instead of closing it. Errors wrap the previously unused
ErrServerError and are matchable with errors.Is.
@mapno
mapno marked this pull request as ready for review July 27, 2026 10:15
@mapno
mapno requested a review from a team as a code owner July 27, 2026 10:15
narqo
narqo previously approved these changes Jul 27, 2026

@narqo narqo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me 🔥

Comment thread memcache/memcache.go Outdated
case bytes.Equal(line, resultNotFound):
return ErrCacheMiss
}
if err := serverErrorFromLine(line); err != nil {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you create a default: case for this switch like you did for another case above for consistency?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, addressed.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't seem like you updated it everywhere?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed now, sorry!

56quarters
56quarters previously approved these changes Jul 28, 2026
@mapno
mapno merged commit 9448343 into master Jul 28, 2026
14 checks passed
@mapno
mapno deleted the mapno/server-error-resumable-conns branch July 28, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants