Retry once on stale-connection ConnectionError in Pool#with

[tycooon]

Summary

We hit this in production:

ClickhouseNative::ConnectionError: closed: Success
  at app/data/ch.rb:53 in CH#execute

The error originates in clickhouse-cpp's SocketInput::DoRead:

if (ret == 0) {
    throw std::system_error(getSocketErrorCode(), getErrorCategory(), "closed");
}

recv() == 0 means the peer cleanly closed the TCP connection (FIN). On a clean close errno isn't set, so strerror(0) == "Success" — hence the cryptic "closed: Success" message.

Common causes for a pooled connection going stale:

• CH server's idle_connection_timeout / tcp_keep_alive_timeout (defaults: 3600s / 290s).
• Load balancer / firewall idle-timeout drops (often 60–300s).
• Server restart / OOM kill.

In the prod log, the failed INSERT took only 8.388ms — far faster than any real round-trip; recv returned 0 immediately. The connection was already half-closed by the server before any meaningful work happened, so the INSERT didn't execute server-side and a retry on a fresh socket is safe.

Implementation

Pool#with already discards the dead client on error (discard_current_connection) so the next checkout would be fresh. We just need to actually do that next checkout instead of bubbling the error to callers. New behavior:

• ConnectionError → discard, retry once. If the retry also fails (e.g. server genuinely down), raise.
• All other errors (ServerError, ProtocolError, EncoderError, UnsupportedTypeError, …) → discard, raise immediately. No retry.

The retry is intentionally narrow — ConnectionError is what fires for stale-pool sockets where the server closed before we sent anything meaningful. Other error classes can fire mid-operation and re-running them could mask bugs or double-execute writes.

Tests

Three new specs in the existing Pool describe block:

• success on retry after a ConnectionError,
• raise after the second ConnectionError,
• no retry on ServerError.

Full suite: 63 examples, 0 failures. Rubocop clean.

Test plan

• bundle exec rspec
• bundle exec rubocop

🤖 Generated with Claude Code