fix(mysql): retry SSLZeroReturnError peer-close on connect - #70542
Conversation
The transient-connect-drop detection only matched the
`[SSL: UNEXPECTED_EOF_WHILE_READING]` rendering of an SSL peer-close. Newer
OpenSSL/Python raise `ssl.SSLZeroReturnError` ("TLS/SSL connection has been
closed (EOF)") for the same condition, which pymysql wraps as the same 2003
connect failure. That message wasn't recognized as transient, so the in-process
retry re-raised it immediately and it hit the non-retryable "Can't connect to
MySQL server on" classifier — giving up the sync on a momentary TLS drop.
Match the new token too so both renderings are retried in-process.
Generated-By: PostHog Code
Task-Id: e6712d85-56fe-4698-a9e1-d8eee288c2f0
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
|
Reviews (1): Last reviewed commit: "fix(mysql): retry SSLZeroReturnError pee..." | Re-trigger Greptile |
![stamphog[bot]](https://avatars.githubusercontent.com/in/3000332?s=60&v=4){kind=small}
There was a problem hiding this comment.
Small, well-tested widening of a transient-error string match in the MySQL warehouse connector; author is on the owning team with strong familiarity, no risky-territory concerns, no unresolved review comments.
- Author wrote 100% of the modified lines and has 13 merged PRs in these paths (familiarity STRONG).
- 👍 on the PR from greptile-apps[bot], hex-security-app[bot].
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| size | ✓ | 19L, 1F substantive, 45L/2F incl. docs/generated/snapshots — within ceiling |
| tier | ✓ | T1-agent / T1b-small (45L, 2F, single-area, fix) |
| stamphog 2.0.0b3 | .stamphog/policy.yml @ 0a0a844 · reviewed head 0013d16 |
1 participant
Problem
Error tracking surfaced an
OperationalError(2003, "Can't connect to MySQL server on '<host>' (TLS/SSL connection has been closed (EOF) (_ssl.c:1032))")from the MySQL warehouse import source (issue). It was raised from_connect_with_transient_retrywhile reconnecting mid-sync.The underlying cause is
ssl.SSLZeroReturnError— the TLS peer closed the connection. This is exactly the transient class the source already retries in-process for the[SSL: UNEXPECTED_EOF_WHILE_READING]rendering (an overloaded server, a proxy/load-balancer idle cull, a failover, a momentary network blip). But_is_transient_connect_droponly matched that one token, so theSSLZeroReturnErrorrendering fell through: the retry re-raised on the first attempt, and the error then hit the non-retryable"Can't connect to MySQL server on"classifier. The sync gave up on a blip a fresh attempt would have recovered.Changes
Recognize the second rendering of the SSL peer-close ("TLS/SSL connection has been closed (EOF)") as a transient connect drop, so both are retried in-process by
_connect_with_transient_retry. Match the stable phrase only, not the volatile_ssl.c:<line>suffix, and keep it scoped to the 2003 branch so the generic "can't connect" config errors (wrong host/port, firewall) stay non-retryable.This does not add anything to
NonRetryableErrorsand does not touch the global retry policy — the fix keeps a transient error retryable, which it should have been all along.How did you test this code?
Extended
TestIsTransientConnectDrop— parameterized the existing SSL peer-close case to cover both theUNEXPECTED_EOF_WHILE_READINGand the newTLS/SSL connection has been closed (EOF)renderings. Guards against re-narrowing the matcher so the new rendering silently becomes non-retryable again; no existing case exercised it.Ran locally:
Also ran
ruff checkandruff format --checkon both changed files — clean.🤖 Agent context
Autonomy: Fully autonomous
Triaged from an error-tracking webhook by Claude (Claude Code). Pulled the issue and a representative event via the PostHog MCP error-tracking tools, read the full stack trace to confirm the failure originates in
mysql.py(_connect_with_transient_retry), and read the source's existing transient/non-retryable classification.Decision: fixable bug, not a user/upstream error. The message is a transient infrastructure drop, and the source already treats the sibling SSL-EOF token as retryable — this rendering just wasn't matched. So the fix is to widen the transient matcher, not to add a
NonRetryableErrorsentry. Invoked/writing-testsbefore touching the test file; chose to parameterize the existing test rather than add a near-duplicate.Created with PostHog Code