Unhandled raw socket ECONNRESET before normal pg error handling crashes the process

[arvinds]

Hi, I ran into a worker crash loop using pg through pg-boss and pg.Pool on Node 20. It looks like there may be a connect-time raw socket error window before pg normal error handling is attached.

What I observed

• periodic worker crashes with uncaught ECONNRESET
• the crash signature consistently looked like a raw Socket with:
  • bytesRead: 0
  • bytesWritten: 20
  • no socket error listeners attached
• this looked like a brand new PostgreSQL connection attempt that reset before the usual pg client error path was ready

Why I think this belongs in pg rather than pg-boss

• the failure appears to happen during raw socket connect, before a client is fully established
• in my app, adding listener coverage at the pg-boss or pool-client level was not sufficient
• pg@8.11.5 lib/stream.js creates the socket with new net.Socket()
• pg@8.11.5 lib/connection.js calls stream.connect(...) before attaching stream.on("error", reportStreamError)

What I tried

• adding listener coverage after pool or client connect was not sufficient
• I also tried a local pg patch in this area, but I was not able to make it sufficient in my environment
• the mitigation that consistently stopped the crash loop was a worker-level guard that suppresses only raw-socket ECONNRESET when listenerCount("error") === 0

Relevant observed signature

• raw Socket
• ECONNRESET
• bytesRead: 0
• bytesWritten: 20

Environment

• Node 20
• pg 8.11.5
• usage via pg-boss / pg.Pool

Possible fix
One possible direction would be to protect the raw socket before connect() can emit an error, for example by either:

• attaching the socket error listener before calling stream.connect(...) in pg/lib/connection.js
• or ensuring sockets created in pg/lib/stream.js have temporary early error handling until the normal pg connection error path is attached

Questions

• is this a known connect-time race in pg?
• is there a recommended way to ensure the raw socket is protected before connect() can emit an error?
• would maintainers be open to a change here, or is there an existing fix or workaround I missed?

I can provide a proposed patch and more detailed logs if helpful.

[charmander]

An ECONNRESET-code error event should never be emitted synchronously by Socket#connect, which is what it would take for the order of connect/on('error', …) calls to matter. (No event should ever be emitted synchronously by Socket#connect, given that net.createConnection is a supported API.)

More detailed logs would be helpful, yes, as well as a minimal reproducer script. I’m guessing you might just be checking clients out of a pool without adding the required error listeners to the client and/or the pool, though. Both are necessary for now (pg ≤8.20.0):

import {Pool} from 'pg';

const pool = new Pool();

pool.on('connect', (client) => {
    client.on('error', (err) => {
        // ⋮
    });
});

pool.on('error', (err) => {
    // ⋮
});

[arvinds]

I agree a minimal repro would be ideal, but I have not been able to reproduce the specific production signature yet.

The main reason I opened this is the production evidence that the recurring fatal path was on the raw socket itself, not just at the pool/client layer. The suppression logs were consistently showing a raw Socket ECONNRESET with:

• socketErrorListeners: 0
• bytesWritten: 20
• bytesRead: 0

That is what made me think this might be earlier than the usual connected-client or pool-level error path.

I also tried adding listener coverage at the pool, client, and socket levels within pg-boss, and that was not sufficient to stop the recurring crash loop in production. The only mitigation that consistently stopped it was patching net.Socket.prototype.emit to intercept the ECONNRESET before it could propagate as an unhandled error event.

I agree there is a separate and documented need for pool-level error handling, the part I still don’t understand is the production 0-socket-listener signature on a brand new socket after 20 bytes were written.

Does that pattern suggest any known code path in pg where the raw socket could emit error before the normal connection-level handler is attached, or where a different socket instance could exist temporarily during connection setup?

[charmander]

Does that pattern suggest any known code path in pg where the raw socket could emit error before the normal connection-level handler is attached, or where a different socket instance could exist temporarily during connection setup?

No. Please do post relevant logs – and avoid passing any relevant information through an AI, if you’ve been doing that so far, since it’s much easier to diagnose problems based on direct evidence than based on a secondhand attempted diagnosis (especially an AI one). For example, why do you think it’s pg-related at all?

[arvinds]

Apologies for the AI-formatting. I don't have direct evidence this is related to PG but I had enough suspicions to raise this issue. Here are some traces from the error:

Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17)

Emitted ‘error’ event on Socket instance at:
at Socket.onerror (node:internal/streams/readable:1028:14)
at Socket.emit (node:events:524:28)
at emitErrorNT (node:internal/streams/destroy:169:8)
at emitErrorCloseNT (node:internal/streams/destroy:128:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -104, code: ‘ECONNRESET’, syscall: ‘read’ }

We were seeing crashes from this error frequently on our pg-boss workers (5-10 / day) and much less frequently on our web application (maybe 1/week). The workers run pg-boss which does things like polling our postgres instance through node-postgres and our web service uses pg-boss's client which uses node-postgres to enqueue jobs in postgres. Enqueueing happens much less frequently than the polling, which is one of the reasons I suspected the crash was coming from an attempted db interaction from pg-boss .

The worker also connects to datadog and statsig but those use http/udp, which is why I suspected the socket was to postgres. I was hypothesizing the error was related to pg-boss trying having stale postgres connections and crashing when trying to establish a new connection. (Though worth noting that playing with keepAlive, keepAliveInitialDelayMillis, connectionTimeoutMillis, idleTimeoutMillis didn't seem to resolve the issue). The error was consistently coming from a socket without error listeners writing a tiny bit of data and not reading anything.

I don't have any other evidence that this is PG related and I'm happy to look for more direct evidence if this doesn't seem pg-related.