RFC: SIGTERM handling and a gradual SV1 drain on tProxy shutdown

[average-gary]

TL;DR. The three binaries only listen for SIGINT. Every supervisor sends SIGTERM, so a stop is never a clean shutdown. Fixing that is a small cross-app change. On top of it, tProxy could drain SV1 miners gradually instead of dropping them all at once, opt-in. Six things are proposed as settled (below, object if you disagree). Seven are open, each with its own thread.

Open questions

#	Thread	Who should look
1	Exit status after a caught SIGTERM: 0, or 143 as if uncaught?	anyone who alerts on exit codes
2	Should tProxy drain at all, or is spreading reconnects the supervisor's job?	tProxy maintainers, operators
3	Send SV1 client.reconnect before closing each miner?	anyone with firmware experience
4	Should disconnect times be jittered within their slot, or evenly spaced?	implementers
5	Should /api/v1/health report "draining"?	monitoring (#577)
6	Pool and JDC analogues, given SV2 Reconnect already exists (#780, #909)	Pool / JDC maintainers
7	Remove stop_signal: SIGINT from docker/docker-compose.yml?	anyone running the compose stack

Please reply inside the thread rather than opening a new one.

Settled

1. SIGTERM and SIGINT both start the same graceful shutdown, in Pool, JDC and tProxy. Signal handling stays in main.rs, libraries keep using CancellationToken (noise_connection installs a process-wide SIGINT handler, breaking Ctrl+C for every consumer #688).
2. Default behaviour is otherwise unchanged. A drain is opt-in: drain_secs = 0 is today's shutdown().
3. The listener closes before the first disconnect, and the upstream stays up until the last miner is gone. Miners not yet drained keep submitting.
4. A second signal during a drain shuts down immediately.
5. The drain has to fit inside the supervisor's stop timeout. The app cannot enforce that; the docs say so next to the config key.
6. The drain never blocks fallback cleanup. Losing the upstream mid-drain ends the drain and shuts down.

What exists today

All three binaries await tokio::signal::ctrl_c() and nothing else: Pool, JDC, tProxy.

The image execs the app as PID 1. A PID 1 only receives signals it has installed handlers for (kill(2)), so SIGTERM from docker stop, systemd or an orchestrator is dropped and the process is SIGKILLed when the stop timeout expires (10s by default for docker stop). Measured on stratumv2/translator_sv2:main as built on 2026-10-06, with the upstream a TCP listener that never completes the handshake so the proxy is alive when the signal lands:

Container	docker stop -t 3	Exit
stock image, app is PID 1, SIGTERM	3.17s, killed at the timeout	137
stock image, --signal SIGINT (the compose workaround)	0.12s	1
stock image under --init, app is not PID 1, SIGTERM	0.12s, default action	143

The exit 1 on SIGINT is the existing bootstrap error path (CouldNotInitiateSystem), not the signal handling. The compose file works around the dropped SIGTERM with stop_signal: SIGINT on all three services; outside compose there is no workaround. Any deployment with a PID-1 shim (--init, tini) gets the default action instead: immediate exit 143, no shutdown().

Even a clean shutdown cancels one global token, so every SV1 miner behind a tProxy loses its connection in the same instant and reconnects in the same instant. One instance or several behind a load balancer, a restart is a reconnect storm.

Proposal

Part 1, cross-app, one PR. A shutdown_signal() in each main.rs that resolves on SIGINT or (Unix) SIGTERM and feeds the existing shutdown(). Behaviour change: SIGTERM goes from dropped-or-instant to graceful, bounded by the existing 5s runtime timeout. Thread 1 decides the exit status, thread 7 the compose lines.

Part 2, tProxy only, one PR, sub-issue of #31. A drain_secs config key. When set, a signal cancels a drain token: the SV1 listener closes at once; connected miners are disconnected one at a time, one per equal slot across the window less a small margin, while the upstream SV2 connection stays up; shutdown starts when the last miner leaves or the window elapses, whichever comes first. Second signal, upstream lost mid-drain, and signal during upstream bootstrap all shut down promptly. Paused-time unit tests cover slot spacing, early exit and the deadline; one integration test uses MockUpstream and raw TCP SV1 clients (the primitive #18 asks for). Threads 2 to 5 shape the details.

A prototype of both parts exists, enough to know the shape above is implementable and testable. Nothing about it is fixed; the threads decide what gets posted. Part 2 touches code that #909 rewrites, so it would rebase on whichever lands second.

Not in scope

SIGHUP as reload, SIGQUIT, and load balancing across replacement instances (a drain spreads reconnects in time, it does not steer them). The ERROR … ChannelErrorReceiver line each disconnect logs today is pre-existing and stays.

History

• 2026-08: noise_connection installs a process-wide SIGINT handler, breaking Ctrl+C for every consumer #688 / fix bad SIGINT patterns #689 move SIGINT handling out of noise_connection into main.rs. This proposal keeps it there.
• 2026-08: separate application-handle lifetime from runtime shutdown #781 notes shutdown() can miss a concurrent teardown. A drain makes that interleaving routine; Part 2 carries the tProxy-side reorder with a Refs, not a close.
• Add graceful shutdown API to roles #309 (closed) is where shutdown() came from. tProxy: handle Reconnect and ChannelEndpointChanged without panicking #780 / harden tProxy channel state, difficulty targets and upstream endpoint changes #909 cover the opposite direction, upstream-initiated SV2 Reconnect.

[plebhash]

I think the SIGTERM handling is a good improvement, but I don't really get the rationale for the tProxy sv1 draining part?

[plebhash]

#915

[average-gary]

To answer your question:

What it changes
Without a drain, N miners lose their connection in the same instant and reconnect in the same instant. With one, the same N reconnects happen over drain_secs. It does not make any single reconnect cheaper: each miner still loses its in-flight share and re-ramps vardiff. It removes the burst and makes the handoff gradual.

Why the burst matters
Every reconnect is a TCP accept, a mining.subscribe / authorize, a vardiff initialisation and, without aggregate_channels, an OpenExtendedMiningChannel upstream, all in the same second. Once #776 bounds admission (#320, settled item 7), a burst larger than the cap means refusals, and a refused SV1 miner fails over to its backup pool. A hashrate cliff followed by a cliff back also fires every "hashrate dropped" alert a pool operator has. With more than one translator behind a load balancer, draining one spreads its miners across the others over the window instead of dumping them on one target.

Why the orchestrator cannot do it
A supervisor can delay SIGTERM (preStop) and stop routing new connections. It cannot disconnect existing long-lived TCP sessions one at a time: when a backend is removed, the load balancer resets all of its flows at once. The only place that can hand miners off gradually is the process that owns the sockets.

Existing spec reconnction
3.3 already recognises maintenance-driven reconnection: an upstream "may occasionally request reconnection of its downstream peers (e.g. due to maintenance reasons)", via Reconnect (3.6.5), where an empty new_host means "reconnect to your present host". That exists for SV2 downstreams. 3.6.5 also says the message "should not be propagated downstream by intermediate proxies", so a translator's SV1 miners never see it. The SV1 side has no equivalent, which is why a translator restart is always a hard cut for everyone behind it. A drain is that 3.3 intent on the SV1 side; SV1 client.reconnect would be the closer analogue, and thread 3 is about whether firmware handles it well enough to use.

Scope
For a translator with a handful of miners this is irrelevant, and drain_secs = 0 keeps it that way. It is for the hosted-pool shape, where one translator fronts a large SV1 fleet and restarts are routine. If that is not a shape SRI wants tProxy to serve, that answers thread 2 and the drain stays out of tree.
/LLM

At scale we don't want a thndering herd problem with reconnects so rolling out a new translator with drain enabled avoids this and we don't get massive swings in hashrate all at once. Coupled with monitoring/metrics, we can identify a problem as it's happening and have time to react (i.e. pause fleet-wide rollout to minimize impact).

I should have split the two topics into separate RFCs, so thank you for correcting that.