fix(validator): prevent double-sign on stuck-duty cleanup race and handle pg pool errors (A-1313, A-1314)

[spalladino]

Hardens the validator HA signing path against an honest-validator double-sign and a Postgres-induced crash. Both issues live entirely in the HA signer.

Context

A-1313 (stuck-duty cleanup race → slashable double-sign). signWithProtection discarded the boolean from recordSuccess and returned the signature regardless. If the background cleanup loop deleted our SIGNING protection row while the (remote) signer was slow, the signature was broadcast with no protection record in place — so a later proposal for the same slot with different data could sign freely, i.e. slashable equivocation by an honest validator. The race is live today: SequencerClient.start() starts the validator client, which starts the cleanup loop. (The issue text claimed this was masked by #1111 never starting the cleanup loop; that is stale and no longer true.)

A-1314 (pg.Pool without an 'error' listener crashes HA validators). createHASigner built a new Pool(...) with no pool.on('error'). pg re-emits idle-client errors (e.g. a Postgres restart severing an idle connection) on the pool; with no listener Node escalates to uncaughtException and crashes the process — taking down every HA replica sharing the DB at once.

Approach

Four defense-in-depth fixes for A-1313 (part 1 alone closes the slashable broadcast; parts 2-3 remove the ways the race arises) plus the independent A-1314 liveness fix:

• Treat recordSuccess === false as a signing failure: throw a new SigningLockLostError and never return/broadcast the signature. The duty is not deleted, since we no longer own the row. This is the hard safety backstop: even under pathological timer behavior the worst case is a failed duty, never a broadcast double-sign.
• Bound each signer call with a hard timeout (executeTimeout), configured via signerCallTimeoutMs (default 30s) and clamped at construction to maxStuckDutiesAgeMs / 2 (default 144s / 2). Since signWithProtection is the only writer of SIGNING rows and every path through it is bounded by this clamped timeout, an in-flight signing always times out and releases its row well before stuck-duty cleanup could reclaim it — so cleanup can never race a live signing. A timeout takes the existing failure path (release the lock, then throw), and the orphaned signer promise is discarded.
• Add a request timeout (AbortSignal.timeout) to both Web3Signer key-store fetches, surfaced as a clear timeout error instead of hanging. The keystore is only constructed directly in tests today (production uses node-keystore's remote signer), so the timeout is a constructor option defaulting to 30s.
• Register a pool.on('error', ...) structured-log handler in createHASigner (covering both the created and injected pool), and add the same handler to the e2e HA fixture pools. Only message/code is logged, never the raw error.

API changes

• New env var VALIDATOR_SIGNER_CALL_TIMEOUT_MS (default 30000) controls the per-call signing timeout (effective value clamped to maxStuckDutiesAgeMs / 2), exposed as config field signerCallTimeoutMs.
• The confusable existing config field signingTimeoutMs (how long to wait for a peer node's in-progress signing, default 3s) is renamed to peerSigningTimeoutMs so each name says whose wait it bounds. Its env var VALIDATOR_HA_SIGNING_TIMEOUT_MS is already shipped and is kept unchanged; the derived CLI flag follows the field rename (--sequencer.signingTimeoutMs → --sequencer.peerSigningTimeoutMs).

Each fix follows red/green with unit tests in validator-ha-signer (errors, signer, slashing-protection service, LMDB + Postgres backends, factory) and validator-client (Web3Signer key store).

Fixes A-1313
Fixes A-1314