[SIGNER BUG] - Testnet signer just hung without any response causing a stall in testnet

[bgridley]

Describe the bug
The Hiro testnet signer process became stalled, failing to respond to metrics queries, process new blocks, or produce any logs. While the process remained active, it was non-functional. The issue was resolved only after manually restarting the signer.

Steps To Reproduce
Manual reproduction of this issue has not been possible. This appears to be an intermittent and rare event.

Expected behavior
The signer process should remain responsive and functional at all times. If a critical error or unrecoverable state is reached, the process should crash gracefully or log a critical error to facilitate automated restarts and debugging. The observed silent failure makes it impossible to detect and recover without manual intervention.

Environment (please complete the following information):

• Hiro testnet signer setup

Additional context

The stall occurred after the following sequence of log messages. No further logs were generated until the signer was manually restarted.

08/14/2025, 03:23:31.779 PM | message=Cycle #88 Signer #2: Block proposal invalid
08/14/2025, 03:23:31.779 PM | message=Miner block proposal has consensus hash that is neither the current or last sortition. Considering invalid.
08/14/2025, 03:23:31.775 PM | message=Miner block proposal has consensus hash that is neither the current or last sortition. Resetting view.
08/14/2025, 03:23:31.774 PM | message=Cycle #88 Signer #2: received a block proposal for a new block.
08/14/2025, 03:23:29.636 PM | message=Last accepted block has timed out
08/14/2025, 03:23:09.623 PM | message=Last accepted block has timed out
08/14/2025, 03:22:49.610 PM | message=Last accepted block has timed out
08/14/2025, 03:22:29.597 PM | message=Last accepted block has timed out
08/14/2025, 03:22:09.584 PM | message=Last accepted block has timed out

[jferrant]

This seems to be due to the signer not properly setting socket timeouts in libsigner/src/http.rs. decode_http_body seems to call read_to_end which appears to hang forever.

2025-08-14, 08:05:02.032 PM	
Signer failed to request latest sortition
	
2025-08-14, 08:04:42.016 PM	Cycle #88 Signer #2: Failed to send block response to stacker-db: RPCError(Deserialize("Failed to decode HTTP headers"))

2025-08-14, 08:04:42.016 PM	
Exceeded max retries during request: Deserialize("Failed to decode HTTP headers")
	
2025-08-14, 02:51:02.019 PM	
Cycle #88 Signer #2: Broadcasting a block response to stacks node: Accepted(BlockAccepted { signer_signature_hash: c89b657bd7029c7263323daa572cde24558b8e46612af565e164351f1cf43c69, signature: 00753f9a1832b6e156c22171ff9980663ddefe424d7ebbbb01232fccd3e2d1deaa4777825929e38cc970543a60aa703e46e976f3c4ca46b4313d47b7ec0a8206ae, metadata: SignerMessageMetadata { server_version: "stacks-signer 3.2.0.0.0.0 (release/3.2.0.0.0:6c76963+, release build, linux [x86_64])" }, response_data: BlockResponseData { version: 3, tenure_extend_timestamp: 1755208327, reject_reason: NotRejected, unknown_bytes: [] } }) 

This does make me think that we are also not gracefully handling cases where for some reason we don't see connection closed from the node (which is required to cause the hang I think). But at the very least a timeout on the tcp connection itself would have prevented this issue.

[jcnelson]

Tracking the fix here: #6400, #6401