Receive: race condition in handler Close() when stopped early

[cincinnat]

Receiver hangs waiting for the HTTP Handler to shutdown if an error occurs before Handler is initialized. This might happen, for example, if the hashring is too small for a given replication factor.

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Fixes a race condition in receive.Handler.Close().

Verification

Before this change, the race condition could be reproduced with the script below. Eventually (in about 1-2 seconds) receiver would be able to start "successfully" and the script would hang until killed with SIGKILL. Health checks would not work since the health check listener is already down.

#!/bin/bash

trap exit SIGINT SIGTERM

while :; do
    thanos receive \
        --tsdb.path=/tmp/data \
        --label='receive="1"' \
        --receive.hashrings-algorithm=ketama \
        --receive.hashrings='[{"endpoints": ["127.0.0.1:19291"]}]' \
        --receive.replication-factor=2
done

[MichaHoffmann]

lgtm