Remote storage blocks shutdown if other end is having issues

[tomwilkie]

From @brian-brazil in #2847:

The remote write generally doesn't seem to like the other end having issues, such as being down for 5+ minutes. In this scenario, it also blocks shutdown.

I propose we have a timeout (1 min) to allow samples to flush, and otherwise exit uncleanly. WDYT?

[brian-brazil]

That sounds okay in principle. That probably should be flag controlled, and be shorter in line with the k8 shutdown. I'd say 20-25s, given k8 is 30s.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.