Client stalls for 180 seconds on 504: Gateway unavailable error

[jwulf]

Reported by @pedesen . It looks like the business error is treated as a gRPC channel failure. The gRPC Client then waits for a state transition to indicate a recovery. The channel, however, is in a READY state, so the next transition is to IDLE after 180 seconds.

See logs here: https://gist.github.com/pedesen/428e02d3fee8028ce0a80f5ef87b0b57