Y24-016 Validate that the TOL consumer can handle Traction API being unavailable

[sdjmchattie]

Description
Recent review of the code base for tol-lab-share would indicate that it probably doesn't handle Traction's API being unavailable very well. In this case, the message probably gets read from the broker, causes a failure while writing to Traction API, then the consumer will trigger a reconnection but will not necessarily wait up to 30 seconds before doing so. If this is happening, the consumer may rapid fire read from the queue and try to process the message very quickly and fill up logs.

We should simulate a fault on the Traction API in a dev environment to confirm the actual behaviour and use the TransientRabbitError exception to do time limited reconnections if the fault manifests.

Who the primary contacts are for this work
Stuart McHattie

[sdjmchattie]

We also need to handle HTTP Response Code 502 which comes back if nginx cannot forward our request to a valid Traction API.

[sdjmchattie]

Result of UAT testing:

Consumer went into 30 second loops while Traction API was unavailable and the messages stayed in the queue. Once Traction Service was re-enabled, the next loop consumed the messages and they ended up in Traction.