Traffic Stats does not failover to another TM when one stops responding

[mabrodis]

I'm submitting a ...

• bug report

Traffic Control components affected ...

• Traffic Stats

Current behavior:

Traffic Stats does not gracefully handle the situation when a Traffic Monitor it is using stops responding. What happens currently is if the Traffic Monitor that TS is getting data from goes away then TS just will stop collecting stats and will not write out any stats/data/metrics to an output platform (e.g. InfluxDB).

Expected behavior:

Traffic Stats ideally should have a list of Online TMs that it can get data from and if one does not respond simply move on and use another one and continue to collect stats and output stats to whatever the desired output platform is.

Minimal reproduction of the problem with instructions:

1. Have TS running and have two Online TMs
2. Stop TM1 and if TS does not log an error then start that one up and stop TM2.
3. Once you stop the TM that TS is connecting to you will cease to see messages like this in the traffic_stats.log:
   [INFO] 2021-06-15 21:03:40 Collected XXX deliveryservice stats values for cdn @ 1623791020
   [INFO] 2021-06-15 21:03:40 Collected XXXX cache stats values for cdn @ 1623791020
4. You won't see those messages because TS is not collecting stats because the TM it was using is not responding and it does not try the other TM, it appears to be stuck trying to use the one that is no longer responding.

Versions in the test:
TS: 3.0
TM: 5.2

Anything else: