Traffic Monitor start doesn't recover when Traffic Ops is unavailable

[smalenfant]

This Bug Report affects these Traffic Control components:

• Traffic Monitor

Current behavior:

When Traffic Ops is offline and Traffic Monitor is restarted. The following happens:

• Doesn't use a back CR-Config (by design, probably)
• It doesn't recover when Traffic Ops comes back
• Manual Traffic Monitor restart needs to occur

Logs:

ERROR: opsconfig.go:77: 2021-08-20T12:58:06.738891318Z: OpsConfigManager: Error getting Traffic Ops data: Error getting CRconfig from Traffic Ops: nil session
WARNING: monitorconfig.go:107: 2021-08-20T12:58:10.05145308Z: MonitorConfigPoller: skipping this iteration, Session is nil
WARNING: monitorconfig.go:107: 2021-08-20T12:58:15.051570293Z: MonitorConfigPoller: skipping this iteration, Session is nil
WARNING: monitorconfig.go:107: 2021-08-20T12:58:20.050974835Z: MonitorConfigPoller: skipping this iteration, Session is nil
WARNING: monitorconfig.go:107: 2021-08-20T12:58:25.051700515Z: MonitorConfigPoller: skipping this iteration, Session is nil
WARNING: monitorconfig.go:107: 2021-08-20T12:58:30.050987553Z: MonitorConfigPoller: skipping this iteration, Session is nil

Expected behavior:

Traffic Monitor should recover from Traffic Ops coming back

Steps to reproduce:

systemctl stop traffic_ops # Stop Traffic Ops
systemctl restart  traffic_monitor # Restart Traffic Monitor while Traffic Ops is unavailable
systemctl start traffic_ops # Start Traffic Ops

• Observe the Traffic Monitor logs

[ocket8888]

Doesn't use a back CR-Config (by design, probably)

Is that by design? I would think we'd want it to use the backup.

Also, is this a regression? Did it used to restart okay while TO was down?

[rawlinp]

Even if it's not an actual regression, it does seem like one, since this is one scenario I think we were supposed to have covered by introducing the snapshot backups. If it restarts when TO is unavailable, it should use the backup snapshots.