Add health client parent health

[rob05c]

Adds parent health to the health client.

This allows caches to log and markdown parent health based on whether the cache can directly request its parents, in addition to the existing Traffic Monitor health.

Intra-datacenter network issues can cause a child cache to be unable to get to its parent, even though externally both are accessible to the Traffic Monitor. When this happens, we implicitly rely on the ATS parent health and markdown system, which is reactive and typically leads to client timeouts before a markdown and retry of a different parent can be completed. Direct parent health provides faster proactive markdown during these intra-datacenter network events.

This also significantly refactors the health client. It was a single thread with a loop to reload config, refresh TO data, get TM health, and markdown. This makes all operations their own goroutine/microthread service, as the single-threaded work loop just wasn't feasible with the size of work for parent health polling.

It adds 3 health mechanisms: L4 health (a TCP syn-ack-rst), L7 health (a successful HTTP response), and a meta-parent poll which polls the parent's own health client parent health and uses a heuristic of unavailable parents on the parent.

All new parent health mechanisms default to disabled, and should be considered experimental.

Which Traffic Control components are affected by this PR?

• Traffic Control Health Client (tc-health-client)

What is the best way to verify this PR?

Enable parent health on the health client, observe logs and markdowns

If this is a bugfix, which Traffic Control versions contained the bug?

Not a bug fix

PR submission checklist

• This PR has tests
• This PR has documentation
• This PR has a CHANGELOG.md entry
• This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

[rob05c]

False positive, bounds are checked in GetAndHoldEphemeralPort.

[rob05c]

False positive, bounds are checked in GetAndHoldEphemeralPort

[jpappa200]

Looks good to me. One small typo in the README.md example tc-health-client.json has comma after last key value pair.
"parent-health-log-location": "/var/log/trafficcontrol/tc-health-client_parent-health.log", "health-methods": ["traffic-monitor", "parent-l4", "parent-l7", "parent-service"], "markdown-methods": ["traffic-monitor", "parent-l4", "parent-l7", "parent-service"], "hostname": "", }

[rob05c]

One small typo in the README.md

Fixed