TM sometimes sets cachegroups to unavailable if all caches are ONLINE instead of REPORTED

[rawlinp]

I'm submitting a ...

• bug report

Traffic Control components affected ...

• Traffic Monitor

Current behavior:

TM sometimes appears to add cachegroups to the disabledLocations array in its /publish/CrStates API when all caches are set to ONLINE. In the following example, all caches are ONLINE, yet disabledLocations contain cachegroups:

{
    "caches": {
        "cache1": {
            "ipv4Available": true,
            "ipv6Available": false,
            "isAvailable": true
        },
        "cache2": {
            "ipv4Available": true,
            "ipv6Available": false,
            "isAvailable": true
        },
        "cache3": {
            "ipv4Available": true,
            "ipv6Available": false,
            "isAvailable": true
        },
        "cache4": {
            "ipv4Available": true,
            "ipv6Available": false,
            "isAvailable": true
        }
    },
    "deliveryServices": {
        "ds1-red": {
            "disabledLocations": [],
            "isAvailable": false
        },
        "ds2": {
            "disabledLocations": [
                "cg1"
            ],
            "isAvailable": true
        },
        "ds3": {
            "disabledLocations": [
                "cg2"
            ],
            "isAvailable": true
        },
        "ds4-red": {
            "disabledLocations": [],
            "isAvailable": false
        },
        "ds5": {
            "disabledLocations": [
                "cg1"
            ],
            "isAvailable": true
        },
        "ds6": {
            "disabledLocations": [
                "cg2"
            ],
            "isAvailable": true
        }
    }
}

Expected behavior:

Since all the caches are ONLINE, the above example should have no cachegroups in disabledLocations. When a cachegroup is in disabledLocations, that essentially tells TR that all caches in that cachegroup are unavailable.

Minimal reproduction of the problem with instructions:

Set all caches to ONLINE in a CDN with one or more delivery services, snapshot, and request TM's /publish/CrStates API.

Anything else:

I've tried to reproduce this in CiaB, but in CiaB TM doesn't populate the disabledLocations array the same as it does in the environment in which this was observed. Honestly, as this is basically just a premature optimization for Traffic Router, we might just want to change TM to never populate the disabledLocations. Traffic Router will still quickly find that all caches are unavailable in a given cachegroup and try the next closest. Checking disabledLocations probably only saves TR a handful of nanoseconds when routing a request (and only if entire cachegroups are unavailable), so I think we'd be fine if TM never populated it.