Clair Health endpoint doesn't monitor dependency availability

[iancward]

Description of Problem:
Something caused DNS resolution inside the docker container to go south. In fact, I ran a busybox container on the same machine and it also can't resolve anything.
Clair logs have several DNS resolution errors that look like this:

{
    "Event": "could not branch Ubuntu repository",
    "Level": "error",
    "Location": "ubuntu.go:177",
    "Time": "2018-04-05 16:18:25.410475",
    "error": "exit status 3",
    "output": "bzr: ERROR: Connection error: Couldn't resolve host 'launchpad.net' [Errno -3] Try again\n"
}

Additionally, there are a few logs like this (host name, key id, and DNS IP have been redacted):

time="2018-04-05T16:19:20Z" level=error msg="Could not get public key from key server: Get https://myregistry.com/keys/services/quay/keys/0000000000000000000000000000000000000000000000000000000000000000: dial tcp: lookup myregistry.com on 10.0.0.1:53: read udp 172.17.0.2:42755->10.0.0.1:53: i/o timeout"

However, curling http://localhost:6061/health returns a 200 OK.

Expected Outcome:
If Clair can't talk with things it needs (e.g. remote repositories, database, quay enterprise), the health check should indicate that the system is unhealthy.

Actual Outcome:
curling http://localhost:6061/health returns a 200 OK.

Environment:

• Clair version/image: quay.io/coreos/clair-jwt:v2.0.1
• Clair client name/version: N/A
• Host OS: Ubuntu 16.04
• Kernel (e.g. uname -a): Linux 4.4.0-1052-aws Question: Does Clair support Mysql database? #61-Ubuntu SMP Mon Feb 12 23:05:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Kubernetes version (use kubectl version): N/A
• Helm version (use helm version): N/A
• Network/Firewall setup: N/A

[jzelinskie]

There's a tricky thing with checking for dependency availability: if the database becomes unhealthy, all of your Clair instances will be taken out of service and now you have a cascading failure.

For other projects that I've worked on, we've done tricky stuff like queried the Kubernetes or AWS API to determine if it was a network partition or the database is truly gone.

I think we should continue a discussion around what we'd like to see in the health endpoint and what makes sense from a failure domain.

[iancward]

I mean, even if you're able to expose the status of the previous fetch of upstream vulnerability data (Red Hat, Ubuntu, etc), that would be really helpful. That, and a date stamp of last fetch and next fetch attempt would also be helpful, as it would serve as a pointer for where to look in the logs.

[hdonnay]

We’re declaring bug bankruptcy as part of the release process for a new major version of Clair. Please open a ticket in our issue tracker if you feel this still needs to be addressed, and we'll triage as part of our v4 development process. Thanks!