Ironic health checks do not check against useful requests

[dankingtech]

What steps did you take and what happened:

By any method, cause the connection to the database to fail. Even though any requests to do actual work will fail, the deployment will be seen as live and ready because the health checks are only checking to see that the base URL is responding which it can do even if the internal connections are down. For instance, if one were to attempt to connect to http://127.0.0.1:6385/v1/nodes/ or other such endpoints, there may be an error but Kubernetes does not know about it.

What did you expect to happen:

Ideally, the liveness probe should detect that at least some other endpoint is successful which relies upon the database connection note that the Ironic instance is not healthy.

Anything else you would like to add:

Unfortunately, I have noticed that there are various occasions when Ironic, for various reasons, may fail to be able to connect to the database. In the past I have seen this caused by the database itself having issues as well as other issues related directly to the running instance of the Ironic API. In most cases, simply restarting Ironic has resolved the issue. Regardless, if the backend is unavailable, Ironic serves little utility. Therefore, I recommend changing the livenessProbe to check /v1/nodes/ rather than just /. The same may be true of the Inspector as well, by adding /v1/rules.

Environment:

• Baremetal Operator version: Observed on v0.3.0, but the check appears in main currently.

/kind bug

[dankingtech]

Actually, it looks like /v1/conductors/ would be a better endpoint than /v1/nodes/ as it would usually require less work than the later.

[dtantsur]

To repeat what I mentioned on IRC: any meaningful endpoint will require authentication. So we need to make sure the healtcheck script can use it.

[metal3-io-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[dtantsur]

/remove-lifecycle stale
/triage accepted

With metal3-io-bot/ironic-image@e44c4f7, we now have a path forward. We also need to finish the discussion around #1685 since it affects how we get the credentials.

[dtantsur]

/kind feature
/help

[metal3-io-bot]

@dtantsur:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/kind feature
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.