Unable to make Kubernetes requests: no such host

[roy-work]

We are seeing the following in our logs:

vault-1 vault 2023-05-12T20:56:43.996Z [DEBUG] auth.kubernetes.auth_kubernetes_689297a0: login unauthorized: err="Post \"https://kubernetes:443/apis/authentication.k8s.io/v1/tokenreviews\": dial tcp: lookup kubernetes on 10.0.128.10:53: no such host"

The error is correct; there is no such host. How/why is Vault attempting to connect to kubernetes:443? (This is the wrong host: the k8s API is kubernetes.default.svc, see the documentation.)

(Weirdly we have two Vaults, in two NSes, one which is working with this plugin, and one which isn't, and has this error. I am confused as to why they're behaving differently, as their configs are fairly similar.)

[roy-work]

Ugh, alright, so the root cause here is PEBKAC. Mea culpa.

But there are two thing in this plugin that make this really painful to figure that out:

1. Kubernetes API errors are not logged as errors #168 — this makes it impossible to see that auth is failing for problems unrelated to the presented credentials.
2. The configuration pane for the auth plugin doesn't show all of the settings. We were comparing a known good Vault cluster and the problem cluster's configuration settings for this auth plugin, and since they were identical, we discounted misconfiguration as a possibility. It's not until you attempt to edit the configuration settings that it becomes apparent that there are identical settings, and that in our case, that they're not set the same.