Use exception metadata to determine if APIKeys are enabled

[kobelb]

Now that elastic/elasticsearch#52811 has been merged, we can now use the exception metadata to determine whether or not API Keys are enabled/disabled. This is presently being done by inspecting the exception message:

kibana/x-pack/plugins/security/server/routes/api_keys/privileges.ts

Line 35 in 2ec82d3

e.message.includes('api keys are not enabled')

Fleet would like to be able to do so as well, so if we can publicly expose a method to do so, that'd help @nchaulet.

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[legrego]

Blocks #49403, specifically the ask to expose a public method for consumers to check

[nchaulet]

It's also blocking us on fleet #54611

[legrego]

We have two requirements commingled in this issue:

• Determine if a request failed because API keys are disabled in ES
• Determine, in advance of making a request, if API keys are disabled in ES

The former is easily solved, but the latter poses additional challenges.
We have a couple of ways to determine if API keys are enabled in ES:

1. Try to make a request which relies on API Keys, and if we get an error back with this specific metadata attached, then we know API Keys are disabled.
2. Query the /_xpack/usage endpoint, and parse the response to determine if API Keys are enabled.

There are complications with both of these options:
Making the pre-flight request requires the kibana_system user to have the necessary privileges to perform a request that both tests API Keys, but does not do anything of importance. For example, we shouldn't attempt to create an API Key, since this key would be of no use. The kibana_system user could attempt to query for its own API Keys (there should never be any), but this user does not have sufficient privileges to do so today, and adding this privilege just to check for an exception seems...strange. Certainly doable though.

The /_xpack/usage response will return results from the currently elected master node, which might not reflect the configuration of the nodes that Kibana is configured to connect to. This can lead to false negatives/positives. The ES team mentioned that they could optionally return results from the local node, so I opened elastic/elasticsearch#53916 to request that. @albertzaharovits had reservations with this approach though, as he believes parsing the exception metadata would be more robust of a check.

[albertzaharovits]

@legrego I indeed think creating a dummy API key with a short lifetime is the most robust way to check if API Keys are enabled. It simply unequivocally validates that API keys work. My only concern is that this records a spurious audit event. Yet I'm not too concerned about it, because we're not in a position where we account, in any way, for all the API keys that the system users create. Here's an example of a dummy API key:

POST /_security/api_key
{
  “name”: “preflight-check-<timestamp>“,
  “expiration”: “1s”,
  “role_descriptors”: {
    “lame_role”: {
      “cluster”: [“none”],
      “index”: [
        {
          “names”: [“*”],
          “privileges”: [“none”]
        }
      ]
    }
  }
}

Another slightly related option is to try to invalidate a non existing API key. If API keys are not enabled, it will fail differently compared to when the API key does not exist. I think it's very unlikely we'll ever return a "missing API key" error code, when in fact API keys are not enabled. However, we can also go with the alternative of parsing the output of the /_xpack/usage endpoint, but we need to tests for it, so we make sure the feature flags never go stale.

I don't have a strong preference either way, just personal taste.