Allow using JWT credentials to grant API keys.

[azasypkin]

Summary

In this PR we:

• Allow using JWT credentials to grant API keys
• Extend default value of elasticsearch.requestHeadersWhitelist to include both authorization and es-client-authentication to support JWT with required client authentication by default. See https://www.elastic.co/guide/en/elasticsearch/reference/8.11/jwt-auth-realm.html#jwt-realm-configuration
• Add API integration tests for both JWTs with client authentication and without it

NOTE: We're not gating this functionality with the config flag (xpack.security.authc.http.jwt.taggedRoutesOnly) as we did for the Serverless offering. It'd be a breaking change as we already implicitly support JWT authentication without client authentication, and to be honest, it's not really necessary anyway.

Testing

Refer to the Testing section in this PR description: #159117.

Or run already pre-configured Kibana functional test server:

1. node scripts/functional_tests_server.js --config x-pack/test/security_api_integration/api_keys.config.ts
2. Create a role mapping for JWT user:

curl -X POST --location "http://localhost:9220/_security/role_mapping/jwt" \
    -H "Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==" \
    -H "Accept: application/json" \
    -H "Content-Type: application/json" \
    -d "{
          \"roles\": [ \"superuser\" ],
          \"enabled\": true,
          \"rules\": { \"all\": [{\"field\" : { \"realm.name\" : \"jwt_with_secret\" }}] }
        }"

3. Send any Kibana API request with the following credentials:

curl -X POST --location "xxxx"
  -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2tpYmFuYS5lbGFzdGljLmNvL2p3dC8iLCJzdWIiOiJlbGFzdGljLWFnZW50IiwiYXVkIjoiZWxhc3RpY3NlYXJjaCIsIm5hbWUiOiJFbGFzdGljIEFnZW50IiwiaWF0Ijo5NDY2ODQ4MDAsImV4cCI6NDA3MDkwODgwMH0.P7RHKZlLskS5DfVRqoVO4ivoIq9rXl2-GW6hhC9NvTSkwphYivcjpTVcyENZvxTTvJJNqcyx6rF3T-7otTTIHBOZIMhZauc5dob-sqcN_mT2htqm3BpSdlJlz60TBq6diOtlNhV212gQCEJMPZj0MNj7kZRj_GsECrTaU7FU0A3HAzkbdx15vQJMKZiFbbQCVI7-X2J0bZzQKIWfMHD-VgHFwOe6nomT-jbYIXtCBDd6fNj1zTKRl-_uzjVqNK-h8YW1h6tE4xvZmXyHQ1-9yNKZIWC7iEaPkBLaBKQulLU5MvW3AtVDUhzm6--5H1J85JH5QhRrnKYRon7ZW5q1AQ'
  -H 'ES-Client-Authentication: SharedSecret my_super_secret'

....for example....
curl -X GET --location "http://localhost:5620/internal/security/me" \
    -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2tpYmFuYS5lbGFzdGljLmNvL2p3dC8iLCJzdWIiOiJlbGFzdGljLWFnZW50IiwiYXVkIjoiZWxhc3RpY3NlYXJjaCIsIm5hbWUiOiJFbGFzdGljIEFnZW50IiwiaWF0Ijo5NDY2ODQ4MDAsImV4cCI6NDA3MDkwODgwMH0.P7RHKZlLskS5DfVRqoVO4ivoIq9rXl2-GW6hhC9NvTSkwphYivcjpTVcyENZvxTTvJJNqcyx6rF3T-7otTTIHBOZIMhZauc5dob-sqcN_mT2htqm3BpSdlJlz60TBq6diOtlNhV212gQCEJMPZj0MNj7kZRj_GsECrTaU7FU0A3HAzkbdx15vQJMKZiFbbQCVI7-X2J0bZzQKIWfMHD-VgHFwOe6nomT-jbYIXtCBDd6fNj1zTKRl-_uzjVqNK-h8YW1h6tE4xvZmXyHQ1-9yNKZIWC7iEaPkBLaBKQulLU5MvW3AtVDUhzm6--5H1J85JH5QhRrnKYRon7ZW5q1AQ' \
    -H 'ES-Client-Authentication: SharedSecret my_super_secret' \
    -H "Accept: application/json"
----
{
  "username": "elastic-agent",
  "roles": [
    "superuser"
  ],
  "full_name": null,
  "email": null,
  "metadata": {
    "jwt_claim_sub": "elastic-agent",
    "jwt_token_type": "access_token",
    "jwt_claim_iss": "https://kibana.elastic.co/jwt/",
    "jwt_claim_name": "Elastic Agent",
    "jwt_claim_aud": [
      "elasticsearch"
    ]
  },
  "enabled": true,
  "authentication_realm": {
    "name": "jwt_with_secret",
    "type": "jwt"
  },
  "lookup_realm": {
    "name": "jwt_with_secret",
    "type": "jwt"
  },
  "authentication_type": "realm",
  "authentication_provider": {
    "type": "http",
    "name": "__http__"
  },
  "elastic_cloud_user": false
}

Fixes: #171522

---

Release note: The default value of the elasticsearch.requestHeadersWhitelist configuration option has been expanded to include the es-client-authentication HTTP header, in addition to authorization.

[azasypkin]

note: the time has come 🙂 See https://www.elastic.co/guide/en/elasticsearch/reference/8.11/jwt-auth-realm.html#jwt-realm-configuration

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[jbudz]

ftr_configs.yml

[legrego]

Tested locally, LGTM with one docs comment and one test question.

I recommend adding a Release Notes section to this PR description, which includes a note about the change to elasticsearch.requestHeadersWhitelist's default value.

[legrego]

🏅 thanks for testing both with and without shared secret

[legrego]

question should we include negative tests to ensure that end-to-end error handling works correctly if we:

• Fail to provide a ES-Client-Authentication header
• Provide a malformed ES-Client-Authentication header
• Provide a well-formed, but incorrect ES-Client-Authentication header
• Provide a valid ES-Client-Authentication header with a valid ES bearer token instead of a JWT

While this should be handled more or less automatically, we've benefited in the past from these types of "redundant" tests.

[azasypkin]

All these tests will fail during authentication stage, so we'll effectively be testing authentication, not grantApiKey API, but I agree that these tests will be useful, so I added them to a separate test suite. Done in e795826!

[azasypkin]

I recommend adding a Release Notes section to this PR description, which includes a note about the change to elasticsearch.requestHeadersWhitelist's default value.

Good idea, thanks. I'll add a release note.

[cauemarcondes]

ObsUX changes LGTM

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: e795826

Failed CI Steps

• Security Solution Cypress Tests #11
• Defend Workflows Cypress Tests #3
• Defend Workflows Cypress Tests #4

Test Failures

• [job] [logs] Security Solution Cypress Tests #11 / Detection ES|QL rules, creation creation creates an ES|QL rule and overrides its name creates an ES|QL rule and overrides its name

Metrics [docs]

Unknown metric groups

API count

id	before	after	diff
security	401	402	+1

History

• 💛 Build #181288 was flaky de9107b
• 💔 Build #181252 failed 70fb07a
• 💔 Build #181238 failed d762cea

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @azasypkin

[legrego]

LGTM!