[Fleet]: Error on navigating to Stack Management>Security>API keys when 10k agents are installed.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.10.0 BC3
BUILD: 66261
COMMIT: 56348fa0ed0719679e24d6c58dc3dbee03928c4e

Host OS and Browser version: All, All

Preconditions:

1. 8.10.0 Kibana cloud environment should be available.
2. 10k agents are installed.

Steps to reproduce:

1. Navigate to Integrations>View Deployment>Create and manage API keys button.
2. Observe user is navigated to: Stack Management>Security>API keys.
3. Observe no API keys are generated and page fails to load with errors:

Error: 
Error
    at fetch_Fetch.fetchResponse (https://b3.kb.europe-west1.gcp.cloud.es.io:9243/66261/bundles/core/core.entry.js:1:290411)
    at async https://b3.kb.europe-west1.gcp.cloud.es.io:9243/66261/bundles/core/core.entry.js:1:288344
    at async https://b3.kb.europe-west1.gcp.cloud.es.io:9243/66261/bundles/core/core.entry.js:1:288301

Expected Result:
No error should be displayed on navigating to Stack Management>Security>API keys.

Screen Recording:

Browse.integrations.-.Integrations.-.Elastic.-.Google.Chrome.2023-09-04.17-11-28.mp4

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[amolnater-qasource]

FYI @jlind23

[jlind23]

@juliaElastic @kpollich have you ever seen something similar? Who own this .js file?

[juliaElastic]

This error is coming from here, owned by @elastic/kibana-security

[thomheymann]

The error indicates that the API request to load the list of API keys timed out.

Can you provide us with the debug logs for both Kibana and Elasticsearch during the timeframe when this error occurs?

[amolnater-qasource]

Hi @thomheymann

Thank you for looking into this.
We don't have access to the logs of the kibana and elasticsearch.

However, we have shared the environment details over slack with you.

Please let us know if anything else is required from our end.

Thanks!

[thomheymann]

I've done some more investigation on this with my team.

The API Keys response coming from ES is 160M and is too large to handle for Kibana since ES does not provide any pagination abilities on that endpoint.

The cluster contains 40,076 API Keys:

cat api_keys.json | jq '.api_keys | length'

40076

Analysing the response I can see that nearly all of them have been created by Fleet:

cat api_keys.json | jq '.api_keys | map(.username) | group_by(.) | map({username: .[0], count: . | length})'

[
  {
    "username": "1688146396",
    "count": 16
  },
  {
    "username": "cloud-internal-enterprise_search-server",
    "count": 1
  },
  {
    "username": "elastic",
    "count": 1
  },
  {
    "username": "elastic/fleet-server",
    "count": 40044
  },
  {
    "username": "found-internal-kibana4-server",
    "count": 14
  }
]

@elastic/fleet Can you please investigate why Fleet has created 40,044 API Keys on this cluster? It looks like something has gone wrong here and it would be good to understand the root cause.

[amolnater-qasource]

Hi Team,

We have attempted to reproduce this issue at our end on 8.11.0 SNAPSHOT and we have observed that this issue is consistently reproducible on installing 10k agents.

Observations:

• Error on navigating to Stack Management>Security>API keys when 10k agents are installed.

Screenshot:

Build details:
VERSION: 8.11.0 SNAPSHOT
BUILD: 67039
COMMIT: d78ecfe

We have updated issue summary and description as per the current observations.

Please let us know if anything else is required from our end.
Thanks!!