Monitoring integrations fail with 403 Forbidden on internal APIs after upgrade to 9.2.0 (Fleet + Elastic Agent)

[ReneMuetterlein]

We are running an air-gapped RHEL 9.6 system with a locally installed Elastic Stack using self-signed certificates. The system has no internet access and is used exclusively within an internal network.

We recently upgraded from version 9.1.4 to 9.2.0 for the entire stack (Elasticsearch, Kibana, Logstash, and Elastic Agent). After the upgrade, Fleet Server and the Elastic Agent were reinstalled and re-enrolled successfully, and all components report as HEALTHY — except for the monitoring integrations.

The elasticsearch/metrics, kibana/metrics, and http/metrics units consistently fail with 403 Forbidden errors, like so:

elastic-agent status
└─ elastic-agent
   ├─ elasticsearch/metrics-default
   │     └─ elasticsearch.ccr: Forbidden (/_nodes/_local/nodes)
   ├─ http/metrics-default
   │     └─ http.json: Forbidden (/api/task_manager/_background_task_utilization)
   └─ kibana/metrics-default
         └─ kibana.status: Forbidden (/api/status)

✅ What we've tested

• Created a dedicated user with:
  • remote_monitoring_agent
  • kibana_system
  • custom role with monitor, cluster:monitor/nodes/stats, cluster:monitor/nodes/info, and index access to .monitoring-*, .kibana*, .fleet-*, .logs-*, .metrics-*
• Verified access with this user using curl and browser: all API endpoints return valid responses.
• Replaced user/password auth with an explicit API key with matching role descriptors → same result.
• Reinstalled and re-enrolled the Agent (clean state).
• Updated monitoring integrations in Kibana and verified credentials are used correctly.

---

🧩 What we observe

• The Agent is fully connected and enrolled.
• The integrations immediately show DEGRADED, usually within 1–2 seconds.
• The API endpoints failing are:
  • /_nodes/_local/nodes
  • /_nodes/_local/stats/ingest
  • /api/task_manager/_background_task_utilization
  • /api/status

We assume these endpoints are protected by additional internal access controls in 9.2.0, which were either less strict or differently handled in 9.1.4.

---

✅ Important to note

• All of this worked flawlessly in 9.1.4
• The upgrade to 9.2.0 broke it without any config change on our part
• We could not find any documentation describing permission changes related to monitoring integrations in 9.2.0

---

💡 Request

Please clarify:

• Have the monitoring integrations changed how they authenticate or access internal APIs in 9.2.0?
• Is it now required to use service accounts or specific internal users (elastic, kibana_system, etc.)?
• Can we use a properly scoped API key instead — and if so, which exact privileges are needed?

Happy to provide sanitized logs or further details if needed.

Thanks in advance!

[cmacknz]

I've pinged the stack monitoring team internally on this as they are most likely to know what change has caused this.

[cmacknz]

The stack monitoring team wasn't able to find anything related to this looking at the Elasticsearch or Kibana logs, we'd need the agent configuration+logs (ideally diagnostics) to diagnose further.

[ReneMuetterlein]

Hello,
Please excuse the late response. I will try to provide you and the team with further information.

The Elastic server is operated exclusively internally and can only be accessed internally.
Therefore, Elastic's self-signed certificates are used. This is the reason why “insecure” flags are used everywhere.

1.) Download TAR-Archive from Website (https://www.elastic.co/de/downloads/elastic-agent)
2.) Login as root and extract the Archive to "/opt/elastic-agent-***"
3.) Remove old Agent-Files

systemctl stop elastic-agent
systemctl disable elastic-agent
rm /etc/systemd/system/elastic-agent.service
systemctl daemon-reexec
systemctl daemon-reload
rm -rf /opt/Elastic/Agent /etc/Elastic/Agent /etc/elastic-agent /var/lib/elastic-agent /var/log/elastic-agent /usr/bin/elastic-agent

4.) Reinstall new Agent

/opt/elastic-agent-***
./elastic-agent install \
  --fleet-server-es=https://localhost:9200 \
  --fleet-server-service-token=<token-from-kibana-for-new-fleet-server-agent> \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-port=8220 \
  --install-servers \
  --certificate-authorities=/etc/elasticsearch/certs/http_ca.crt \
  --fleet-server-es-insecure \
  --insecure

Listing from new Directory:

[root@***-***es1 ~]# cd /opt/Elastic/Agent/
[root@***-***es1 Agent]# ls -trla
total 163096
-rw-r----- 1 root root        0 Nov  5 08:38 .installed
drwxr-xr-x 3 root root     4096 Nov  5 08:38 ..
-rw-r----- 1 root root       41 Nov  5 08:38 .build_hash.txt
-rw-r----- 1 root root       41 Nov  5 08:38 .elastic-agent.active.commit
-rw-r----- 1 root root     3860 Nov  5 08:38 LICENSE.txt
-rwxr-x--- 1 root root       85 Nov  5 08:38 otelcol
-rw------- 1 root root      807 Nov  5 08:38 otel.yml
-rw-r----- 1 root root    16803 Nov  5 08:38 elastic-agent.reference.yml
-rw------- 1 root root    14137 Nov  5 08:38 elastic-agent.yml.2025-11-05T08-38-40.4273.bak
-rw-r----- 1 root root      315 Nov  5 08:38 README.md
drwxr-x--- 3 root root     4096 Nov  5 08:38 otel_samples
-rw-r----- 1 root root  5655825 Nov  5 08:38 NOTICE.txt
-rw------- 1 root root        7 Nov  5 08:38 .flavor
lrwxrwxrwx 1 root root       45 Nov  5 08:38 elastic-agent -> data/elastic-agent-9.2.0-e7bf31/elastic-agent
drwxr-x--- 2 root root     4096 Nov  5 08:38 vault
-rw------- 1 root root        0 Nov  5 08:38 fleet.enc.lock
-rw------- 1 root root        0 Nov  5 08:38 watcher.lock
drwxr-x--- 4 root root     4096 Nov  5 08:38 data
-rw------- 1 root root     1739 Nov  5 08:38 elastic-agent.yml
srwx------ 1 root root        0 Nov  7 13:47 elastic-agent.sock
-rw------- 1 root root     7535 Nov  7 13:47 fleet.enc
-rw------- 1 root root 20971942 Nov 16 20:12 elastic-agent-20251116-2.ndjson
-rw------- 1 root root 20972064 Nov 17 01:18 elastic-agent-20251116-3.ndjson
-rw------- 1 root root 20970879 Nov 17 06:25 elastic-agent-20251117.ndjson
-rw------- 1 root root 20972860 Nov 17 11:31 elastic-agent-20251117-1.ndjson
-rw------- 1 root root 20970661 Nov 17 16:37 elastic-agent-20251117-2.ndjson
-rw------- 1 root root 20972152 Nov 17 21:44 elastic-agent-20251117-3.ndjson
-rw------- 1 root root 20972401 Nov 18 02:50 elastic-agent-20251117-4.ndjson
drwxr-x--- 5 root root     4096 Nov 18 02:50 .
-rw------- 1 root root 14400500 Nov 18 06:20 elastic-agent-20251118.ndjson

[root@***-***es1 Agent]# cat elastic-agent.yml
# ================================ General =====================================
# Beats is configured under Fleet, you can define most settings
# from the Kibana UI. You can update this file to configure the settings that
# are not supported by Fleet.
fleet:
  enabled: true

# agent.download:
#   # source of the artifacts, requires elastic like structure and naming of the binaries
#   # e.g /windows-x86.zip
#   sourceURI: "https://artifacts.elastic.co/downloads/beats/"
#   # path to the directory containing downloaded packages
#   target_directory: "${path.data}/downloads"
#   # timeout for downloading package
#   timeout: 120s
#   # install_path describes the location of installed packages/programs. It is also used
#   # for reading program specifications.
#   install_path: "${path.data}/install"

# agent.process:
#   # minimal port number for spawned processes
#   min_port: 10000
#   # maximum port number for spawned processes
#   max_port: 30000
#   # timeout for creating new processes. when process is not successfully created by this timeout
#   # start operation is considered a failure
#   spawn_timeout: 30s

# agent.retry:
#   # enabled determines whether retry is possible. Default is false.
#   enabled: true
#   # retries_count specifies number of retries. Default is 3.
#   # Retry count of 1 means it will be retried one time after one failure.
#   retries_count: 3
#   # delay specifies delay in ms between retries. Default is 30s
#   delay: 30s
#   # max_delay specifies maximum delay in ms between retries. Default is 300s
#   max_delay: 5m
#   # Exponential determines whether delay is treated as exponential.
#   # With 30s delay and 3 retries: 30, 60, 120s
#   # Default is false
#   exponential: false

The agent has been assigned a role in Elasticsearch that grants it full access to the system. I created this role after the predefined role did not function and there was no change despite multiple reinstallations.

I have compiled all the data and placed it on our file server. You can access this data there until December 17, 2025.
https://sidas6.extranet.sachsen.de/public/download-shares/G8xxqBUqhbu8PoU1U5bQS7qHQqkL23pq