Security Onion 3.1.0 — Elastic Agent Fleet Upgrade Failure (Air-Gapped Environment)

[bounc4]

Version

3.1.0

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

airgap

Hardware Specs

Meets minimum requirements

CPU

8

RAM

32

Storage for /

2TB

Storage for /nsm

1.8T

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Environment:

Air-gapped network
Security Onion 3.1.0 distributed architecture: 1 Manager, 1 Search Node, 2 Sensor Nodes
Upgraded from SO 3.0 to SO 3.1.0
Elastic Agent upgrading from 9.0.8 to 9.3.3
Windows endpoints managed via Fleet

Problem:
After upgrading Security Onion from 3.0 to 3.1.0, attempting to upgrade Elastic Agents on Windows endpoints from 9.0.8 to 9.3.3 via Fleet UI fails with:
"Upgrade failed: failed download of agent binary: unable to download package: package '...elastic-agent-9.3.3-windows-x86_64.zip' not found... fetching package failed: Get 'http://so-manager:8443/artifacts/beats/elastic-agent/elastic-agent-9.3.3-windows-x86_64.zip': EOF"
Investigation:

SO 3.1.0 no longer runs a dedicated artifact registry container. so-status confirmed no artifact registry container exists — this is a architectural change from SO 3.0.
Via docker inspect so-nginx, identified that /nsm/elastic-fleet/artifacts/ is the host-side path mounted into nginx and served at http://so-manager:8443/artifacts/.
Fleet Settings → Agent Binary Download confirmed default source is http://so-manager:8443/artifacts/ — correctly configured.
Mounted SO 3.1.0 ISO and located agent binaries at /nsm/soiso/fleet/elastic-agent_SO-9.3.3.tar.gz. Extracted and copied elastic-agent-9.3.3+build202604082258-windows-x86_64.zip and .sha512 to /nsm/elastic-fleet/artifacts/beats/elastic-agent/.
Confirmed nginx returns HTTP 200 for the file on port 8443 from the manager.
Confirmed Windows endpoint can resolve manager FQDN via DNS and TCP port 8443 is reachable (TcpTestSucceeded: True).
SO firewall hostgroup elastic_agent_endpoint confirmed wide open for the Windows subnet.
nginx access log shows no GET request for the 9.3.3 zip during upgrade attempts — the Windows agent is never initiating the download.

Current Status:
The 9.3.3 Windows binary is correctly staged and being served by nginx. All network connectivity checks pass. However, the Elastic Agent on the Windows endpoint (9.0.8) is not initiating the download request. The upgrade action shows 0% in Fleet UI and fails without any artifact download attempt appearing in nginx logs.
Question:
In SO 3.1.0 air-gapped environments, is there an additional configuration step required for Fleet to correctly instruct the 9.0.8 agent to fetch the upgrade binary from http://so-manager:8443/artifacts/? The filename includes a build suffix (+build202604082258) — is Fleet sending the correct filename to the agent, or is there a version/filename mapping that needs to be updated?
Relevant file location confirmed present:
/nsm/elastic-fleet/artifacts/beats/elastic-agent/elastic-agent-9.3.3+build202604082258-windows-x86_64.zipSonnet 4.6 Low

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[Zer0-cyber-web]

Same here. Non-air-gapped environment. :)
Version 3.0.0 had limitation of Elastic Agents version to 9.0.8, and it was working good. Version 3.1.0 now shows upgrade available up to 9.3.5, but only version "9.3.3+build202604082258" is downloaded and installed properly. And even after this proper upgrade - Elastic Fleet shows that there is "Upgrade available"

I guess it should be limited somehow on configurational level, or just described somewhere.

[bounc4]

I thought I saw that reflected on the server itself. I tested today selecting the agent 9.3.3 itself but it failed. Tomorrow I will try try version "9.3.3+build202604082258".

[Zer0-cyber-web]

The problem is that I tried to upgrade to 9.3.3 on few nodes - it faied. Now I am trying to restart upgrade to "9.3.3+build202604082258" - and it also fails now. Nodes that I upgraded directly to "9.3.3+build202604082258" - was upgraded properly.

[bounc4]

Thanks for heads up. I will definitely try tomorrow when I am back into the office. I will most likely just manually redeploy the agents to my test machines.

[Zer0-cyber-web]

and in addition - reinstall upgrades for "9.3.3+build202604082258" after failed upgrade to 9.3.3 version will continue properly when you just restart Elastic Agents services on affected nodes.

[bounc4]

Here is the latest update within my architecture.

The Manager, Search, and Sensor nodes are all running 9.3.3+build202604082258.

However, when attempting to upgrade an agent endpoint, the highest version available for selection is only 9.3.3, which has already failed during two previous upgrade attempts.

For this morning’s test, I attempted deployment using:

9.3.0+build202602051825

The result returned:

"The system cannot find the file specified. Fetching package failed."

My question is: how were you able to select and deploy 9.3.3+build202604082258 for the agent package itself?

From what I understand, the agent package physically resides within the ISO image. Did you manually extract the package from the ISO?

If so, do you happen to have a guide or documented process for doing this? I was actually working on attempting that extraction yesterday before creating this post.

[reyesj2]

Hello,

Security Onion sets up a self hosted artifact repo of the needed elastic agent components in /nsm/elastic-fleet/artifacts/beats/elastic-agent this is where agents connected to Elastic Fleet can pull packages from and upgrade. This directory does NOT include all Elastic Agent versions as that would require a lot more storage and including the additional variable of so many EA versions isn't something we could easily test for compatibility between versions. This is why we recommend that you keep elastic agents at the same version of Elasticsearch that is running (When SOUP runs we include the new packages for the given Elasticsearch version eg 8.18.8 - 9.0.8 - 9.3.3).

This last upgrade to Elasticsearch 9.3.3 was a bit different, as we were working through our internal testing on ES 9.3.3, Elastic had found and fixed a bug with the 9.3.3 Elastic Agent (https://www.elastic.co/docs/release-notes/elastic-agent#elastic-agent-release-notes-9.3.343build202604082258). We decided to follow that independent upgrade (vs waiting for ES9.3.4 and re-running through the dev/test cycle, delaying the release) and include the 9.3.3+build202604082258 Elastic Agent in Security Onion, so that users upgrading wouldn't encounter this bug.

As for different versions showing up in the Elastic Fleet UI we can't AFAIK modify that behavior via a configuration file, it would require modifying the Fleet source directly and could potentially introduce other unexpected behavior.

If you do not have '9.3.3+build...' as an option from the fleet UI for upgrading I would try just restarting Elastic Fleet sudo so-elastic-fleet restart --force