[Agent-Upgrade]: For Linux .tar deploy; Agent goes Unhealthy on upgrade with Endpoint Security

[amolnater-qasource]

Kibana version: 7.15.0 Snapshot Kibana Cloud environment

Host OS and Browser version: VSphere Ubuntu and MAC, All

Build details:

Build: 43937
Commit: d4c2d0476c622ba2314ab35c3439f1fad4dc0b34
Artifact Link: https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.14.1-linux-x86_64.tar.gz

Preconditions:

1. 7.15.0 Snapshot Kibana Cloud environment should be available.
2. 7.14.1 released Agent must be installed with Default policy having System and Endpoint Security integration.

Steps to reproduce:

1. Login to Kibana environment.
2. Trigger Agent upgrade from Fleet UI for 7.14.1 release agent.
3. Observe agent went Unhealthy after upgrade.

Debug level Logs:
logs.zip
endpoint-000000.zip

Note:

• This issue is observed on Vsphere machines name: linux qa-ubuntu20.04-desktop and mac qa-mac-bigsur-11.0.1-release-nosip-clone-base
• This issue is not observed on AWS-Ubuntu 20

Expected Result:
7.14.1 Ubuntu .tar agent should upgrade to 7.15.0 with Endpoint Security and should remain Healthy.

Screenshots:

[ph]

@amolnater-qasource Is this is still an issue ? fyi @jlind23

[amolnater-qasource]

Hi @ph
We have revalidated upgrading 8.1.0 agent upgrade to 8.2 Snapshot and found this issue still reproducible.

Observations:

Agent OS	Platform	Version	Upgrade Status to 8.2 Snapshot
Windows	Vsphere Windows 10	8.1.0 released	PASS
MAC	Vsphere BigSur 11	8.1.0 released	PASS
Linux	Vsphere Ubuntu 20	8.1.0 released	FAIL
Linux	Vsphere Centos 8	8.1.0 released	FAIL
Linux	AWS Ubuntu 20.04	8.1.0 released	PASS
Linux	AWS Ubuntu 20.04 ARM64	8.1.0 released	PASS

Logs for failed VSphere Linux Agent(v8.1.0):
elastic-agent-diagnostics-2022-03-10T07-06-56Z-00.zip

Screenshot:

Build Details:

BUILD: 50826
COMMIT: e694507960d61def6ebbde146a6cb296b1a74c96
Artifact Link(8.1.0): https://www.elastic.co/downloads/past-releases/elastic-agent-8-1-0

Please let us know if anything else is required from our end.
Thanks

[amolnater-qasource]

Hi Team pasting latest results from #245
We have revalidated this issue on latest 8.2 Snapshot by upgrading 8.1.3 Snapshot agents.
We had below observations:

• Windows and MAC agent upgraded successfully.
• Linux agent gets unhealthy and fails to upgrade.

Build details:
BUILD: 52012
COMMIT: 14a83300ba28df6c7f5318c4f3fe86f29369438d
Artifact Link: https://snapshots.elastic.co/8.1.3-0c4e7b42/downloads/beats/elastic-agent/elastic-agent-8.1.3-SNAPSHOT-linux-x86_64.tar.gz

Screenshots:

Ubuntu Agent Logs:
elastic-agent-diagnostics-2022-04-21T08-14-17Z-00.zip

Please let us know if anything else is required from our end.

Thanks

[amolnater-qasource]

Results on latest 8.3 Snapshot by upgrading 8.2 Snapshot agents.
We had below observations:

• Windows and MAC agent upgraded successfully.
• Linux agent gets unhealthy after triggering upgrade from Fleet UI.

Build details:
BUILD: 52217
COMMIT: 9430e41eb29640692d2d60ee93178726fdc356b4
Artifact Link: https://snapshots.elastic.co/8.2.0-a7e9513d/downloads/beats/elastic-agent/elastic-agent-8.2.0-SNAPSHOT-linux-x86_64.tar.gz

Screenshots:

Ubuntu Agent Logs:
elastic-agent-diagnostics-2022-04-20T08-28-47Z-00.zip

Thanks!

[amolnater-qasource]

FYI @blakerouse @jlind23

[jlind23]

@amolnater-qasource still waiting for Blake's fix on this.

[cachedout]

@jlind23 and @blakerouse Hi! Any chance of getting an ETA on this work? This issue is blocking some of the scenarios we're working through re: fleet scaling and it's quite important to us that we get a fix in place ASAP. Happy to jump on a call to discuss if needed. Thanks.

cc: @pjbertels

[jlind23]

@cachedout - @blakerouse added some more debugs logs through this particular PR: #308

[amolnater-qasource]

Hi Team
While validating kibana upgrades from released cloud builds 7.13.4>7.14.2>7.15.2>7.16.3>7.17.3>8.0.1>8.1.3>8.2.0.

We have observed that linux agent upgraded successfully from 7.13.4>7.14.2.
However from 7.15.2 none of the build was able to upgrade Linux .tar agent.

Thanks

[jlind23]

@amolnater-qasource it always happened with endpoint security enabled?

[amolnater-qasource]

Yes @jlind23 we had endpoint security integration added to the linux .tar agent.
Thanks

[jlind23]

And without endpoint it is working well right? If yes, could you please try an 8.3 upgrade as @blakerouse put some new logs in place that may help us.

[amolnater-qasource]

Hi @jlind23
We have revalidated this issue on 8.3 Snapshot by upgrading 8.2 linux agent.

• We are now successfully able to upgrade Linux agent on 8.3 Snapshot with and without endpoint security.

Build details:
BUILD: 52349
COMMIT: eaf47aeb1af9b935685fe12f5591be498999a9ab
Artifact Link: https://snapshots.elastic.co/8.2.0-8b4cfac3/downloads/beats/elastic-agent/elastic-agent-8.2.0-SNAPSHOT-linux-x86_64.tar.gz

Screenshot:

Logs:

elastic-agent-diagnostics-2022-04-28T07-06-31Z-00.zip

Could you please confirm if this will be merged to 8.2? or we can close this issue.
Thanks

[jlind23]

This really seems to be flaky and network related. Can we close this issue as it will never be fixed on 7.15?

[blakerouse]

This was just backported to 8.2 yesterday - #382

[amolnater-qasource]

Hi @blakerouse @jlind23
We have attempted to revalidate this on upgrading linux 8.1.3 released agent on latest 8.2 Snapshot.

• We are still not able to upgrade on 8.2 Snapshot.
• Agent goes unhealthy on triggering upgrade.
• We have tested this with and without endpoint security and got no success in upgrading linux agent.

Build details:
BUILD: 52057
COMMIT: a163de5804c03c590dba0e16fab866211b3e1f61
Artifact Link: https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.1.3-linux-x86_64.tar.gz

Screenshot:

Can we close this issue as it will never be fixed on 7.15?

@jlind23 we have kept this issue open to track linux agent upgrade failure on 7.17.x and 8.2 builds.
Thanks

[blakerouse]

The fix will never go back to 8.1 as there is not going to be another release. It should be in 8.2-SNAPSHOT so you could test that to 8.3-SNAPSHOT. It just landed in 7.17 today so next snapshot of 7.17 should fix the upgrade from 7.17-SNAPSHOT.

[Chadwiki]

I am seeing this fail on Windows, Linux, and MAC. Endpoint is not enabled for these policies.
Background: I upgrade my 8.1.3 cluster to 8.2.0 yesterday. most of my elastic-agents were not upgraded previously and still reside on the 8.1.1 version.

Windows

[elastic_agent][error] 2022-05-04T09:21:00-04:00 - message: Application: [2be01bb1-e5d0-4e43-9a7a-74fb679bb7ab]: State changed to FAILED: failed verification of agent binary: 2 errors occurred:
* hash for 'elastic-agent-8.2.0-windows-x86_64.zip' not found in 'C:\Program Files\Elastic\Agent\data\elastic-agent-7f30bb\downloads\elastic-agent-8.2.0-windows-x86_64.zip.sha512'
* open C:\Program Files\Elastic\Agent\data\elastic-agent-7f30bb\downloads\elastic-agent-8.2.0-windows-x86_64.zip.sha512: The system cannot find the file specified.

Linux

[elastic_agent][error] failed to dispatch actions, error: failed verification of agent binary: 2 errors occurred:
* fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-7f30bb/downloads/elastic-agent-8.2.0-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-7f30bb/downloads/elastic-agent-8.2.0-linux-x86_64.tar.gz.asc: no such file or directory
* open /opt/Elastic/Agent/data/elastic-agent-7f30bb/downloads/elastic-agent-8.2.0-linux-x86_64.tar.gz.sha512: no such file or directory

MAC

[elastic_agent][error] failed to dispatch actions, error: failed verification of agent binary: 2 errors occurred:
* fetching asc file from '/Library/Elastic/Agent/data/elastic-agent-7f30bb/downloads/elastic-agent-8.2.0-darwin-x86_64.tar.gz.asc': open /Library/Elastic/Agent/data/elastic-agent-7f30bb/downloads/elastic-agent-8.2.0-darwin-x86_64.tar.gz.asc: no such file or directory
* open /Library/Elastic/Agent/data/elastic-agent-7f30bb/downloads/elastic-agent-8.2.0-darwin-x86_64.tar.gz.sha512: no such file or directory

Another user has tested this with agents 8.1.2 on windows and centos 8 boxes. It has also failed.

[eric-ooi]

Looks like upgrading from 8.1.0 or 8.1.3 to 8.2.0 works (yay!). Unfortunately, 8.1.1 and 8.1.2 continue to fail (boo!). From what I can tell, it looks like @Chadwiki was upgrading from 8.1.1.

[joshdover]

Looks like upgrading from 8.1.0 or 8.1.3 to 8.2.0 works (yay!). Unfortunately, 8.1.1 and 8.1.2 continue to fail (boo!). From what I can tell, it looks like @Chadwiki was upgrading from 8.1.1.

My understanding is that we cannot fix this since the fix would need to be included in the binary you’re upgrading from, so upgrading from 8.1.1 and 8.1.2 will never succeed. These agents need to be unenrolled, uninstalled, and re-enrolled with a working version (8.1.3+).

[eric-ooi]

Yeah, I figured as much and saw the note in the 8.1.2 release notes confirming that. Had hoped that there might be some other fix that could accommodate the issue in later releases, but doesn't sound like it's possible.

Am now in the process of re-installing. I'm curious, is there a way to do in-place upgrades (without re-enrolling) outside of Fleet? Running "elastic-agent upgarde 8.2.0" on a Fleet-managed system didn't seem to work. Would be great if we could just upgrade via our MDM solution as opposed to clicking through Fleet.

[amolnater-qasource]

Hi @blakerouse @jlind23
We have tested agent upgrades on latest 8.3 Snapshot and had below observations:

• 8.2 released Windows and MAC agent upgraded to 8.3 Snapshot successfully.
• 8.2 released Linux .tar agent failed to upgrade and goes to Unhealthy state.
• 8.2 Snapshot Linux .tar agent upgraded successfully to 8.3 Snapshot.
• Further 7.17.4 Snapshot Windows, Linux and MAC agents upgraded successfully to 8.3 Snapshot.

Build details:
BUILD: 52477
COMMIT: 1c47b2349c4b8a406e90854957344e249e48c134

Successful 8.2 Linux agent artifact:

Please let us know if anything else is required from our end.

Thanks!

[WiegerElastic]

@peasead made a comment earlier today on how to fix this using a workaround.

[peasead]

Workaround

Note: the elastic-agent-xxxxxx path will differ for your installation

Windows

• Download elastic-agent-8.2.0-windows-x86_64.zip.asc and elastic-agent-8.2.0-windows-x86_64.zip.sha512 from the Elastic Agent download page
• Place the files in C:\Program Files\Elastic\Agent\data\elastic-agent-xxxxxx\downloads\
• Run the Fleet upgrade process

macOS Workaround

• Download elastic-agent-8.2.0-darwin-x86_64.tar.gz.sha512 and elastic-agent-8.2.0-darwin-x86_64.tar.gz.asc from the Elastic Agent download page
  • this worked for M1 and Intel boxes, but both M1 and Intel used the ...x86_64... files
  • (ensure the files are owned by root:wheel and have permissions of 644
• Place the files in /Library/Elastic/Agent/data/elastic-agent-xxxxxx/downloads
• Run the Fleet upgrade process

Linux

• Download elastic-agent-8.2.0-linux-x86_64.tar.gz.sha512 and elastic-agent-8.2.0-linux-x86_64.tar.gz.asc from the Elastic Agent download page
• Place the files in /opt/Elastic/Agent/data/elastic-agent-xxxxxx/downloads
  • tested on Debian and RPM distributions
  • ensure the files are owned by root:root and have permissions of 640
• Run the Fleet upgrade process

[jdixon-86]

I am having issues with this workaround. I download all three files as a test and make sure to set 640 permissions. After going to Fleet to upgrade it removes the files from the system and then fails because it can't find the files. This is going from 8.1.2 to 8.2.0

[peasead]

I am having issues with this workaround. I download all three files as a test and make sure to set 640 permissions. After going to Fleet to upgrade it removes the files from the system and then fails because it can't find the files. This is going from 8.1.2 to 8.2.0

Don't download all 3. Just the .asc and sha512.

[amolnater-qasource]

Hi @peasead
Thanks for the detailed steps for Agent upgrade workaround.
We have revalidated agent upgrades from 8.1.1 to 8.2.0 released version.

• We are now successfully able to upgrade Windows, MAC and Linux agents to latest version using this workaround.

Build details:
BUILD: 52005
COMMIT: 9a5003d8cf0062bf24ef64d6712b44823888cc03

Screenshots:

Thanks

[ghost]

Hi Team
We have tested agent upgrades on latest 7.14.4 Snapshot and had below observations:

• 7.16.3 and 7.17.3 released Windows and MAC agent upgraded to 7.17.4 Snapshot successfully.
• 7.16.3 and 7.17.3 released Linux .tar agent failed to upgrade and goes to Unhealthy state.

Build details:
BUILD: 46906
COMMIT: b239a118b5da2f4393153d8beea04ca5516468d4

Screenshot:

Please let us know if anything else is required from our end.

Thanks!

[ghost]

Hi Team
We have tested agent upgrades on latest 8.3.0-Snapshot build and had below observations:

• 8.2.0 released Windows and MAC agent upgraded to 8.3.0-Snapshot successfully.
• 8.2.0 released Linux agent failed to upgrade and goes to Unhealthy state.

Screenshot:

Build details:
VERSION: 8.3.0-SNAPSHOT
BUILD: 53487
COMMIT: b6dbc7712cc5c383bdd2f8392a2423f724068a2a

Logs:
Linux_logs.zip

Please let us know if anything else is required from our end.

Thanks!

[aleksmaus]

• 8.2.0 released Linux agent failed to upgrade and goes to Unhealthy state.

Looks like there is a problem fetching linux artifacts, 404:

fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc: call to 'https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc' returned unsuccessful status code: 404\n\n","ecs.version":"1.6.0"}

Looks like the linux build is not in elastic artifactory:

➜  wget https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz
--2022-06-23 06:51:51--  https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130
Connecting to artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2022-06-23 06:51:51 ERROR 404: Not Found.

@ph who should we ping on the release issues?

[joshdover]

@ph @cmacknz This looks related to the bug I was seeing upgrades via Horde with 404s on the windows artifact. It seems this is related to elastic/beats#32076 which is causing the DRAs to not be available?

[joshdover]

Actually that doesn't check out, because there are still builds available for beats. However, shouldn't snapshots be pulled from snapshots.elastic.co instead of artifacts.elastic.co? Here's the relevant parts of curl https://artifacts-api.elastic.co/v1/versions/8.3.0-SNAPSHOT:

              "elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz": {
                "url": "https://snapshots.elastic.co/8.3.0-02849d84/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz",
                "sha_url": "https://snapshots.elastic.co/8.3.0-02849d84/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512",
                "asc_url": "https://snapshots.elastic.co/8.3.0-02849d84/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc",
                "type": "tar",
                "architecture": "x86_64",
                "os": [
                  "linux"
                ]
              },

[ph]

@joshdover Didn't we override that value in kibana? the source_URI?

[joshdover]

@ph I checked and I don't see anywhere in our git history where Fleet specified a source_uri with a snapshots.elastic URL. We should get clarity on where we should pulling snapshots from for this use case.

[jlind23]

@joshdover @ph Agent will try to download artifacts from both, first from snapshots, then from the official artifact repo.

[joshdover]

@joshdover @ph Agent will try to download artifacts from both, first from snapshots, then from the official artifact repo.

Is this new? Horde doesn't have matching implementation so we would need to add that behavior. Also, it must be broken right now looking at the current behavior.

[jlind23]

@joshdover I don't think this is new. @blakerouse may be able to give more details here.

[joshdover]

It seems the problem I was seeing is that the upgrade from the UI on snapshot builds is telling agents to upgrade to 8.3.0 (which isn't released) from production instead of 8.3.0-SNAPSHOT. I believe this changed from 8.2 and is one unrelated issue that is happening.

[michalpristas]

looks like URI for releases changed from artifacts to snapshot
what we expect is
https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz
https://snapshots.elastic.co/8.3.0-86bc2a3e/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz
is what there actually is. this change from release is breaking for us.

but we should be able to fetch uri out of it, will verify

[michalpristas]

michal read the logs before playing smart.

change of uri is not a problem, we parse this out of the body.
the problem is this
fetching package failed: Get \"https://snapshots.elastic.co/8.3.0-47d97929/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n\t

uri was correctly parsed and it is functional but it took long time to download so it was cancelled due to request canceled while waiting for connection so maybe we have connectivity issues or firewall setup or DNS misconfig?

downloading of artifacts is same for all OS so i don't see a reasong why it should not work on linux machine specifically

[cachedout]

downloading of artifacts is same for all OS so i don't see a reasong why it should not work on linux machine specifically

Are these Horde instances or something else? I wonder if they're set up with dual-stack ipv4/ip6 and it's just not finding a route out. I have seen this before where ipv6 DNS requests will fail for this reason but ingress connectivity and non-DNS egress requests will be ok.

[aarju]

We are currently upgrading several hundred agents and documenting issues we see in this issue https://github.com/elastic/infosec/issues/10827

[amolnater-qasource]

Hi Team
We have revalidated upgrading 8.2.3 agent to 8.3.0 Kibana cloud production environment and had below observations:

• We are successfully able to upgrade 8.2.3>8.3.0 Windows, Linux and MAC agents.

Build details:
BUILD: 53518
COMMIT: 8a54c809495bc08782359073d9392f25c8eb6499
Artifact Link: https://www.elastic.co/downloads/past-releases/elastic-agent-8-2-3

Integrations:
System & Endpoint Security

Screenshots:

Thanks

[Guncixx]

Had the same problem, was not able to upgrade from 8.1.1 to 8.2 and also to 8.3, tried a workaround mentioned above, that way successfully upgraded agents to 8.2, after that I initiated upgrade to 8.3, it went through well too but after some time agent went offline. I then tried to restart an agent on one of the hosts, after that it failed and changed status to

Active: activating (auto-restart) (Result: exit-code) since Wed 2022-06-29 08:41:39 CDT; 1min 23s ago

from log files - elastic-agent.service: Failed with result 'exit-code'.

When I execute command elastic-agent status it shows

/usr/bin/elastic-agent: 2: exec: /opt/Elastic/Agent/elastic-agent: not found

then looking if such a directory exists I get

lrwxrwxrwx 1 root root 58 Jun 29 07:07 /opt/Elastic/Agent/elastic-agent -> /opt/Elastic/Agent/data/elastic-agent-1a0f39/elastic-agent

and lastly looking under /opt/Elastic/Agent/data/elastic-agent-1a0f39/ I see that there are just logs and vault directories

ls -l /opt/Elastic/Agent/data/elastic-agent-1a0f39/
total 8
drwx------ 3 root root 4096 Jun 29 08:00 logs
drwxr-x--- 2 root root 4096 Jun 29 07:17 vault

If I execute elastic-agent status in system where first installed version of an agent was 8.2 and then upgraded to 8.3 I get expected response:

Status: HEALTHY
Message: (no message)
Applications:
  * endpoint-security      (HEALTHY)
                           Protecting with policy {afdd95df-beed-42a7-8233-1bff0ad7ccb7}
  * filebeat_monitoring    (HEALTHY)
                           Running
  * metricbeat_monitoring  (HEALTHY)
                           Running

and listing elastic-agent-1a0f39 directory there are also more directories then on failing system.

sudo ls -l /Library/Elastic/Agent/data/elastic-agent-1a0f39
total 99400
drwxr-xr-x  23 root  wheel       736 Jun 29 12:18 downloads
-rwxr-xr-x   1 root  wheel  50870704 Jun 29 12:18 elastic-agent
drwxr-xr-x   5 root  wheel       160 Jun 29 12:20 install
drwx------   3 root  wheel        96 Jun 29 12:18 logs
drwxr-xr-x   3 root  wheel        96 Jun 29 12:18 run
-rw-------   1 root  wheel     16449 Jun 29 16:31 state.enc

[jlind23]

@Guncixx issue while upgrading from 8.1.1 to 8.2 is a known issue documented here: https://www.elastic.co/guide/en/fleet/8.1/fleet-troubleshooting.html

[Guncixx]

@Guncixx issue while upgrading from 8.1.1 to 8.2 is a known issue documented here: https://www.elastic.co/guide/en/fleet/8.1/fleet-troubleshooting.html

Yes, I know it, that's why I wrote that I was able to upgrade to 8.2 using workaround and agent seemed to be Healthy and running but then initiating upgrade from 8.2 to 8.3 agent goes offline and elastic-agent path is missing.

[nimarezainia]

@Guncixx could you please let us know what integrations you have in your policy? what you are experiencing may not be the exact same issue as described here.

it would be interesting to see the logs and see if a particular integration you have failed to download.

[nimarezainia]

Going to close this issue for now. The originator of the issue confirmed that it has been fixed already and if there are other issues during the upgrade process we need to look at those environments separate to what was configured here.

[ghost]

Hi Team,

We have revalidated upgrading 8.3.2 and 8.3.0 agents to 8.4.0 SNAPSHOT Kibana cloud production environment.

Build details:

Version: 8.4.0 SNAPSHOT
Build: 54427
Commit: 2de673f665783d7525f3cde8b6b266a471faef81

Below are the observations:

• We are successfully able to upgrade 8.3.2 > 8.4.0 Windows, Linux and MAC agents.
• We are successfully able to upgrade 8.3.0 > 8.4.0 Windows, Linux and MAC agents.

Integrations:
System & Endpoint Security

Screenshots & Recordings:

Agents.-.Fleet.-.Elastic.-.Google.Chrome.2022-07-13.12-06-28_.mp4

Upgrading one agent:

Upgrading more than one agents:

Hence, marking this ticket as QA Validated.

Thanks!

[Guncixx]

Wanted to ask if original issue with upgrading from agent version older than 8.2 will be fixed in coming elastic / agent versions or only solution is to reinstall agents manually?
As for issue I posted previously could not repeat it, I got it when upgrading agent with workaround solution to 8.2 and then directly to 8.3 but if I wait a bit before upgrading to 8.3 then everything worked.