Known Issue: Failed upgrades leave Elastic Agent stuck until restart

[pkoutsovasilis]

Description:

A regression introduced in #9634 can cause Elastic Agent upgrades to get stuck if an upgrade attempt fails early. This happens because the coordinator’s overrideState remains set, leaving the agent in a state that appears to be upgrading.

Affected Versions:

• Versions 8.18.7, 9.0.7
• Fixed in #9992, will be included in 9.1.4 (raised blocker), 8.19.4 (raised blocker), 9.0.8 (next patch release), 8.18.7 (next patch release)

Conditions:

This issue is triggered if the upgrade fails during one of the early checks inside Coordinator.Upgrade, for example:

• The agent is not upgradeable
• Capabilities check denies the upgrade
• Most commonly: when Elastic Agent is tamper-protected and Endpoint returns an error during action proxying — for example because the upgrade action signature is invalid, missing, or fails verification. This causes the coordinator’s override state to be stuck.

Symptoms:

• Upgrade remains stuck (Fleet shows the upgrade action in progress)
• No further upgrade attempts succeed
• elastic-agent status shows an override state indicating upgrade

Workaround:

Restarting the Elastic Agent clears the coordinator’s overrideState and allows new upgrade attempts to proceed.

Resolution:

This bug is fixed by #9992, which ensures that the coordinator clears its override state whenever an early failure occurs.

Action:

• Please add a Known Issue entry to the release notes for affected versions.
• Include the workaround (manual restart of Elastic Agent).

cc @lucabelluccini for visibility so that Support is aware
cc @ebeahan @cmacknz

[cmacknz]

Thanks for this, just a quick clarification:

Endpoint action proxying returns an error; this is the most probable cause and we have already user reports with that reason

What "Endpoint action proxying returns an error" means in terms users are likely to understand is "when Elastic Defend is tamper protected and the upgrade action signature check to allow the upgrade fails for any reason"

[lucabelluccini]

Thank you for the update to the docs already filed.

I've created a known issue https://support.elastic.co/knowledge/8bf72056 - feel free to review

Content is mostly what you've written with some edits (posting only the edits):

Environment

• Elastic Agent 8.18.7, 9.0.7 only
  • 8.19.x and 9.1.x are not affected
• Typical but not mandatory for this problem to occur: using Elastic Defend integration with tamper protection enabled in the policy of the affected Elastic Agent

Fix

The fix will be in versions 9.1.4, 8.19.4, 9.0.8, 8.18.8

[karenzone]

PRs

• 8.18.7: Doc: Add Known issue for 8.18.7: Agent stuck on failed upgrade ingest-docs#1856
• 9.0.7: Doc: Add known issue 9.0.7: Agent stuck on failed upgrade elastic-agent#10042