Offline agents not unenrolled after unenrollment timeout has expired

[juliaElastic]

On upgrading the stack, old version of Fleet Servers remain in Agent list as Offline, even long after 1 day (default unenrollment timeout of fleet servers in cloud).

This deployment has the issue for example: https://admin.found.no/deployments/0934b24bbdb34df2ae14912c9266a46a

In Fleet Server logs, seeing many of these errors:

failed to unenroll offline agents

fail InvalidateAPIKey: [400 Bad Request] {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"Failed to build [invalidate_api_key] after last required field arrived"}],"type":"x_content_parse_exception","reason":"Failed to build [invalidate_api_key] after last required field arrived","caused_by":{"type":"action_request_validation_exception","reason":"Validation Failed: 1: Field [ids] must not contain blank id, but got blank id at index position: [1];"}},"status":400}

Fail apiKey invalidate

• Version: 8.5.1 and 8.6
• Operating System: Cloud
• Discuss Forum URL:
• Steps to Reproduce:

Could reproduce this with local fleet server and kibana:

• create a Fleet Server policy with enrollment timeout: 60 seconds
• enroll a Fleet Server with docker
• stop the Fleet Server
• start the local Fleet Server and enroll it to kibana with a service token
• wait a few minutes until the unenroll logic kicks in
• the logs keep showing this error:

{"log.level":"error","ecs.version":"1.6.0","service.name":"fleet-server","ctx":"policy leader manager","policy_id":"fleet-server-policy","unenroller_uuid":"6a728042-0fab-4b9d-b899-4b578519f654","error.message":"fail InvalidateAPIKey: [400 Bad Request] {\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"Failed to build [invalidate_api_key] after last required field arrived\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"Failed to build [invalidate_api_key] after last required field arrived\",\"caused_by\":{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: Field [ids] must not contain blank id, but got blank ids at index positions: [1, 2];\"}},\"status\":400}","unenroll_timeout":60000,"@timestamp":"2022-11-17T14:00:33.474Z","message":"failed to unenroll offline agents"}

I think the reason is that the array of api keys contains blank ids that the invalidate api doesn't like:

"fleet.apikey.id":["49TbhYQBqTAlThoVXNcQ","",""],
"Validation Failed: 1: Field [ids] must not contain blank id, but got blank ids at index positions: [1, 2];\

[juliaElastic]

@elastic/elastic-agent-control-plane @michel-laterman Does this error ring a bell for you?

[juliaElastic]

I think the empty keys were introduced in this change: https://github.com/elastic/fleet-server/pull/1879/files#diff-093304d653ec6c08cbb650ebe6a6f76fc8e048b8ebfd06c6e49ea4f1c3906f80

Because the key ids are not checked for being empty before appending:
https://github.com/AndersonQ/fleet-server/blob/e7756bd3f3f7d532a5ade9a01526d10418a5f3d0/internal/pkg/model/ext.go#L53