[Fleet] Improve status reporting for Agents

[nchaulet]

Description

Currently we report a very basic status during checkin.
To allow us to give users more details on the status of their agents we want to send more complete policy status (Format is defined here elastic/kibana#82298)

The status will be send during agent checkin:

• This will allow Fleet Kibana (and Fleet Server in the future) to update the agent saved object and allow user to search per agent status, input status
• A future coordinator process to take actions after a status change

Questions

How we persist status in ES?

• I was thinking of updating the current agent SO with the actual status

Open question should the agent also send that data to ES directly?
Is this already the case if status change are in the agent logs? if yes are this log data will be searchable

Pro:

• Allow to have agent status for agent non managed per Fleet
• Allow to have historical data of agent status.
• Have agent status when the connection Agent -> fleet is broken

@blakerouse @ruflin I am curious to have your thoughts here on how this can work with the future Fleet Server too

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[blakerouse]

I think with the new Fleet Server architecture we should have the Elastic Agent send the status information directly to Fleet Server and from their Fleet Server should append the status information to an index. This removes the need to Elastic Agent to have to write directly to elasticsearch, and it ensures that the Fleet Server has the updated information to perform required adjustments based on the coordinator work that the Fleet Server will need to perform in the future.

[nchaulet]

I think with the new Fleet Server architecture we should have the Elastic Agent send the status information directly to Fleet Server and from their Fleet Server should append the status information to an index

I agree that we should send the data to Fleet server, so the coordinator can take advantage of that, I am just wondering if the agent should also persist is status (maybe in the logs) for agent observability purpose it will allow to have agent status for standalone agents too.

If we go to the direction where Fleet Server write the agent status, we should probably implement the same in Kibana, I think we can have an index similar to uptime synthetics-agent-status-default, (this could be defined in an elastic agent package ?) Right now is kind of tricky to write from the Kibana checkin route because we do not have the permission to write to ES unless we use the fleet_enroll admin user.

[ruflin]

The status should be reported the Fleet Server, but I still think Elastic Agent should persist it to ES directly as log or similar. This means, status reporting also works in standalone case for example.

[michalpristas]

Agent can fail/be degraded in these cases (* meaning overall agent failure not component related):

* - metadata loading	 				failed to load metadata	

* - recover previous state 				could not recover state, error %+v, skipping... 

* - prepare FS 						filesystem error from os 

gateway - dispatch failed				failed to dispatch actions, error: %s", err

gateway - encode request				fail to encode the checkin request

gateway - decode response				fail to decode checkin response

gateway - request failed(auth,network)			fail to checkin to fleet (more detail in a message)


process - download - req fail				fetching package failed

process - download - !200 respose 			call to '%s' returned unsuccessful status code: %d

process - download - FS error				{details from os}

process - verify - hash not found			hash for '%s' not found

process - verify - hash mismatch			hash does not match

process - verify - asc fetch req fail 			fetching asc file from %s {details}
	
process - verify - asc armored key fail 		read armored key ring

process - verify - asc signature check 			check detached signature


process - install - os error				e.g "TarInstaller: error writing to %s: %v" or "tar contained invalid filename: %q"


process - start						failed to start '{details from go runtime}'

process - config					e.g application stopping

agent server 						e.g "failed to marshal connection information", "failed to write connection information"

upgrade - non upgradeable agent				cannot be upgraded; must be installed with install sub-command and running under control of the systems supervisor

upgrade - copy actions store 				failed to copy action store

upgrade - fs errors					error details from FS

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[jlind23]

@michalpristas do you think that it's still relevant?

[jlind23]

Closing this as done, we implement a more detailed status reporting already.
cc @ycombinator