[Agent] Improve error log messages when agent binary cannot be downloaded

[joshdover]

When a user initiates remote Agent upgrades from the Fleet UI, upgrades can fail due to firewalls or similar from preventing the agent binary from being downloaded. In this case, the error in Agent logs will look something like:

13:39:33.586 elastic_agent [elastic_agent][info] 2021-12-08T13:39:33+01:00 - message: Application: [24abba90-a8e6-430d-a889-33ec756b3574]: State changed to UPDATING: Update to version '7.16.0' started - type: 'STATE' - sub_type: 'UPDATING'
13:41:35.011 elastic_agent [elastic_agent][error] 2021-12-08T13:41:35+01:00 - message: Application: [24abba90-a8e6-430d-a889-33ec756b3574]: State changed to FAILED: failed upgrade of agent binary: 2 errors occurred:
	* package '/Library/Elastic/Agent/data/elastic-agent-fd322d/downloads/elastic-agent-7.16.0-darwin-x86_64.tar.gz.sha512' not found: open /Library/Elastic/Agent/data/elastic-agent-fd322d/downloads/elastic-agent-7.16.0-darwin-x86_64.tar.gz.sha512: no such file or directory
	* fetching package failed: context deadline exceeded (Client.Timeout or context cancellation while reading body)

 - type: 'ERROR' - sub_type: 'FAILED'
13:41:35.011 elastic_agent [elastic_agent][error] failed to dispatch actions, error: failed upgrade of agent binary: 2 errors occurred:
	* package '/Library/Elastic/Agent/data/elastic-agent-fd322d/downloads/elastic-agent-7.16.0-darwin-x86_64.tar.gz.sha512' not found: open /Library/Elastic/Agent/data/elastic-agent-fd322d/downloads/elastic-agent-7.16.0-darwin-x86_64.tar.gz.sha512: no such file or directory
	* fetching package failed: context deadline exceeded (Client.Timeout or context cancellation while reading body)

13:41:35.011 elastic_agent [elastic_agent][error] Elastic Agent status changed to: 'error'
13:46:10.847 elastic_agent [elastic_agent][info] Elastic Agent status changed to: 'online'

It's not obvious exactly what the issue is becuase:

• The no such file or directory error message seems to indicate a missing file, which is not obvious that it comes from a download issue
• The context deadline exceeded (Client.Timeout or context cancellation while reading body) error message is a little obscure and could be improved with a more explicit message like Agent binary could not be retrieved.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[nimarezainia]

Documenting what was discussed earlier as a small modification to the download timeout and making it a bit more robust:

• Default agent.download.timeout to be extended to 10minutes
• Every minute log an informational alert that at a minimum identifies the agent and logs the transmission rate achieved, and x of y bytes downloaded.
• When the time out is reached there should again be an informational alert (if need be this can be a warning) timeout and the alert progression interval to be configurable via UI as part of [Fleet] Give the ability to set up elastic agent download timeout kibana#121069

[fyi @blakerouse @michel-laterman @jlind23 ]

[jlind23]

@blakerouse any update on this one?

[jlind23]

Per discussed with @blakerouse postponing it to 8.3
cc @nimarezainia @ph

[jlind23]

@blakerouse we discussed yesterday with @ph and @nimarezainia the outcome was that we shouldn't backport it and rather keep it for 8.3 and every further versions.