When upgrading, the elastic-agent watch command should be invoked using the new agent binary

[cmacknz]

When the agent upgrades it launches an instance of the elastic-agent watch command to ensure that the new version of the elastic agent remains healthy and does not crash within 10 minutes of being started.

The elastic-agent version invoked is the old version we are upgrading from, not the new version we are upgrading to:

elastic-agent/internal/pkg/agent/application/upgrade/rollback.go

Lines 109 to 118 in 9c06231

	// InvokeWatcher invokes an agent instance using watcher argument for watching behavior of
	// agent during upgrade period.
	func InvokeWatcher(log *logger.Logger) error {
	if !IsUpgradeable() {
	log.Debug("agent is not upgradable, not starting watcher")
	return nil
	}
	versionedHome := paths.VersionedHome(paths.Top())
	cmd := invokeCmd(versionedHome)

The elastic-agent watch command should instead be invoked using the new version of the elastic-agent binary we are upgrading to. For example, in an upgrade from 8.7.1 to 8.8.2 we should invoke the elastic-agent watch command using the 8.8.2 elastic-agent binary.

1. This means the upgrade watcher can be aware of any changes in the Elastic Agent that are introduced in the new version of the upgrade, for example the changes in Add the full version number to the installation directory name #2579
2. This protects from the situation where there is a bug in the upgrade watcher by allowing us to solve it by upgrading to a different version. Today a bug in the upgrade watcher that would prevent a successful upgrade cannot be fixed without a reinstall of the existing agent.

[cmacknz]

Spoke about this in the agent weekly, decided we should do this ASAP as we are making changes to the upgrade watcher and need to make sure the fixes benefit future releases.