[Feature]: shim and runner update mechanism

[un-def]

Problem

Although every effort is made to maintain server↔shim and server↔runner compatibility, new features require shim/runner update quite often, and compatibility by itself is not free.

If there was a way to update both shim and runner without recreating the fleet, then

1. Users could use new feature right away after the the server update.
2. Developers would spend less effort maintaining compatibility.
3. Breaking API changes could be introduced in any release.
4. Users wouldn't spend time and resources recreating fleets in order to update shim/runner, additionally risking compute resources with limited availability.

Solution

Add new API method(s) to shim to instruct it to install a specific version of shim/runner. It should be quite easy, and non-intrusive, as

1. Running jobs can use the previous version of the runner even if the newer is already installed (handled by the Linux kernel).
2. The shim can replace its binary and just exit
   • shim service (re)start is already maintained by systemd with SSH fleets, which can be used for cloud instances as well.
   • shim already restores its state on restart, no additional effort required.

Workaround

No response

Would you like to help us implement this feature by sending a PR?

Yes