fc-agent: improve maintenance scheduling

[dpausp]

• New requests can now be merged with existing ones if their activities have the same type. Significant changes to activities cause postponing of the updated request.
• Requests can be cancelled by requesting an activity which nullifies the original activity (for example, reset channel back to current system channel => planned update will be cancelled)
• UpdateActivity with metadata and better comment generation replaces dumb shell scripts for planned system updates.
• VMChangeActivity with metadata replaces RebootActivity for mem and core changes.
• All activities can request a reboot which will be done after all due requests have been executed.
• Continously scheduled requests will be executed in one go if at least the first request is due, avoiding repeated switching to maintenance mode in a short time frame and possibly unneccessary reboots.
• Overdue requests (more than 30 minutes after scheduled start time) will be postponed to avoid overrunning the planned maintenance window or interfering with other machines going into maintenance mode.
• Maintenance preparation time and request execution time are different concepts now. Execution of requests is typically quite fast but there may be commands delaying the execution of all requests. Directory doesn't support this yet so we just report the sum of preparation time and estimated execution time (but at least 15min).
• Un-tangled maintenance code and manage.py: all maintenance requests are now generated in maintenance.py.
• Fix handling of postponed requests and cleaned up state updates in the process. tempfail and retrylimit don't exist anymore as dedicated states.
• Update shortcut saving time: if the new channel of an UpdateActivity results in the same system, just set the system channel and forget about the update.
• Explicitly exit after calling the reboot command.
• Reduce number of channel URL resolve calls (which impact Hydra), UpdateActivity expects a resolved URL now.

@flyingcircusio/release-managers

Release process

Impact:

Changelog:

• agent: improve scheduling of maintenance activities for system updates and VM property changes (memory, CPU cores). The main change is that activities can be merged/updated now. As a result, the number of reboots is reduced and multiple pending updates can be applied faster. Activities can be cancelled if they are no longer effective, for example if a memory change is requested in error and reset to the previous value some time later.
  Reboots for kernel updates now happen directly after system updates, avoiding scheduling another maintenance for the reboot. This also fixes the long-standing bug that delayed activities could be executed outside of maintenance windows. Activities that are overdue (more than 30min after planned time) are postponed for at least 8 hours and scheduled again (PL-129777).

Security implications

• Security requirements defined? (WHERE)
  • nothing new, only affects internal handling of how we prepare and request updates and VM property changes.
• Security requirements tested? (EVIDENCE)
  • Python tests cover new and old functionality
  • extensive manual tests on a dev VM and one with production flag set
    • updates, reboots and VM property (mem, cpu) changes merge properly
    • significant updates postpone the request and are announced via mail
    • serialized legacy requests still load and run
    • updates are not prepared multiple times
    • executing an updateActivity properly registers and switches to the new system, reboots are done
    • continuously scheduled requests are executed in one go