Skip to content

Support managed blue/green Flex Node agent upgrades - #266

Draft
hbc (bcho) wants to merge 21 commits into
mainfrom
hbc/daemon-upgrade
Draft

Support managed blue/green Flex Node agent upgrades#266
hbc (bcho) wants to merge 21 commits into
mainfrom
hbc/daemon-upgrade

Conversation

@bcho

Copy link
Copy Markdown
Member

Summary

  • handle Machina AgentUpgrade MachineOperations in the Flex daemon
  • migrate legacy direct-file installs into an idempotent blue/green binary layout
  • validate HTTPS release archives, compressed archive SHA-256, architecture-specific members, size bounds, and candidate version
  • synchronize the active nspawn exec-credential binary with the host candidate
  • persist upgrade state across daemon restarts and recover the verified last-good host/nspawn binaries through systemd
  • redact credential-bearing URL query strings from logs and operation status
  • add daemon MachineOperation RBAC, operations documentation, and focused E2E coverage

Upgrade lifecycle

  1. Mark the operation in progress and persist the initiating daemon instance.
  2. Download and verify the archive and candidate in the inactive slot.
  3. Atomically switch host links and synchronize the active nspawn rootfs.
  4. Schedule restart through a separate transient systemd unit.
  5. Publish success only after the restarted manager reaches cache readiness.
  6. Preserve recovery state until a last-good process is confirmed running after failure.

Duplicate reconciliation is guarded by the durable signal, and rollback uses a bounded context independent of daemon shutdown cancellation.

Testing

  • make check
  • shell syntax and ShellCheck for the E2E/recovery scripts
  • real AKS/Flex Node validation of MachineOperation success, binary replacement, host/nspawn digest equality, Node readiness, and kubelet Lease renewal

The E2E command is available as:

./hack/e2e/run.sh agent-upgrade

A clean full forced-rollback/retry cloud run should be completed before marking this PR ready for review.

Fixes #265

@bcho

Copy link
Copy Markdown
Member Author

Temporarily reopening to retrigger the repository's native PR checks; keeping this PR in draft.

@bcho hbc (bcho) closed this Aug 6, 2026
@bcho hbc (bcho) reopened this Aug 6, 2026
@bcho
hbc (bcho) marked this pull request as ready for review August 6, 2026 21:36
Copilot AI lite review requested due to automatic review settings August 6, 2026 21:36
@bcho hbc (bcho) closed this Aug 6, 2026
@bcho hbc (bcho) reopened this Aug 6, 2026
@bcho
hbc (bcho) marked this pull request as draft August 6, 2026 21:36
@bcho

Copy link
Copy Markdown
Member Author

Dependency update: the secure HTTPS/SHA-256/archive/blue-green installer was moved to Azure/unbounded#585. This PR now consumes Unbounded commit 5e0aab6f, removing about 430 lines of duplicated FlexNode implementation and tests.

@bcho

Copy link
Copy Markdown
Member Author

Self-review updates pushed in 3448ea9: updated to Unbounded 8173c7ee, removed goal-state type coupling, and now converges the blue/green layout plus recovery service/script and OnFailure unit on every daemon startup. Added an in-place legacy service/binary migration test.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@bcho

Copy link
Copy Markdown
Member Author

Final focused real-cluster E2E passed on the updated implementation: successful MachineOperation upgrade, host/nspawn digest equality, Node readiness, kubelet Lease renewal, forced candidate failure with automatic last-good rollback, sensitive URL status check, retry into the inactive slot, and workload smoke test.

@bcho

Copy link
Copy Markdown
Member Author

Updated the dependency to Unbounded 72a21798, where Unbounded's own managed AgentUpgrade path now uses the same secure installer.

@bcho

Copy link
Copy Markdown
Member Author

Updated to Unbounded 75333be3, which removes the duplicate internal staging implementation/tests after migrating Unbounded production to the shared secure installer.

@bcho

Copy link
Copy Markdown
Member Author

Updated to Unbounded d68bbe5c, which fixes the upstream AgentUpgrade E2E for the new HTTPS + sha256 contract.

@bcho
hbc (bcho) deployed to e2e-testing August 7, 2026 20:11 — with GitHub Actions Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support managed blue/green AKS Flex Node agent binary upgrades

2 participants