fix(guest): make gateway refresh failover atomic (re-land #962)

[kvinwang]

Problem

PR #962 shows as merged on GitHub, but its change is not on the mainline and never was.

#962 was stacked: its base was fix/guest-local-provider-inventory-master, not the default branch. That parent branch had already been merged to the mainline via #1016 (merge commit 87abbcccd). #962 landed on the parent after that, so nothing carried it forward. Its merge_commit_sha e1745d672 is not an ancestor of next.

The result is a silent regression: getQuote-adjacent gateway registration refresh in dstack-util/src/system_setup.rs still has the non-atomic behaviour #962 fixed, and the three tests that pinned that behaviour do not exist on next.

Verified by content rather than by commit graph — of the added lines sampled from a8240ffec, fn load_from(path: &Path) -> Option<Self> and Self::load_from(Path::new(GATEWAY_CACHE_PATH)) are absent from next, along with the surrounding body.

Fix

Cherry-picks the five commits of #962 onto next, unchanged and in their original order (-x recorded, so each commit references the original SHA):

• fix(guest): make gateway refresh failover atomic
• style(guest): format gateway refresh tests
• fix(guest): retain gateway keys across failover attempts
• style(guest): format gateway failover closure
• refactor(guest): keep gateway failover loop inline

All five applied without conflict despite 73 commits having landed on next since the merge base, and despite system_setup.rs itself having changed by 21 insertions / 43 deletions over that window — the two sets of edits touch different regions.

Net effect is 99 insertions / 8 deletions in dstack/dstack-util/src/system_setup.rs.

Verification

• cargo check -p dstack-util passes.
• cargo test -p dstack-util — 82 passed, 0 failed.
• cargo fmt --all --check clean; cargo clippy -p dstack-util produces no new warnings.
• Three tests restored by this PR were confirmed absent from next beforehand and present after: key_store_round_trip_is_private_and_stable, malformed_replacement_does_not_overwrite_working_cache, gateway_rpc_urls_are_normalized_once.

Note on the stacked-PR pattern

This was found while auditing branches for cleanup: every merged-PR branch still alive had commits pushed after its own merge, and three of them were stacked onto another feature branch rather than the mainline. #960 hit the same trap and was rescued manually via #1020; #962 was not. delete_branch_on_merge is now enabled, which removes the branch at merge time and makes this specific failure mode much harder to reproduce.

[Copilot]

Pull request overview

Re-lands the previously stacked gateway refresh changes into next by making gateway key cache handling and gateway RPC URL normalization more robust and testable, and by restoring regression tests that pin atomic refresh behavior.

Changes:

• Refactors GatewayKeyStore to support load_from/save_to, adds is_cert_valid_at, and centralizes gateway RPC URL normalization.
• Updates gateway client construction to use the normalized RPC URL helper.
• Restores/introduces focused regression tests covering URL normalization, key cache persistence/permissions, malformed cache handling, and certificate refresh boundary behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.