feat: active-active HA mode using Redis lease

[sakkiii]

what

Implements multi-replica active-active HA for Atlantis. Each PR is "owned" by one replica via a Redis lease. Webhooks arriving at a non-owning replica are forwarded over HTTP to the owner. Ownership renews every TTL/3 seconds. When a lease expires, another replica takes over.

Plan Storage Modes

Local Plans

Configuration:

--locking-db-type=redis
--replica-advertise-url=http://atlantis-0.atlantis-headless:4141
ATLANTIS_INTERNAL_COMMAND_TOKEN=<shared-secret>
--enable-external-stores=false

• Planning writes .tfplan files only beneath the owner's data directory.
• Commands continue to use owner-local plans while the ownership claim remains live.
• After takeover, the new owner starts with an empty local pull directory.
• Apply reports the normal missing working directory or missing plan error.
• The user must run plan again before apply.

External Plans

Configuration:

--locking-db-type=redis
--replica-advertise-url=http://atlantis-0.atlantis-headless:4141
ATLANTIS_INTERNAL_COMMAND_TOKEN=<shared-secret>
--enable-external-stores=true

• Planning writes locally and saves the plan through the existing external PlanStore.
• Owner takeover clears only local state; remote plans remain available.
• A targeted apply reclones the required workspace and loads its plan through the PlanStore.
• Apply-all lists stored workspaces, clones them before restoration, and restores their plans for discovery.
• The apply runner loads each plan and validates its stored head commit before Terraform executes.
• A missing plan, absent commit metadata, or commit mismatch fails the command and requires re-plan.
• External-store unavailability fails the command; Atlantis does not apply an unvalidated local fallback.
• Successful apply and pull-close cleanup retain the existing best-effort removal behavior for external objects.

External storage does not remove the need for ownership. Ownership still provides pull request command ordering, stable local runtime state, forwarding, log locality, and cancellation affinity.

why

Implements multi-replica active-active HA for Atlantis. Each PR is "owned" by one replica via a Redis lease. Webhooks arriving at a non-owning replica are forwarded over HTTP to the owner. Ownership renews every TTL/3 seconds. When a lease expires, another replica takes over.

tests

1. Redis configuration alone does not activate replica routing.
2. Complete routing settings activate routing without an enable flag.
3. Partial routing settings fail startup with a precise validation error.
4. Replica ID defaults to the process hostname and supports an explicit override.
5. Startup accepts replica routing with external storage configured.
6. Scale-up leaves an existing pull request with its current owner.
7. Graceful release permits lazy takeover by another replica.
8. Lease expiry permits lazy takeover after owner failure.
9. Local-plan takeover deletes stale local state and requires re-plan.
10. External-plan takeover deletes stale local state, restores a matching plan, and permits apply.
11. External-plan takeover rejects a plan from a different head commit.
12. Missing or unavailable external plans fail without executing Terraform.
13. Pull-close cleanup attempts external plan deletion before releasing ownership.
14. Existing single-replica, Redis-only, and external-store-only modes remain unchanged.

references

closes #6751
closes #1571

Assisted-by: Claude