fix(agent): scale down using agent shutdown hook

[lucaspin]

Motivation

This change, which is present on Kubernetes 1.26+, added validation to not allow multiple HPAs to point at the same target. That restriction causes our autoscaling to break.

Solution

Instead of using 2 HPAs (one for scaling up, and one for scaling down), we use only 1 HPA now - for scaling up the agent deployment. The scaling down logic will be handled by a shutdown hook, which executes every time an agent gets idle for a specified period of time. The idle timeout period can be configured through the agent.autoscaling.idleTimeoutForScaleDown configuration value; by default, that value is 30min.

Shutdown hook logic

Since we can't choose which pods to delete when scaling a deployment, when an agent gets idle and shuts down, we:

1. Annotate the pod for the agent shutting down with a pod deletion cost of -1
2. Decrease the number of replicas in the deployment by 1. This only happens if we are not already at the minimum replica count, specified by agent.autoscaling.min. If we are at the minimum number of replicas already, we delete the pod, without decreasing the replica count, to avoid potentially getting into a CrashLoopBackOff.

Since multiple shutdown hooks can be executing at the same time, we use optimistic locking on the deployment, using a semaphoreci.com/handle annotation.

[lucaspin]

Due to Kubernetes' eventual consistency model, and its lack of more granular control over deployment scaling operations, this solution is too messy. Too many race conditions all over the place. Even though it "works", it would be terrible to maintain and troubleshoot, so I'm closing this.