Skip to content

Releases: kylan11/understudy

v0.4.1

Choose a tag to compare

@kylan11 kylan11 released this 10 Aug 09:22

Nothing breaks, and there is a new UPGRADING.md that says so at more
length, covering this release and the move from 0.3.x.

hostageMode is now holdMode. The old name described a budget that took
evictions hostage, and 0.4.0 removed the budget. Both fields are read,
holdMode wins when both are set, and a resource that sets only the old
one keeps working while the operator raises a single warning event
against it. The field will be removed in a later release. Because a
defaulted field cannot be told apart from a chosen one, the schema
default moved into the resolver, and the mode the operator settled on is
published as status.mode, which is what the MODE column now prints.

The field was also never explained anywhere. It gates one thing, whether
an eviction is made to wait, and the surge happens in all three modes:
even with holdMode off a stand-in is brought up in parallel, so the
workload is down for as long as a replacement takes rather than as long
as a reschedule takes. voluntary-only exists for workloads that drain
connections on SIGTERM, where refusing an eviction whose node is leaving
at a fixed moment only burns the window the pod could have used.

Also fixes the intermittent build failure. The eviction specs attempted
one eviction and then waited for the surge, which races the doom lease
those attempts refresh; a drainer that gives up after a single try is not
the scenario being tested. They now keep evicting while they wait, the
way a real drainer retries.

Upgrading:

helm upgrade understudy oci://ghcr.io/kylan11/charts/understudy \
  --namespace understudy-system

The image is built for arm64 and amd64.

v0.4.0

Choose a tag to compare

@kylan11 kylan11 released this 07 Aug 13:34
7ac0e59

Understudy now holds protected evictions at admission instead of owning a PodDisruptionBudget. This removes the structural node-pinning failure: disrupters can select and begin terminating a node, while the fail-open webhook returns HTTP 429 until a healthy stand-in is ready.

Breaking changes

  • understudy_pdb_relaxed_total is renamed to understudy_hold_relaxed_total.
  • understudy_node_pinned_total and the NodePinned condition are removed.
  • Helm values signals.evictionWebhook*, signals.pinDetection, and failsafe are retired. Webhook settings are now webhook.timeoutSeconds and webhook.namespaceSelector.
  • The ValidatingWebhookConfiguration suffix changes from -eviction-observer to -eviction-hold; the admission URL remains /observe-eviction for safe upgrades.
  • Operator unavailability now creates a protection gap rather than leaving blocking budgets behind. Use two replicas on separate nodes for production.

Upgrade behavior

A leader-gated startup sweep deletes legacy PodDisruptionBudgets labeled understudy.sh/owned=true. PDB RBAC is reduced to get, list, and delete for this migration and will be removed in v0.5.0. Existing webhook certificate material is reused across Helm upgrades.

Artifacts

  • Image: ghcr.io/kylan11/understudy:0.4.0 (linux/amd64, linux/arm64)
  • Chart: oci://ghcr.io/kylan11/charts/understudy --version 0.4.0

v0.3.1

Choose a tag to compare

@kylan11 kylan11 released this 07 Aug 10:36

Nothing breaking.

A hostage budget on a singleton can pin its node: Karpenter checks budgets
before consolidating, drifting or expiring a node and simply never begins
while disruptions are not allowed, producing no signal anything could
react to. This release names that tradeoff in the README and makes it
visible per workload. A new watcher follows the DisruptionBlocked events
the blocker leaves behind and, when the named budget is one this operator
owns, raises a NodePinned condition on the affected Understudy, with a
Warning event and an understudy_node_pinned_total counter. The condition
clears when the refusals stop. Detection only, on by default
(signals.pinDetection in the chart): a pinned node is reported, never
resolved. The tradeoff section in the README lists the mitigations; what
actually moves a pinned pod is rescheduling it, via a deploy, a rollout
restart, or deleting the node claim so a real drain starts.

Also fixes the cross-build make target, which duplicated the platform
flag the Dockerfile already carries and masked the resulting failure.

The image is now tagged to match the chart appVersion exactly (0.3.1, no
v prefix), so installs no longer need an image.tag override.

Upgrading:

helm upgrade understudy oci://ghcr.io/kylan11/charts/understudy \
  --namespace understudy-system

The image is built for arm64 and amd64.

v0.3.0

Choose a tag to compare

@kylan11 kylan11 released this 06 Aug 12:30

Zerover: breaking changes bump the minor until 1.0.

Breaking. The API group moved from apps.understudy.dev to
apps.understudy.sh, because the old domain belongs to an unrelated party. The
annotation and label keys moved with it. Existing installs need the old
custom resource definition deleted and their Understudy objects recreated.

The chart is now published, so installing no longer needs a checkout:

helm install understudy oci://ghcr.io/kylan11/charts/understudy \
  --namespace understudy-system --create-namespace

The image is built for arm64 and amd64. Documentation covering the mechanism,
the signal contract and the failure behaviour is in docs/how-it-works.md.

v0.2.0

Choose a tag to compare

@kylan11 kylan11 released this 06 Aug 11:32

Zerover: the API may still change before 1.0.

Milestone 2 and milestone 3, plus a fail-safe for operator outages.

Eviction webhook. An observe-only admission webhook on pods/eviction sees
every eviction attempt, including those the PDB then rejects, so drainers
that neither cordon nor taint are now covered. It always admits and is
fail-open. It surges only for pods covered by a PDB this operator owns, so an
unrelated eviction on the same node does not disturb protected workloads.

Cloud sentinels. An optional DaemonSet reads the cloud's termination
notice and taints its own node with the deadline. The core decodes that as an
involuntary signal and can now decide between holding the eviction and
releasing it immediately. In v0.1.0 a spot interruption reached the operator
only as an ordinary drainer taint, so it was held blind against a hard
termination wall. AWS is validated against real spot interruptions driven by
Fault Injection Simulator. GCP and Azure probes are covered by unit tests
only and should be treated as experimental until run on GKE and AKS.

Operator-down fail-safe. The operator heartbeats the PDBs it owns and a
CronJob deletes those whose heartbeat has gone stale, so an outage degrades
to plain Kubernetes rather than leaving a blocking PDB behind. Verified live:
the stale budget was removed automatically once the operator stopped.

Also in this release: a metrics Service, image pull secret support, and a
chart that no longer doubles the release name.

v0.1.0

Choose a tag to compare

@kylan11 kylan11 released this 06 Aug 11:15

First tagged release. Zerover: the API may still change before 1.0.

Milestone 1 is complete and validated against a live EKS cluster running
Karpenter on arm64 spot nodes:

  • Hostage PodDisruptionBudget routes every eviction through the operator.
  • Surge on node cordon and on drainer taints (Karpenter, cluster-autoscaler),
    matched by key.
  • Release only once the stand-in is traffic-ready, including pod readiness
    gates, so load balancer registration is respected with no provider code.
  • Deterministic scale-down of the doomed pod via pod-deletion-cost.
  • Never-wedge TTL relaxes the PDB rather than stalling a drain forever.
  • Self-exclusion, graceful skip of unsupported targets, fail-open posture.

Measured on a single-replica workload behind an ALB, with no spare capacity
so the replacement waited for a cold EC2 node: a drain cost 1 failed request
in 350, and a real AWS spot interruption cost 1 in 764 with the handover
finishing 29 seconds before the instance was terminated. For comparison, an
ordinary rolling update of the same workload cost 2 in 38.