0.22.7

Backends

Kubernetes

Orphaned resources

Orphan cleanup, introduced in 0.22.6, now supports the kubernetes backend. It finds pods, services, and secrets left behind by dstack and reports or deletes them, depending on DSTACK_SERVER_ORPHAN_CLEANUP_MODE.

Proxy jump

The new proxy_jump.node_selector and proxy_jump.tolerations properties let you run the jump pod only on specific nodes, e.g. nodes with an ExternalIP or CPU-only nodes:

type: kubernetes
kubeconfig:
  filename: ~/.kube/config
proxy_jump:
  node_selector:
  - example.com/jump-host
  tolerations:
  - example.com/dedicated=jump-host:NoSchedule

tolerations uses the kubectl taint format. See Kubernetes for details.

Kubernetes now recreates the jump pod used for SSH access to runs if it is evicted or its node is removed. Previously, an evicted jump pod prevented new runs from being provisioned.

New jump pods also reserve CPU and memory, making them less likely to be evicted under node pressure. Existing jump pods keep their resource settings until they are recreated.

Important

If dstack uses a Kubernetes user or service account with a custom Role, update the Role before upgrading the server. dstack now needs get and create on replicasets (API group apps) and list on pods. Without them, provisioning fails with 403 Forbidden.

If list on services or secrets is not granted, orphan cleanup skips those resources and logs a warning. See Required permissions for the full Role.

Gateways

Python 3.12

Gateways now run on Python 3.12 managed by uv instead of the OS Python. Existing gateways are updated automatically when you upgrade the server. Services are unavailable for a few seconds while their gateway restarts.

Gateway hosts now download uv and Python from GitHub when the gateway is installed or updated, so they need outbound access to github.com, in addition to PyPI.

What's changed

• Re-creating failed gateways with a single apply by @jvstme in #4442
• Install gateway app with uv-managed Python 3.12 by @r4victor in #4439
• [Docs] Fix outdated facts, broken links, and examples by @peterschmidt85 in #4446
• Kubernetes: manage the jump pod with a ReplicaSet by @un-def in #4447
• Avoid fetching all projects in the UI by @r4victor in #4449
• Kubernetes: run the jump pod with Guaranteed QoS by @un-def in #4451
• Support orphan cleanup in the Kubernetes backend by @r4victor in #4455
• [Docs] Fix head-node key requirements with the SSH proxy by @peterschmidt85 in #4457
• Kubernetes: add proxy_jump.node_selector and tolerations by @un-def in #4459
• Close Nebius SDKs and lease cached backends by @r4victor in #4460
• [UX] Grouped GPU prices are hard to compare by @peterschmidt85 in #4456

Full changelog: 0.22.6...0.22.7