Skip to content

0.22.8

Latest
Latest

Choose a tag to compare

View all tags
@un-def un-def released this 12 Nov 14:08
· 2 commits to master since this release
0.22.8
5e8977f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Backends

Kubernetes

MIG

The kubernetes backend now supports NVIDIA MIG slices configured with the GPU Operator's MIG Manager (the single or mixed strategy). dstack treats each slice as a GPU with the slice's memory, e.g., a 1g.10gb slice of an A100 is A100:10GB. To run on slices, specify the GPU memory:

type: dev-environment
ide: vscode
resources:
  gpu: A100:10GB

If the GPU memory isn't specified or has no maximum (e.g., gpu: A100 or gpu: 24GB..), the run gets whole GPUs only. With --full-offers, dstack offer also lists matching slices:

$ dstack offer --gpu A100 --full-offers

 #  BACKEND                                        RESOURCES                                           INSTANCE TYPE                     PRICE
 1  kubernetes (gke_dstack_us-central1-a_mig-e2e)  cpu=x86:23 mem=156.9GB disk=97.4GB gpu=A100:40GB:1  gke-mig-e2e-a100x2-946ce248-zsmm  $0
 2  kubernetes (gke_dstack_us-central1-a_mig-e2e)  cpu=x86:23 mem=156.9GB disk=97.4GB gpu=A100:20GB:1  gke-mig-e2e-a100x2-946ce248-zsmm  $0
 3  kubernetes (gke_dstack_us-central1-a_mig-e2e)  cpu=x86:23 mem=156.9GB disk=97.4GB gpu=A100:20GB:1  gke-mig-e2e-a100x2-946ce248-zsmm  $0

Slices with the same memory are the same GPU for dstack, so a specific MIG profile (e.g., 3g.40gb vs 4g.40gb) can't be requested. dstack metrics doesn't show GPU metrics for slices yet. See Kubernetes for details.

Important

On clusters that already use the single strategy, slices used to be offered as whole GPUs (e.g., A100:80GB), so gpu: A100 runs could land on them. Now these runs get whole GPUs only. To keep running on slices, specify the GPU memory, e.g., gpu: A100:10GB.

Nebius

Capacity advisor

Nebius offers used to have unknown availability, so dstack could attempt configurations blocked by quota or capacity. The new capacity_advisor setting uses Nebius Capacity Advisor to show availability and quota in offers and skip offers that can't be provisioned:

projects:
  - name: main
    backends:
      - type: nebius
        creds:
          type: service_account
          service_account_id: serviceaccount-e00dhnv9ftgb3cqmej
          public_key_id: publickey-e00ngaex668htswqy4
          private_key_file: ~/path/to/key.pem
        capacity_advisor: true

The setting is off by default because it adds roughly 1–3 seconds to getting offers. Availability and quota are cached for 10 minutes. See Nebius.

Fleets

SSH fleets

SSH fleets now support in-place updates of ssh_config.identity_file and ssh_config.proxy_jump.identity_file, including per-host overrides. Previously, these changes required re-creating the fleet, which is blocked while runs are active.

Authorize the new public key on the host, update the identity file path, and apply the fleet configuration:

$ dstack apply -f fleet.dstack.yml

Found fleet my-fleet. Detected changes that can be updated in-place:
- Configuration properties:
  - ssh_config.identity_file
  - ssh_config.proxy_jump.identity_file

Runs continue without restarts, and established SSH connections, including attach sessions and service tunnels, stay open. After updating proxy_jump.identity_file, allow about 20 seconds before removing the old public key from the proxy_jump host, so that gateways and the SSH proxy receive the new key. See Update fleets.

Exports

In dstack Enterprise, Factory, and Sky, backend fleets can now be exported to other projects. Importer projects can run on the fleet's existing instances:

$ dstack import list
 NAME            FLEETS        GATEWAYS
 main/my-export  shared-fleet  -

$ dstack fleet list
 NAME               NODES  GPU  SPOT  BACKEND           PRICE    STATUS  CREATED
 main/shared-fleet  2..4        auto  aws                        active  17 mins ago
    instance=0             -    spot  aws (eu-north-1)  $0.012   idle    17 mins ago
    instance=1             -    spot  aws (eu-north-1)  $0.012   idle    17 mins ago

Note

Limitation: Currently, this is intended to be used with fleets that already have provisioned nodes. New instances are not provisioned for importer runs. Additionally, importer runs cannot mount network volumes on such fleets. This limitation will be lifted in one of the upcoming releases.

Services

Router

Services with the dynamo router now support retry and in-place updates with dstack apply, like any other service.

retry:
  on_events: [error, interruption]

Replicas

Previously, a service whose replicas exited with code 0 either had them resubmitted or ended as done, depending on whether they all finished at the same time. Now services never end because their replicas exited. Such replicas are brought back by scaling, and if all of them exit at once, the service is resubmitted with a backoff.

UI

Login

User tokens throughout the UI are now hidden until you click Show token, so they don't leak when you share or record your screen. Copy buttons still copy the real token.

Breaking changes

  • dstack now requires Python 3.11 or later. Run configurations can still use python: "3.9" and "3.10".

  • The DSTACK_ROUTER_INTERNAL_IP environment variable, deprecated in 0.22.6, has been removed. Services that still use it must switch to ${{ groups[i].replicas[j].IP_ADDRESS }}:

    # Before
- export ETCD_ENDPOINTS=http://$DSTACK_ROUTER_INTERNAL_IP:2379
# After, if the router is the first group
- export ETCD_ENDPOINTS=http://${{ groups[0].replicas[0].IP_ADDRESS }}:2379

What's changed

Full changelog: 0.22.7...0.22.8

Contributors

un-def, r4victor, and 2 other contributors
Assets 2
Loading