Skip to content

[Bug]: Blocks param for SSH fleet does not work at top yaml level #3278

@pawelkopec

Description

@pawelkopec

Steps to reproduce

I was trying to create SSH fleet with Tenstorrent chips with blocks: auto so that we can deploy workloads which use less than all chips on the node.

When I create the fleet with dstack apply -f tt-fleet.yaml with blocks: auto defined at the top yaml level

type: fleet
name: tt-fleet
blocks: auto

ssh_config:
  user: kopec
  identity_file: ~/.ssh/id_ed25519
  hosts:
    - <IP>

then dev env creation fails with No matching instance offers available as if the configured node was not available.

However when I move blocks: auto to the node level like so

type: fleet
name: tt-fleet

ssh_config:
  user: kopec
  identity_file: ~/.ssh/id_ed25519
  hosts:
    - hostname: <IP>
      blocks: auto

it works properly

Actual behaviour

This is what I got when blocks: auto was set at the top yaml level

$ dstack fleet
 FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED 
 tt-fleet 0 ssh (remote) cpu=32 mem=504GB disk=2804GB n300:24GB:4 $0 idle 3 mins ago

$ dstack apply -f pawelk-dev-tt.yaml 
The working_dir is not set — using legacy default "/workflow". Future versions will default to the image's working directory.

 Project main 
 User admin 
 Configuration pawelk-dev-tt.yaml 
 Type dev-environment 
 Resources cpu=2.. mem=8GB.. disk=100GB.. n300:1 
 Spot policy on-demand 
 Max price - 
 Retry policy - 
 Creation policy reuse-or-create 
 Idle duration 5m 
 Max duration - 
 Inactivity duration - 
 Reservation - 

No matching instance offers available. Possible reasons: https://dstack.ai/docs/guides/troubleshooting/#no-offers

Expected behaviour

This is the desired output I got once I moved blocks: auto to the node level

$ dstack apply -f pawelk-dev-tt.yaml 
The working_dir is not set — using legacy default "/workflow". Future versions will default to the image's working directory.

 Project              main                                  
 User                 admin                                 
 Configuration        pawelk-dev-tt.yaml                    
 Type                 dev-environment                       
 Resources            cpu=2.. mem=8GB.. disk=100GB.. n300:1 
 Spot policy          on-demand                             
 Max price            -                                     
 Retry policy         -                                     
 Creation policy      reuse-or-create                       
 Idle duration        5m                                    
 Max duration         -                                     
 Inactivity duration  -                                     
 Reservation          -                                     

 #  BACKEND       RESOURCES                                INSTANCE TYPE   PRICE       
 1  ssh (remote)  cpu=8 mem=126GB disk=2804GB n300:24GB:1  instance (1/4)  $0     idle 

Submit the run pawelk-dev-tt? [y/n]: y
 NAME           BACKEND       GPU          PRICE  STATUS   SUBMITTED 
 pawelk-dev-tt  ssh (remote)  n300:24GB:1  $0     running  12:13     

pawelk-dev-tt provisioning completed (running)
/tmp/vscode-server- 100%[===================>]  64.74M  38.1MB/s    in 1.7s    
mkdir: created directory '/root/.vscode-server'
mkdir: created directory '/root/.vscode-server/bin'
mkdir: created directory '/root/.vscode-server/bin/2901c5ac6db8a986a5666c3af51ff804d05af0d4'
Installing extensions...
Installing extension 'ms-python.python'...
Installing extension 'ms-toolsai.jupyter'...
Extension 'ms-toolsai.vscode-jupyter-cell-tags' v0.1.9 was successfully installed.
Extension 'ms-python.vscode-python-envs' v1.2.0 was successfully installed.
Extension 'ms-toolsai.jupyter-renderers' v1.3.0 was successfully installed.
Extension 'ms-python.debugpy' v2025.14.1 was successfully installed.
Extension 'ms-toolsai.jupyter' v2023.8.1002501831 was successfully installed.
Extension 'ms-python.python' v2025.16.0 was successfully installed.
Extension 'ms-toolsai.vscode-jupyter-slideshow' v0.1.6 was successfully installed.
Extension 'ms-toolsai.jupyter-keymap' v1.1.2 was successfully installed.
Extension 'ms-python.vscode-pylance' v2025.9.1 was successfully installed.
pip install ipykernel...

To open in VS Code Desktop, use link below:

  vscode://vscode-remote/ssh-remote+pawelk-dev-tt/workflow

To connect via SSH, use: `ssh pawelk-dev-tt`

dstack version

0.19.36

Server logs

Additional information

Solved offline with @peterschmidt85. Suspected cause is wrong field validation at global level

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions