Skip to content

dstack pool remove doesn't work #892

@TheBits

Description

@TheBits

Steps to reproduce:

  • Run dstack run
  • Stop the run
  • Remove instance with dstack pool remove
  • The instance was still in the ready state

CLI

(venv) ➜  my_dstack_public git:(master) ✗ dstack run . -b azure
 Configuration  .dstack.yml    
 Project        main           
 User           admin          
 Pool name      default-pool   
 Min resources  2..xCPU, 8GB.. 
 Max price      -              
 Max duration   6h             
 Spot policy    auto           
 Retry policy   no             

 #  BACKEND  REGION      INSTANCE         RESOURCES         SPOT  PRICE       
 1  azure    westeurope  Standard_D2s_v3  2xCPU, 8GB,       yes   $0.012      
                                          100GB (disk)                        
 2  azure    westeurope  Standard_E2s_v4  2xCPU, 16GB,      yes   $0.015246   
                                          100GB (disk)                        
 3  azure    westeurope  Standard_D4s_v3  4xCPU, 16GB,      yes   $0.024      
                                          100GB (disk)                        
    ...                                                                       
 Shown 3 of 88 offers, $40.962 max

Continue? [y/n]: y
moody-mayfly-1 provisioning completed (running)
          /tmp/vsco   0%[                    ]       0  --.-KB/s                       /tmp/vscod  75%[==============>     ]  36.44M   182MB/s              /tmp/vscode-server- 100%[===================>]  48.09M   185MB/s    in 0.3s    
mkdir: created directory '/root/.vscode-server'
mkdir: created directory '/root/.vscode-server/bin'
mkdir: created directory '/root/.vscode-server/bin/0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2'
Installing extensions...
Installing extension 'ms-python.python'...
Installing extension 'ms-toolsai.jupyter'...
Extension 'ms-toolsai.jupyter' v2023.11.1100101639 was successfully installed.
^C
Stop the run before detaching? [y/n]: y
Stopped


(venv) ➜  my_dstack_public git:(master) ✗ dstack pool remove --name main-moody-mayfly-1-0-westeurope
 Pool name  default-pool

 INSTANCE NAME                     BACKEND  INSTANCE TYPE             STATUS  PRICE  
 main-moody-mayfly-1-0-westeurope  azure    2xCPU, 8GB, 100GB (disk)  ready   $0.012 

Remove instance 'main-moody-mayfly-1-0-westeurope'? [y/n]: y
Instance 'main-moody-mayfly-1-0-westeurope' removed


(venv) ➜  my_dstack_public git:(master) ✗ dstack pool show
 Pool name  default-pool

 INSTANCE NAME                     BACKEND  INSTANCE TYPE             STATUS  PRICE  
 main-moody-mayfly-1-0-westeurope  azure    2xCPU, 8GB, 100GB (disk)  ready   $0.012 

Server

DEBUG 2024-02-12T11:20:11.719 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/pool/show in 0.011866s
WARNING 2024-02-12T11:20:14.024 dstack._internal.server.services.pools Couldn't find instance to terminate
DEBUG 2024-02-12T11:20:14.025 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/pool/remove in 0.010520s
INFO 2024-02-12T11:20:22.037 dstack._internal.server.background.tasks.process_pools check instance moody-mayfly-1-0 status: True

There's a problem because the client isn't getting the error message from server

CLI
Instance 'main-moody-mayfly-1-0-westeurope' removed

Server log
WARNING 2024-02-12T11:20:14.024 dstack._internal.server.services.pools Couldn't find instance to terminate

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions