Skip to content

[Bug]: Incorrect idle blocks counter #2274

@jvstme

Description

@jvstme

Steps to reproduce

Get an instance with a few blocks (example: 8).

> cat fleets/cloud.dstack.yml 
type: fleet
name: cloud-1
nodes: 1
spot_policy: auto
resources:
  cpu: 8..
  disk: 300GB..
  gpu: 0
blocks: auto

> dstack apply -f fleets/cloud.dstack.yml 
 FLEET    INSTANCE  BACKEND              RESOURCES                          PRICE  STATUS  CREATED 
 cloud-1  0         datacrunch (FIN-01)  8xCPU, 32GB, 300.0GB (disk), SPOT  $0.02  idle    15:46

Occupy part of the blocks (example: 6).

> cat services/.dstack.yml 
type: service
image: nginx
port: 80
replicas: 6
spot_policy: auto
resources:
  cpu: 1..
  memory: 1GB..

> dstack apply -f services/.dstack.yml --reuse
 NAME                 BACKEND              RESOURCES                         PRICE  STATUS   SUBMITTED 
 dangerous-termite-1                                                                running  15:51     
   replica=0 job=0    datacrunch (FIN-01)  1xCPU, 4GB, 300.0GB (disk), SPOT  $0.02  running  15:51     
   replica=1 job=0    datacrunch (FIN-01)  1xCPU, 4GB, 300.0GB (disk), SPOT  $0.02  running  15:51     
   replica=2 job=0    datacrunch (FIN-01)  1xCPU, 4GB, 300.0GB (disk), SPOT  $0.02  running  15:51     
   replica=3 job=0    datacrunch (FIN-01)  1xCPU, 4GB, 300.0GB (disk), SPOT  $0.02  running  15:51     
   replica=4 job=0    datacrunch (FIN-01)  1xCPU, 4GB, 300.0GB (disk), SPOT  $0.02  running  15:51     
   replica=5 job=0    datacrunch (FIN-01)  1xCPU, 4GB, 300.0GB (disk), SPOT  $0.02  running  15:51

Check how many blocks are idle.

> dstack fleet

Actual behaviour

The blocks counter is incorrect (example: 2 busy, 6 idle).

> dstack fleet
 FLEET    INSTANCE  BACKEND              RESOURCES                          PRICE   STATUS    CREATED     
 cloud-1  0         datacrunch (FIN-01)  8xCPU, 32GB, 300.0GB (disk), SPOT  $0.02   6/8 idle  45 mins ago

If you then stop the run, at some point there will be more idle blocks than total.

> dstack fleet -w
 FLEET    INSTANCE  BACKEND              RESOURCES                          PRICE   STATUS     CREATED     
 cloud-1  0         datacrunch (FIN-01)  8xCPU, 32GB, 300.0GB (disk), SPOT  $0.02   11/8 idle  50 mins ago

Expected behaviour

The blocks counter is always correct (example: 6 busy, 2 idle).

dstack version

master

Server logs

Additional information

After stopping the run, the busy_blocks counter in the database is negative.

> sqlite3 ~/.dstack/server/data/sqlite.db "SELECT name, total_blocks, busy_blocks FROM instances"
cloud-1-0|8|-4

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingfleets

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions