-
Notifications
You must be signed in to change notification settings - Fork 205
Closed
Labels
Description
Steps to reproduce
Get an instance with a few blocks (example: 8).
> cat fleets/cloud.dstack.yml
type: fleet
name: cloud-1
nodes: 1
spot_policy: auto
resources:
cpu: 8..
disk: 300GB..
gpu: 0
blocks: auto
> dstack apply -f fleets/cloud.dstack.yml
FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED
cloud-1 0 datacrunch (FIN-01) 8xCPU, 32GB, 300.0GB (disk), SPOT $0.02 idle 15:46Occupy part of the blocks (example: 6).
> cat services/.dstack.yml
type: service
image: nginx
port: 80
replicas: 6
spot_policy: auto
resources:
cpu: 1..
memory: 1GB..
> dstack apply -f services/.dstack.yml --reuse
NAME BACKEND RESOURCES PRICE STATUS SUBMITTED
dangerous-termite-1 running 15:51
replica=0 job=0 datacrunch (FIN-01) 1xCPU, 4GB, 300.0GB (disk), SPOT $0.02 running 15:51
replica=1 job=0 datacrunch (FIN-01) 1xCPU, 4GB, 300.0GB (disk), SPOT $0.02 running 15:51
replica=2 job=0 datacrunch (FIN-01) 1xCPU, 4GB, 300.0GB (disk), SPOT $0.02 running 15:51
replica=3 job=0 datacrunch (FIN-01) 1xCPU, 4GB, 300.0GB (disk), SPOT $0.02 running 15:51
replica=4 job=0 datacrunch (FIN-01) 1xCPU, 4GB, 300.0GB (disk), SPOT $0.02 running 15:51
replica=5 job=0 datacrunch (FIN-01) 1xCPU, 4GB, 300.0GB (disk), SPOT $0.02 running 15:51Check how many blocks are idle.
> dstack fleetActual behaviour
The blocks counter is incorrect (example: 2 busy, 6 idle).
> dstack fleet
FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED
cloud-1 0 datacrunch (FIN-01) 8xCPU, 32GB, 300.0GB (disk), SPOT $0.02 6/8 idle 45 mins agoIf you then stop the run, at some point there will be more idle blocks than total.
> dstack fleet -w
FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED
cloud-1 0 datacrunch (FIN-01) 8xCPU, 32GB, 300.0GB (disk), SPOT $0.02 11/8 idle 50 mins agoExpected behaviour
The blocks counter is always correct (example: 6 busy, 2 idle).
dstack version
master
Server logs
Additional information
After stopping the run, the busy_blocks counter in the database is negative.
> sqlite3 ~/.dstack/server/data/sqlite.db "SELECT name, total_blocks, busy_blocks FROM instances"
cloud-1-0|8|-4