[Bug]: `dstack` shows Community Cloud price but provisions Secure Cloud instances on RunPod

[peterschmidt85]

Steps to reproduce

1. Set up a config

resources:
  gpu: A6000:2

2. Run dstack apply -f examples/.dstack.yml -b runpod --on-demand

Output:

#  BACKEND  REGION   INSTANCE          RESOURCES                                      SPOT  PRICE
 1  runpod   EU-SE-1  NVIDIA RTX A6000  18xCPU, 100GB, 2xA6000 (48GB), 100.0GB (disk)  no    $0.98
 2  runpod   EU-SE-1  NVIDIA RTX A6000  18xCPU, 100GB, 2xA6000 (48GB), 100.0GB (disk)  no    $1.52

3. Say n (cancel)

4. Run dstack apply -f examples/.dstack.yml -b runpod --on-demand --max-price 1

Output:

 #  BACKEND  REGION   INSTANCE          RESOURCES                                      SPOT  PRICE
 1  runpod   EU-SE-1  NVIDIA RTX A6000  18xCPU, 100GB, 2xA6000 (48GB), 100.0GB (disk)  no    $0.98

5. Say y (confirm)

Actual behaviour

It creates an instance that costs $1.52

UPD: yet it believes that it actually created the $0.98 instance:

> dstack ps
 NAME           BACKEND  REGION   RESOURCES                                      SPOT  PRICE  STATUS   SUBMITTED 
 heavy-mouse-1  runpod   EU-SE-1  18xCPU, 100GB, 2xA6000 (48GB), 100.0GB (disk)  no    $0.98  running  1 min ago

Expected behaviour

It should only allow to use the instance that costs $0.98

dstack version

master

Server logs

No response

Additional information

No response

[jvstme]

The problem is specific to RunPod. In this example $0.98 is the Community Cloud price and $1.52 is the Secure Cloud price. dstack believes that it provisioned a $0.98 Community Cloud instance, while in fact it provisioned a $1.52 Secure Cloud instance.

Apparently we always provision Secure Cloud instances, so I just removed all Community Cloud offers from the gpuhunt catalog. This fixes the issue in all existing dstack releases, a new release is not required.

p.s.: Community Cloud offers can offer very competitive prices, so we should look into supporting them properly.

[movchan74]

@jvstme Is it already supposed to work? I've tried it after your last comment about gpuhunt update. But the bug still persist.

dstack showed me this instance when max_price is set to 1.0.

 #  BACKEND  REGION   INSTANCE          RESOURCES                                      SPOT  PRICE   
 1  runpod   EU-SE-1  NVIDIA RTX A6000  18xCPU, 100GB, 2xA6000 (48GB), 100.0GB (disk)  no    $0.98  

But provisioned the one that costs $1.52.

[jvstme]

@movchan74, for self-hosted installations, restart dstack server. For dstack Sky, it could have taken some time for changes to take effect, but it seems to work now.

 Configuration          .dstack.yml                             
 Type                   dev-environment                         
 Resources              2..xCPU, 8GB.., 2xA6000, 100GB.. (disk) 
 Max price              -                                       
 Max duration           6h                                      
 Spot policy            on-demand                               
 Retry policy           no                                      
 Creation policy        reuse-or-create                         
 Termination policy     destroy-after-idle                      
 Termination idle time  5m                                      

 #  BACKEND  REGION   INSTANCE          RESOURCES                                      SPOT  PRICE   
 1  runpod   EU-SE-1  NVIDIA RTX A6000  18xCPU, 100GB, 2xA6000 (48GB), 100.0GB (disk)  no    $1.52

[movchan74]

@movchan74, for self-hosted installations, restart dstack server. For dstack Sky, it could have taken some time for changes to take effect, but it seems to work now.

 Configuration          .dstack.yml                             
 Type                   dev-environment                         
 Resources              2..xCPU, 8GB.., 2xA6000, 100GB.. (disk) 
 Max price              -                                       
 Max duration           6h                                      
 Spot policy            on-demand                               
 Retry policy           no                                      
 Creation policy        reuse-or-create                         
 Termination policy     destroy-after-idle                      
 Termination idle time  5m                                      

 #  BACKEND  REGION   INSTANCE          RESOURCES                                      SPOT  PRICE   
 1  runpod   EU-SE-1  NVIDIA RTX A6000  18xCPU, 100GB, 2xA6000 (48GB), 100.0GB (disk)  no    $1.52

I was using Sky. It is working now.

[jvstme]

Apparently we always provision Secure Cloud instances

This assumption wasn't quite right, sometimes instances are provisioned in Community Cloud. Right now there is no capacity for A40 in Secure Cloud and RunPod provisioned it in Community Cloud, which turned out to be more expensive for some reason.

dstack - $0.78

 #  BACKEND  REGION    INSTANCE    RESOURCES                                    SPOT  PRICE   
 1  runpod   EU-SE-1   NVIDIA A40  18xCPU, 100GB, 2xA40 (48GB), 100.0GB (disk)  no    $0.78

Runpod - $0.94 ($0.47 * 2 GPU)

[jvstme]

I made a fix to only deploy to Secure Cloud on RunPod. The fix will be available in the upcoming dstack 0.18.21 release and will ensure the prices shown in dstack match those in the RunPod console.

Disabling the RunPod Community Cloud is a temporary measure and we will bring it back soon in #1892