Instance start with failed status #891
Description
- Run
dstack run - Stop the run
dstack pool showshows failed status in the instance- After some time the status changed to the correct status
ready
CLI
(venv) ➜ my_dstack_public git:(master) ✗ dstack run .
Configuration .dstack.yml
Project main
User admin
Pool name default-pool
Min resources 2..xCPU, 8GB..
Max price -
Max duration 6h
Spot policy auto
Retry policy no
# BACKEND REGION INSTANCE RESOURCES SPOT PRICE
1 gcp us-central1 e2-standard-2 2xCPU, 8GB, 100GB yes $0.020103
(disk)
2 gcp us-east1 e2-standard-2 2xCPU, 8GB, 100GB yes $0.020103
(disk)
3 gcp us-central1 e2-highmem-2 2xCPU, 16GB, 100GB yes $0.02712
(disk)
...
Shown 3 of 1064 offers, $55.7395 max
Continue? [y/n]: y
ordinary-firefox-1 provisioning completed (running)
/tmp/vscode-server- 100%[===================>] 48.09M 174MB/s in 0.3s
mkdir: created directory '/root/.vscode-server'
mkdir: created directory '/root/.vscode-server/bin'
mkdir: created directory '/root/.vscode-server/bin/0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2'
Installing extensions...
Installing extension 'ms-python.python'...
Installing extension 'ms-toolsai.jupyter'...
Extension 'ms-toolsai.jupyter' v2023.11.1100101639 was successfully installed.
Extension 'ms-python.python' v2024.0.1 was successfully installed.
pip install ipykernel...
End setup
To open in VS Code Desktop, use link below:
vscode://vscode-remote/ssh-remote+ordinary-firefox-1/workflow
To connect via SSH, use: `ssh ordinary-firefox-1`
To exit, press Ctrl+C.^C
Stop the run before detaching? [y/n]: y
Stopped
(venv) ➜ my_dstack_public git:(master) ✗ dstack pool show
Pool name default-pool
INSTANCE NAME BACKEND INSTANCE TYPE STATUS PRICE
main-ordinary-firefox-1-0 gcp 2xCPU, 8GB, 100GB (disk) failed $0.0201
(venv) ➜ my_dstack_public git:(master) ✗ dstack pool show
Pool name default-pool
INSTANCE NAME BACKEND INSTANCE TYPE STATUS PRICE
main-ordinary-firefox-1-0 gcp 2xCPU, 8GB, 100GB (disk) ready $0.0201 Server
INFO 2024-02-12T10:50:37.046 dstack._internal.server.background.tasks.process_submitted_jobs (a3ec2b)ordinary-firefox-1-0: now is provisioning
DEBUG 2024-02-12T10:50:38.576 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:00:36.820659
DEBUG 2024-02-12T10:50:41.597 dstack._internal.core.services.ssh.tunnel SSH tunnel failed: b'ssh: connect to host 34.135.229.233 port 22: Operation timed out\r\n'
DEBUG 2024-02-12T10:50:41.998 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.022525s
DEBUG 2024-02-12T10:50:42.563 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:00:40.808080
DEBUG 2024-02-12T10:50:45.587 dstack._internal.core.services.ssh.tunnel SSH tunnel failed: b'ssh: connect to host 34.135.229.233 port 22: Operation timed out\r\n'
DEBUG 2024-02-12T10:50:45.760 dstack._internal.core.services.ssh.tunnel SSH tunnel failed: b'ssh: connect to host 34.135.229.233 port 22: Connection refused\r\n'
INFO 2024-02-12T10:50:45.762 dstack._internal.server.background.tasks.process_pools check instance ordinary-firefox-1-0 status: False
DEBUG 2024-02-12T10:50:46.571 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:00:44.816038
DEBUG 2024-02-12T10:50:46.747 dstack._internal.core.services.ssh.tunnel SSH tunnel failed: b'ssh: connect to host 34.135.229.233 port 22: Connection refused\r\n'
DEBUG 2024-02-12T10:50:47.031 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.020856s
DEBUG 2024-02-12T10:50:48.567 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:00:46.811861
DEBUG 2024-02-12T10:50:48.747 dstack._internal.core.services.ssh.tunnel SSH tunnel failed: b'ssh: connect to host 34.135.229.233 port 22: Connection refused\r\n'
DEBUG 2024-02-12T10:50:50.570 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:00:48.814863
DEBUG 2024-02-12T10:50:50.750 dstack._internal.core.services.ssh.tunnel SSH tunnel failed: b'ssh: connect to host 34.135.229.233 port 22: Connection refused\r\n'
DEBUG 2024-02-12T10:50:52.057 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.019651s
DEBUG 2024-02-12T10:50:52.567 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:00:50.812184
DEBUG 2024-02-12T10:50:52.743 dstack._internal.core.services.ssh.tunnel SSH tunnel failed: b'ssh: connect to host 34.135.229.233 port 22: Connection refused\r\n'
DEBUG 2024-02-12T10:50:54.578 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:00:52.823010
DEBUG 2024-02-12T10:50:54.753 dstack._internal.core.services.ssh.tunnel SSH tunnel failed: b'ssh: connect to host 34.135.229.233 port 22: Connection refused\r\n'
DEBUG 2024-02-12T10:50:56.571 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:00:54.816218
DEBUG 2024-02-12T10:50:56.759 dstack._internal.core.services.ssh.tunnel SSH tunnel failed: b'ssh: connect to host 34.135.229.233 port 22: Connection refused\r\n'
DEBUG 2024-02-12T10:50:57.076 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.012314s
DEBUG 2024-02-12T10:50:58.564 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:00:56.808439
DEBUG 2024-02-12T10:50:58.749 dstack._internal.core.services.ssh.tunnel SSH tunnel failed: b'ssh: connect to host 34.135.229.233 port 22: Connection refused\r\n'
DEBUG 2024-02-12T10:51:00.570 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:00:58.814821
DEBUG 2024-02-12T10:51:00.752 dstack._internal.core.services.ssh.tunnel SSH tunnel failed: b'ssh: connect to host 34.135.229.233 port 22: Connection refused\r\n'
DEBUG 2024-02-12T10:51:02.096 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.011652s
DEBUG 2024-02-12T10:51:02.565 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:01:00.809727
DEBUG 2024-02-12T10:51:05.174 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: shim is not available yet
DEBUG 2024-02-12T10:51:06.571 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process provisioning job with shim, age=0:01:04.815888
DEBUG 2024-02-12T10:51:07.111 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.010529s
INFO 2024-02-12T10:51:09.037 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: now is pulling
DEBUG 2024-02-12T10:51:10.570 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process pulling job with shim, age=0:01:08.814462
DEBUG 2024-02-12T10:51:12.130 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.011202s
DEBUG 2024-02-12T10:51:14.578 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process pulling job with shim, age=0:01:12.823208
DEBUG 2024-02-12T10:51:17.151 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.012055s
DEBUG 2024-02-12T10:51:18.564 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process pulling job with shim, age=0:01:16.809240
DEBUG 2024-02-12T10:51:22.167 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.012025s
DEBUG 2024-02-12T10:51:22.567 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process pulling job with shim, age=0:01:20.812204
DEBUG 2024-02-12T10:51:26.572 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process pulling job with shim, age=0:01:24.816469
DEBUG 2024-02-12T10:51:27.183 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.011562s
DEBUG 2024-02-12T10:51:30.570 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process pulling job with shim, age=0:01:28.814800
DEBUG 2024-02-12T10:51:32.206 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.014144s
DEBUG 2024-02-12T10:51:34.579 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process pulling job with shim, age=0:01:32.823731
DEBUG 2024-02-12T10:51:37.228 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.017319s
DEBUG 2024-02-12T10:51:38.563 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process pulling job with shim, age=0:01:36.808003
DEBUG 2024-02-12T10:51:42.571 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process pulling job with shim, age=0:01:40.815813
DEBUG 2024-02-12T10:51:43.263 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.021218s
DEBUG 2024-02-12T10:51:45.051 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: submitting job spec
DEBUG 2024-02-12T10:51:45.051 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: repo credentials are https
DEBUG 2024-02-12T10:51:45.542 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: uploading code
DEBUG 2024-02-12T10:51:47.376 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: starting job
DEBUG 2024-02-12T10:51:48.297 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.022517s
DEBUG 2024-02-12T10:51:48.570 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:01:46.814468
INFO 2024-02-12T10:51:50.779 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: now is running
DEBUG 2024-02-12T10:51:52.568 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:01:50.813312
DEBUG 2024-02-12T10:51:56.565 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:01:54.810233
DEBUG 2024-02-12T10:52:00.570 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:01:58.814671
DEBUG 2024-02-12T10:52:04.576 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:02:02.821204
DEBUG 2024-02-12T10:52:08.569 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:02:06.813746
DEBUG 2024-02-12T10:52:12.570 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:02:10.815217
DEBUG 2024-02-12T10:52:16.566 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:02:14.811117
DEBUG 2024-02-12T10:52:20.567 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:02:18.812225
DEBUG 2024-02-12T10:52:24.575 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:02:22.819925
DEBUG 2024-02-12T10:52:28.569 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:02:26.813512
DEBUG 2024-02-12T10:52:32.571 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:02:30.815980
DEBUG 2024-02-12T10:52:34.565 dstack._internal.server.background.tasks.process_running_jobs (a3ec2b)ordinary-firefox-1-0: process running job, age=0:02:32.809803
DEBUG 2024-02-12T10:52:38.522 dstack._internal.server.services.jobs Stopping runner 34.135.229.233
INFO 2024-02-12T10:52:39.046 dstack._internal.server.services.jobs (a3ec2b)ordinary-firefox-1-0: terminated by user
DEBUG 2024-02-12T10:52:39.050 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/stop in 3.183679s
DEBUG 2024-02-12T10:52:41.092 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/runs/get in 0.020452s
DEBUG 2024-02-12T10:52:44.399 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/pool/list in 0.007524s
DEBUG 2024-02-12T10:52:48.201 dstack._internal.server.app Processed request POST http://127.0.0.1:8000/api/project/main/pool/show in 0.005704s
INFO 2024-02-12T10:52:54.593 dstack._internal.server.background.tasks.process_finished_jobs (a3ec2b)ordinary-firefox-1-0: marked as removed
INFO 2024-02-12T10:53:06.645 dstack._internal.server.background.tasks.process_pools check instance ordinary-firefox-1-0 status: True
INFO 2024-02-12T10:53:16.898