Skip to content

[Bug]: dstack-shim.service fails to start on system reboot #1476

@un-def

Description

@un-def

Steps to reproduce

  1. Create a fleet with an on-prem server
  2. Reboot this server

Actual behaviour

dstack-shim.service fails to start

Expected behaviour

dstack-shim.service starts

dstack version

0.18.7

Server logs

No response

Additional information

$ journalctl -u dstack-shim --boot=0
Jul 31 04:45:49 defpc systemd[1]: Started dstack-shim.
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Scheduled restart job, restart counter is at 1.
Jul 31 04:45:49 defpc systemd[1]: Stopped dstack-shim.
Jul 31 04:45:49 defpc systemd[1]: Started dstack-shim.
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Scheduled restart job, restart counter is at 2.
Jul 31 04:45:49 defpc systemd[1]: Stopped dstack-shim.
Jul 31 04:45:49 defpc systemd[1]: Started dstack-shim.
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Scheduled restart job, restart counter is at 3.
Jul 31 04:45:50 defpc systemd[1]: Stopped dstack-shim.
Jul 31 04:45:50 defpc systemd[1]: Started dstack-shim.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Scheduled restart job, restart counter is at 4.
Jul 31 04:45:50 defpc systemd[1]: Stopped dstack-shim.
Jul 31 04:45:50 defpc systemd[1]: Started dstack-shim.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Scheduled restart job, restart counter is at 5.
Jul 31 04:45:50 defpc systemd[1]: Stopped dstack-shim.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Start request repeated too quickly.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:50 defpc systemd[1]: Failed to start dstack-shim.
$ journalctl --boot=0 | grep -E 'Reached target Network|dstack-shim'
Jul 31 04:45:49 defpc systemd[1]: Reached target Network.
Jul 31 04:45:49 defpc systemd[1]: Started dstack-shim.
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Scheduled restart job, restart counter is at 1.
Jul 31 04:45:49 defpc systemd[1]: Stopped dstack-shim.
Jul 31 04:45:49 defpc systemd[1]: Started dstack-shim.
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Scheduled restart job, restart counter is at 2.
Jul 31 04:45:49 defpc systemd[1]: Stopped dstack-shim.
Jul 31 04:45:49 defpc systemd[1]: Started dstack-shim.
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 04:45:49 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Scheduled restart job, restart counter is at 3.
Jul 31 04:45:50 defpc systemd[1]: Stopped dstack-shim.
Jul 31 04:45:50 defpc systemd[1]: Started dstack-shim.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Scheduled restart job, restart counter is at 4.
Jul 31 04:45:50 defpc systemd[1]: Stopped dstack-shim.
Jul 31 04:45:50 defpc systemd[1]: Started dstack-shim.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Scheduled restart job, restart counter is at 5.
Jul 31 04:45:50 defpc systemd[1]: Stopped dstack-shim.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Start request repeated too quickly.
Jul 31 04:45:50 defpc systemd[1]: dstack-shim.service: Failed with result 'exit-code'.
Jul 31 04:45:50 defpc systemd[1]: Failed to start dstack-shim.
Jul 31 04:45:53 defpc systemd[1]: Reached target Network is Online.
# tail -5 /root/.dstack/shim.log
[runner.go:37 shim.(*CLIArgs).DownloadRunner] [runner.go:92 shim.downloadRunner] Get "https://dstack-runner-downloads-stgn.s3.eu-west-1.amazonaws.com/latest/binaries/dstack-runner-linux-amd64": dial tcp: lookup dstack-runner-downloads-stgn.s3.eu-west-1.amazonaws.com on 127.0.0.53:53: server misbehaving
2024/07/31 04:45:50 Downloading runner from https://dstack-runner-downloads-stgn.s3.eu-west-1.amazonaws.com/latest/binaries/dstack-runner-linux-amd64
[runner.go:37 shim.(*CLIArgs).DownloadRunner] [runner.go:92 shim.downloadRunner] Get "https://dstack-runner-downloads-stgn.s3.eu-west-1.amazonaws.com/latest/binaries/dstack-runner-linux-amd64": dial tcp: lookup dstack-runner-downloads-stgn.s3.eu-west-1.amazonaws.com on 127.0.0.53:53: server misbehaving
2024/07/31 04:45:50 Downloading runner from https://dstack-runner-downloads-stgn.s3.eu-west-1.amazonaws.com/latest/binaries/dstack-runner-linux-amd64
[runner.go:37 shim.(*CLIArgs).DownloadRunner] [runner.go:92 shim.downloadRunner] Get "https://dstack-runner-downloads-stgn.s3.eu-west-1.amazonaws.com/latest/binaries/dstack-runner-linux-amd64": dial tcp: lookup dstack-runner-downloads-stgn.s3.eu-west-1.amazonaws.com on 127.0.0.53:53: server misbehaving

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions