Skip to content

[Bug]: Shim fails on /api/instance/health if libdcgm.so is not installed #2979

@un-def

Description

@un-def

Steps to reproduce

  1. Make sure libdcgm.so.4 is not installed
  2. Run curl http://localhost:10998/api/instance/health on the instance
  3. Check /root/.dstack/shim.log

Actual behaviour

Null pointer dereference:

[ 68.516645] cloud-init[1852]: created by net/http.(*Server).Serve in goroutine 1
13 Aug 11:42:07.000
[ 68.516534] cloud-init[1852]: /opt/hostedtoolcache/go/1.23.8/x64/src/net/http/server.go:2092 +0x5d0
13 Aug 11:42:07.872
[ 68.516421] cloud-init[1852]: net/http.(*conn).serve(0xc00031b0e0, {0xb19828, 0xc0000bd7d0})
13 Aug 11:42:07.992
[ 68.516307] cloud-init[1852]: /opt/hostedtoolcache/go/1.23.8/x64/src/net/http/server.go:3210 +0x8e
13 Aug 11:42:07.976
[ 68.516196] cloud-init[1852]: net/http.serverHandler.ServeHTTP({0xc00012ea80?}, {0xb18858?, 0xc000276460?}, 0x6?)
13 Aug 11:42:07.040
[ 68.515966] cloud-init[1852]: /opt/hostedtoolcache/go/1.23.8/x64/src/net/http/server.go:2747 +0x1ca
13 Aug 11:42:07.936
[ 68.515855] cloud-init[1852]: net/http.(*ServeMux).ServeHTTP(0xc0003d9b88?, {0xb18858, 0xc000276460}, 0xc00038cb40)
13 Aug 11:42:07.080
[ 68.515741] cloud-init[1852]: /opt/hostedtoolcache/go/1.23.8/x64/src/net/http/server.go:2220 +0x29
13 Aug 11:42:07.032
[ 68.515626] cloud-init[1852]: net/http.HandlerFunc.ServeHTTP(0xc0002760e0?, {0xb18858?, 0xc000276460?}, 0xc0003d9b38?)
13 Aug 11:42:07.928
[ 68.515514] cloud-init[1852]: /home/runner/work/dstack/dstack/runner/internal/api/common.go:111 +0x77
13 Aug 11:42:07.968
[ 68.515399] cloud-init[1852]: github.com/dstackai/dstack/runner/internal/api.(*Router).AddHandler.JSONResponseHandler.func1({0xb18858, 0xc000276460}, 0xc00038cb40)
13 Aug 11:42:07.984
[ 68.515293] cloud-init[1852]: /home/runner/work/dstack/dstack/runner/internal/shim/api/handlers.go:28 +0x95
13 Aug 11:42:07.096
[ 68.515177] cloud-init[1852]: github.com/dstackai/dstack/runner/internal/shim/api.(*ShimServer).InstanceHealthHandler(0xc000328180, {0x8?, 0x9848e0?}, 0xc000122501?)
13 Aug 11:42:07.112
[ 68.515061] cloud-init[1852]: /home/runner/work/dstack/dstack/runner/internal/shim/dcgm/wrapper_linux.go:77 +0x1d
13 Aug 11:42:07.040
[ 68.514948] cloud-init[1852]: github.com/dstackai/dstack/runner/internal/shim/dcgm.(*DCGMWrapper).GetHealth(0x14?)
13 Aug 11:42:07.088
[ 68.514846] cloud-init[1852]: /opt/hostedtoolcache/go/1.23.8/x64/src/runtime/panic.go:791 +0x132
13 Aug 11:42:07.048
[ 68.514730] cloud-init[1852]: panic({0x98fda0?, 0xe73240?})
13 Aug 11:42:07.888
[ 68.514589] cloud-init[1852]: /opt/hostedtoolcache/go/1.23.8/x64/src/net/http/server.go:1947 +0xbe
13 Aug 11:42:07.016
[ 68.514474] cloud-init[1852]: net/http.(*conn).serve.func1()
13 Aug 11:42:07.896
[ 68.514355] cloud-init[1852]: goroutine 27 [running]:
13 Aug 11:42:07.080
[ 68.514263] cloud-init[1852]: 2025/08/13 09:42:07 http: panic serving 127.0.0.1:52508: runtime error: invalid memory address or nil pointer dereference

Expected behaviour

No response

dstack version

0.19.23

Server logs

Additional information

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingmajor

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions