panic: during agent shutdown with drain_on_shutdown, got this panic

[shoenig]

While investigating #17299 I got this panic during shutdown. Probably not the root cause of that issue, but something we should fix even if it is only during agent shutdown.

Jun 06 15:46:15 ip-172-31-24-55 nomad[2446]: ==> Caught signal: interrupt
Jun 06 15:46:15 ip-172-31-24-55 nomad[2446]: ==> Gracefully shutting down agent...
Jun 06 15:46:15 ip-172-31-24-55 nomad[2446]:     2023-06-06T15:46:15.336Z [INFO]  client.drain: monitoring self-drain
Jun 06 15:46:15 ip-172-31-24-55 systemd[1]: Stopping Nomad...
Jun 06 15:46:15 ip-172-31-24-55 nomad[2446]:     2023-06-06T15:46:15.930Z [ERROR] client.driver_mgr.docker: failed to inspect container: container_id=43dbf2b5fc31860b5987f7347bf2a12>
Jun 06 15:46:15 ip-172-31-24-55 nomad[2446]:     2023-06-06T15:46:15.931Z [ERROR] client.driver_mgr.docker: error stopping container: container_id=43dbf2b5fc31860b5987f7347bf2a12839>
Jun 06 15:46:15 ip-172-31-24-55 nomad[2446]:     2023-06-06T15:46:15.931Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=b94d9ebd-2c76-955c-a1d7-a681046e4e61 task=red>
Jun 06 15:46:15 ip-172-31-24-55 nomad[2446]:     2023-06-06T15:46:15.934Z [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pi>
Jun 06 15:46:15 ip-172-31-24-55 nomad[2446]:     2023-06-06T15:46:15.935Z [INFO]  client.alloc_runner.task_runner: restarting task: alloc_id=b94d9ebd-2c76-955c-a1d7-a681046e4e61 tas>
Jun 06 15:46:15 ip-172-31-24-55 nomad[2446]:     2023-06-06T15:46:15.935Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=b94d9ebd-2c76-955c-a1d7-a681046e4e61 task=red>
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]:     2023-06-06T15:46:16.348Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=b94d9ebd-2c76-955c-a1d7-a681046e4e61 task=red>
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]:     2023-06-06T15:46:16.359Z [INFO]  client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=b94d9ebd-2c76-955>
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]: panic: runtime error: invalid memory address or nil pointer dereference
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x180f59e]
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]: goroutine 359 [running]:
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]: github.com/hashicorp/nomad/client/lib/cgutil.(*cpusetManagerV2).cleanup.func1({0xc0008814a0, 0x1a}, {0x0?, 0x0?}, {0x523805?, 0xc000ed6f>
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]:         github.com/hashicorp/nomad/client/lib/cgutil/cpuset_manager_v2.go:248 +0x3e
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]: path/filepath.WalkDir({0xc0008814a0, 0x1a}, 0xc000a9da58)
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]:         path/filepath/path.go:533 +0x50
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]: github.com/hashicorp/nomad/client/lib/cgutil.(*cpusetManagerV2).cleanup(0xc000a7fa40)
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]:         github.com/hashicorp/nomad/client/lib/cgutil/cpuset_manager_v2.go:246 +0x209
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]: github.com/hashicorp/nomad/client/lib/cgutil.(*cpusetManagerV2).RemoveAlloc(0xc000a7fa40, {0xc000aa6270, 0x24})
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]:         github.com/hashicorp/nomad/client/lib/cgutil/cpuset_manager_v2.go:177 +0x2d5
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]: github.com/hashicorp/nomad/client/allocrunner.(*cgroupHook).Postrun(0xc000a9de08?)
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]:         github.com/hashicorp/nomad/client/allocrunner/cgroup_hook.go:30 +0x2d
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).postrun(0xc000545680)
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner_hooks.go:250 +0x3ae
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).Run(0xc000545680)
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:356 +0x27d
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]: created by github.com/hashicorp/nomad/client.(*Client).addAlloc
Jun 06 15:46:16 ip-172-31-24-55 nomad[2446]:         github.com/hashicorp/nomad/client/client.go:2633 +0xb2d
Jun 06 15:46:16 ip-172-31-24-55 systemd[1]: nomad.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 06 15:46:16 ip-172-31-24-55 systemd[1]: nomad.service: Failed with result 'exit-code'.