Draining a node does not remove old pods

[jdoss]

While working on a systemd unit to drain my client nodes of jobs before shutdown, I noticed that if you have a job that uses podman pods, it does not remove the pods as part of the drain:

These are left over after a few drains testing the unit.

# podman ps -a
CONTAINER ID  IMAGE                 COMMAND     CREATED       STATUS                         PORTS                                                     NAMES
b6868d8e0485  k8s.gcr.io/pause:3.1              40 hours ago  Exited (0) 37 hours ago        100.71.2.23:30608->8080/tcp, 100.71.2.23:30608->8080/udp  pod-76dd69dc-694c-2549-e988-38f3f4ab4ea2
e462e4822b71  k8s.gcr.io/pause:3.1              37 hours ago  Exited (0) 26 hours ago        100.71.2.23:29984->8080/tcp, 100.71.2.23:29984->8080/udp  pod-67df082b-d322-d5f0-64d2-d097f587e09c
1dae89051a88  k8s.gcr.io/pause:3.1              5 hours ago   Exited (0) About a minute ago  100.71.2.23:25426->8080/tcp, 100.71.2.23:25426->8080/udp  pod-1cbb8973-df27-7f93-dae5-a464bc0f6073

[towe75]

"draining" means nomad node drain, right?

Can you share a example job definition?

[jdoss]

Sorry for the late reply. Yes, I mean nomad node drain. If you use the example https://github.com/hashicorp/nomad-driver-podman/blob/main/examples/jobs/nats_pod.nomad and then issue a nomad node drainit should leave the pod container from the job behind.

If the example doesn't reproduce the issue, I can fix up the job file that I have for Netbox so I can share it, but I believe the nats pod example should reprio the issue as well.