podman rm, with more than one arg: error removing container, unlinkat, EBUSY

[edsantiago]

It looks like the cause of our new unlinkat/EBUSY flake is podman rm with more than one argument:

$ while :;do ./bin/podman run --name foo alpine true;./bin/podman run --name bar alpine true;./bin/podman rm foo bar || break;done
....
Error: error removing container 6b1063e2084c626ce8c4c11b02ee9caa8848a4f625a256d02febaa54e448a0c4 root filesystem: 1 error occurred:
        * unlinkat /home/esm/.local/share/containers/storage/overlay/bb7a8f6f9ba33943c30956aca2e90533a6aca86546c48b513fc55407603f83a8/merged: device or resource busy

It can take one minute, but more typically it's about five.

master @ 78df4f6. Running rootless on f34.

[edsantiago]

Got it to fail with --log-level=debug:

DEBU[0000] environment variable TMPDIR is already defined, skip the settings from containers.conf
INFO[0000] Setting parallel job count to 25
DEBU[0000] Starting parallel job on container 2e7fd598ead121a836c788756db812685e3daa9f2698dfc08c21bfca32ed9d9f
DEBU[0000] Starting parallel job on container b2097e1c0ef4716bf6dc0d0f0871511a78a2effa1ab6d2aa03754ef985fe8ae3
DEBU[0000] Removing container b2097e1c0ef4716bf6dc0d0f0871511a78a2effa1ab6d2aa03754ef985fe8ae3
DEBU[0000] Removing all exec sessions for container b2097e1c0ef4716bf6dc0d0f0871511a78a2effa1ab6d2aa03754ef985fe8ae3
DEBU[0000] Cleaning up container b2097e1c0ef4716bf6dc0d0f0871511a78a2effa1ab6d2aa03754ef985fe8ae3
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container b2097e1c0ef4716bf6dc0d0f0871511a78a2effa1ab6d2aa03754ef985fe8ae3 storage is already unmounted, skipping...
DEBU[0000] Container b2097e1c0ef4716bf6dc0d0f0871511a78a2effa1ab6d2aa03754ef985fe8ae3 storage is already unmounted, skipping...
DEBU[0010] failed to delete container "b2097e1c0ef4716bf6dc0d0f0871511a78a2effa1ab6d2aa03754ef985fe8ae3": 1 error occurred:
        * unlinkat /home/esm/.local/share/containers/storage/overlay/d7c83d85fbae9140223161f715bd606cf051ac591ade4d6f6bb833b43313676f/merged: device or resource busy

DEBU[0010] Failed to remove container b2097e1c0ef4716bf6dc0d0f0871511a78a2effa1ab6d2aa03754ef985fe8ae3: error removing container b2097e1c0ef4716bf6dc0d0f0871511a78a2effa1ab6d2aa03754ef985fe8ae3 root filesystem: 1 error occurred:
        * unlinkat /home/esm/.local/share/containers/storage/overlay/d7c83d85fbae9140223161f715bd606cf051ac591ade4d6f6bb833b43313676f/merged: device or resource busy

DEBU[0010] Removing container 2e7fd598ead121a836c788756db812685e3daa9f2698dfc08c21bfca32ed9d9f
DEBU[0010] Removing all exec sessions for container 2e7fd598ead121a836c788756db812685e3daa9f2698dfc08c21bfca32ed9d9f
DEBU[0010] Cleaning up container 2e7fd598ead121a836c788756db812685e3daa9f2698dfc08c21bfca32ed9d9f
DEBU[0010] Network is already cleaned up, skipping...
DEBU[0010] Container 2e7fd598ead121a836c788756db812685e3daa9f2698dfc08c21bfca32ed9d9f storage is already unmounted, skipping...
DEBU[0010] Container 2e7fd598ead121a836c788756db812685e3daa9f2698dfc08c21bfca32ed9d9f storage is already unmounted, skipping...
2e7fd598ead121a836c788756db812685e3daa9f2698dfc08c21bfca32ed9d9f
Error: error removing container b2097e1c0ef4716bf6dc0d0f0871511a78a2effa1ab6d2aa03754ef985fe8ae3 root filesystem: 1 error occurred:
        * unlinkat /home/esm/.local/share/containers/storage/overlay/d7c83d85fbae9140223161f715bd606cf051ac591ade4d6f6bb833b43313676f/merged: device or resource busy

[rhatdan]

@mheon I believe this could happen if the c.state.State == define.ContainerStateStopped and you attempt to remove it. Should we wait in the condition? If we wait, will the status be able to change?

[mheon]

I don't think that's possible - the container is already locked, there's no potential for a race.

DEBU[0010] Container 2e7fd598ead121a836c788756db812685e3daa9f2698dfc08c21bfca32ed9d9f storage is already unmounted, skipping...

That right there is extremely suspicious. The container is marked as unmounted in the DB, yet clearly it's still mounted because it's failing. I'm not sure how this happens - maybe the cleanup process already fired and failed to unmount?

[rhatdan]

@edsantiago could you attempt to remove the container afterwards to prove this is a race.

I was thinking the container could be stopped but not cleaned up yet, and we go in and attempt to remove it.

[edsantiago]

I'm not sure what you mean by "remove the container afterwards" -- in all cases, both containers are gone: the rm is successful, at least as far as I can tell. podman ps -a always shows empty. Is that what you're asking?

[mheon]

@rhatdan We clean up as part of removal - you can see it in the logs.

The container is guaranteed to be gone after podman rm runs - if we're leaving anything behind, it's a storage container.

[rhatdan]

So we don't know if there is any storage container left? If there is a failure on removing from storage, should we remove from the database?

[rhatdan]

The only reason that the image would be busy, would be if something is mounted on the image or if a process is running on the image.

[mheon]

@rhatdan It's much better to get rid of the containers, even if there's a storage container left over - we had so many issues in the early days with containers that could not be removed at all without manual intervention. Defaulting to getting rid of the container is much safer.

[rhatdan]

Ok but any thoughts on what is happening?

[mheon]

Initial thought: cleanup process has already fired but failed to clean up, and the container was marked as unmounted despite this failure. As to why: lingering container process using the mountpoint, maybe? We really need to get the initial error message on the unmount attempt. Does this replicate outside of CI?

[edsantiago]

Does this replicate outside of CI?

Trivially. See initial comment for one-line reproducer.