Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Cannot delete dead containers with overlay2 on RHEL 7.4 #34538
RedHat Enterprise Linux 7.4 with kernel 3.10.0-693.el7.x86_64 adds support for overlay2. I've switched over to overlay2 in a non-production cluster and it seems to be working ok most of the time. However, it seems dead containers cannot be deleted. The daemon says the underlying device or resource for the filesystem is busy.
Steps to reproduce the issue:
Describe the results you received:
Dead containers (as opposed to containers that exited normally) cannot be removed.
Describe the results you expected:
When I was on the overlay driver (not overlay2), I didn't have any problems removing dead containers, or volumes associated with them.
Additional information you deem important (e.g. issue happens only occasionally):
My impression is that this behaviour is new since switching to the overlay2 driver.
This may be relevant to #34368.
Restarting the node allows the dead containers to be deleted.
Additional environment details (AWS, VirtualBox, physical, etc.):
10 node Docker swarm on RHEL 7.4.
Here's the info for one of the dead containers.
9b5c2b171adf 9ce41916ff3f "/bin/sh -c 'npm i..." 2 days ago Dead peaceful_gates
This is new behavior in 17.06, where 17.03 just ignored the error and leaked data, 17.06 doesn't ignore it anymore. (note that it only ignored the error if you used
Fundamentally what's happened is a mount has leaked into another mount namespace and it's preventing removal.
I'm sorry, can you please provide me with exact steps to fix this issue?