whiteout files present in singularity but not in docker container

[kaczmarj]

Version of Singularity:

$ singularity --version
2.3.1-HEAD.ge214d4e

$ docker --version
Docker version 17.07.0-ce-rc3, build 665d244

Expected behavior

Consistency in presence of whiteout files between Docker and Singularity.

Actual behavior

Whiteout files (e.g. ,.wh..wh..opq) are present in the Singularity container (pulled from DockerHub) but not in the Docker container.

Steps to reproduce behavior

Singularity

$ singularity pull docker://tensorflow/tensorflow:1.3.0-gpu-py3
$ singularity shell --containall --shell bash tensorflow-1.3.0-gpu-py3.img
(container)$ find / -name ".wh.*" 2>/dev/null | grep -v "Permission denied"

Docker

$ docker run --rm -it --entrypoint bash tensorflow/tensorflow:1.3.0-gpu-py3
(container)$ find / -name ".wh.*" 2>/dev/null | grep -v "Permission denied"

related to #571

[vsoch]

Did you try the singularity commands using sudo? It could be permissions on pull / run (as Docker has write permissions). I'm not familiar with whiteout files - could you talk a bit about how/why they exist? For the longest time I thought those were python wheels :)

[vsoch]

I just tried with sudo and they indeed persist, I wonder why this happens? Do you think it would be appropriate for singularity to do a sweep of a container, always, and remove?

find / -name ".wh.*" -exec rm -f {} \;

[kaczmarj]

I know very little about whiteout files, but they seem to be part of the overlay filesystem. I found these resources (linux kernel docs and opencontainers layer spec) useful.

Here is a quote from opencontainers:

An opaque whiteout entry is a file with the name .wh..wh..opq indicating that all siblings are hidden in the lower layer.

So I am not sure whether it is safe to remove these files. I will try to dig into the docker source to see what they do with the opaque whiteout files. For what it's worth, the directories that contain whiteout files in singularity are present when running with docker (ie the directories+files are not hidden).

[vsoch]

ok, thanks for looking into this! It's definitely a big issue for some of our users.

[chrisgorgo]

I just run into this with 2.4. Looking forward to the fix!

[vsoch]

@chrisfilo see the PR I linked - if you have some bandwidth please give it a try and report back! Travis is having network issues for pulling docker images, but the local testing is far more important for this change.

[chrisgorgo]

Has this issue been resolved by #1181?

[dtrudg]

Closing this as PRs fixing whiteout issues have been merged to development. Let us know if there are any remaining problems. Thanks!