OverlayFS and 2.6.32-573 kernel

[hra0031]

We are using CentOS 6.7/6.9 with kernel 2.6.32-573/2.6.32-696. This kernel doesn't have overlayfs.ko, but this functionality is really handy (we bind /scratch and /apps directories). This functionality is disabled by default because configure.ac don't find kernel module. When I study the source code I found a way to bypass the settings by setting the environment variable SINGULARITY_OVERLAYFS_ENABLED=1. In this case, OVERLAYFS_ENABLED is propagated to the registry and the functionality is enabled.

$ singularity --debug shell -e --bind /scratch docker://centos:latest
...
DEBUG   [U=4573,P=1123]    singularity_runtime_overlayfs()           Checking if overlayfs should be used
DEBUG   [U=4573,P=1123]    singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(enable overlay, yes)
DEBUG   [U=4573,P=1123]    singularity_config_get_value_impl()       Returning configuration value enable overlay='yes'
DEBUG   [U=4573,P=1123]    singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(enable overlay, yes) = 1
DEBUG   [U=4573,P=1123]    singularity_registry_get()                Returning NULL on 'DISABLE_OVERLAYFS'
DEBUG   [U=4573,P=1123]    singularity_registry_get()                Returning NULL on 'WRITABLE'
VERBOSE [U=4573,P=1123]    singularity_runtime_overlayfs()           OverlayFS not supported by host build
...
DEBUG   [U=4573,P=13971]   singularity_registry_get()                Returning value from registry: 'BINDPATH' = '/scratch,'
DEBUG   [U=4573,P=13971]   singularity_runtime_mount_userbinds()     Checking for 'user bind control' in config
DEBUG   [U=4573,P=13971]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(user bind control, yes)
DEBUG   [U=4573,P=13971]   singularity_config_get_value_impl()       Returning configuration value user bind control='yes'
DEBUG   [U=4573,P=13971]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(user bind control, yes) = 1
DEBUG   [U=4573,P=13971]   singularity_runtime_mount_userbinds()     Parsing SINGULARITY_BINDPATH for user-specified bind mounts.
DEBUG   [U=4573,P=13971]   singularity_runtime_mount_userbinds()     Found bind: /scratch -> container:/scratch
DEBUG   [U=4573,P=13971]   singularity_runtime_mount_userbinds()     Checking if bind point is already mounted: /scratch
DEBUG   [U=4573,P=13971]   singularity_runtime_rootfs()              Returning container_directory: /apps/all/Singularity/2.3-develop/var/singularity/mnt/container
DEBUG   [U=4573,P=13971]   check_mounted()                           Opening /proc/mounts
DEBUG   [U=4573,P=13971]   check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=4573,P=13971]   singularity_registry_get()                Returning value from registry: 'OVERLAYFS_ENABLED' = '1'
VERBOSE [U=4573,P=13971]   singularity_runtime_mount_userbinds()     Creating bind directory on overlay file system: /scratch
...
Singularity centos:latest:/tmp> ls /scratch
backup  sys  temp  work

Version of Singularity:

Singularity 2.3/develop

Expected behavior

OverlayFS shouldn't work in that old kernel.

Actual behavior

It is working!

Steps to reproduce behavior

set env SINGULARITY_OVERLAYFS_ENABLED=1

[vsoch]

ruhroh. @gmkurtzer this probably needs priority attention!

[gmkurtzer]

This is actually working only because you happen to be in a Docker container rather then an image. If you did this in an image, you would get a failure, but when run like this, Docker containers are just directories on the host system, and Singularity will indeed try to create the directory, and it works.

The entire overlay workhorse code is wrapped in an #ifdef that is thus getting skipped and instead printing "OverlayFS not supported by host build".

You are right that you can use the registry to "fool" singularity into doing something that it wasn't intending which runs into a registry change where we can namespace key/values.

[hra0031]

@gmkurtzer thanks for the explanation. Perhaps it would be useful to make this functionality available to users in the future.

[hra0031]

@gmkurtzer I tried it today on singularity image and is working as well. 😆
I created an image from docker using import on my notebook (Ubuntu 16.04):

$ singularity create /tmp/c7.img
$ singularity import c7.img docker://centos:latest
$ singularity shell /tmp/c7.img 
Singularity: Invoking an interactive shell within container...
Singularity c7.img:~> ls /
anaconda-post.log  dev	home  lib64	  media  opt   root  sbin	  srv  tmp  var
bin		   etc	lib   lost+found  mnt	 proc  run   singularity  sys  usr

Next, I transferred this image to cluster and run with bind and writable:

$ singularity --debug shell --bind /scratch --writable c7.img
Singularity c7.img:~> ls /scratch
backup  sys  temp  work

Sure, it doesn't work without writable. So why is this function disabled completely at build time?

[vsoch]

Are you asking why writable works? I would suggest trying to write to places (that you shouldn't have permission to). I don't think adding writable will kill a shell if you aren't root on a resource, but it will only give you "writable" to those places you have permission to write.

[hra0031]

No, the question is different. Singularity check OverlayFS at compile time and in our scenario is this functionality disabled. So any try to bind folder, that not exist inside the container is forbidden. But when I bypass the check I can use this bind inside Docker and Singularity image. Singularity creates this folder inside the image and then mount them. (if the image is writable) So why is this bind option disabled completely when I don't need OverlayFS for this?

[gmkurtzer]

Hi @hra0031,

I think I follow your point, and it is a good question with an unfortunate lack of a good answer. In short, running containers directly via a Docker URI is not the preferred workflow and as a result, the internal code is not optimized for that runtime condition. With that said, this is not a good excuse, so this will be considered a code fix for the next major version. Unfortunately, I can't slip it in as a point release because it will require a significant amount of code changes.

To summarize: Creation of bind points should not be blindly skipped if overlay is not supported or enabled. It is quite possible for these bind points to be created if the file system is a non-persistent user owned directory (e.g. as is when invoking an action against a Docker URI).

Greg

[mr-c]

@gmkurtzer Any update on creating bind points on the fly when a Docker URI is presented?

[DrDaveD]

The idea in issue #1207 would solve this problem.

[dtrudg]

@mr-c there is also a PR on this at #1124 I wrote to satisfy this need at my previous employer. Is this something that is a major need?

[mr-c]

@dctrud Thank you for the pointer. In theory it is, as two CWL runners (the reference runner and Toil) want to mount in many paths that are not part of user provided Docker containers.

Here's a typical singularity call:

/tmp/tmpzog5d46c$ singularity \
    --quiet \
    exec \
    --contain \
    --pid \
    --ipc \
    --bind \
    /tmp/tmpzog5d46c:/var/spool/cwl:rw \
    --bind \
    /tmp/tmpr8dbo03g:/tmp:rw \
    --bind \
    /home/michael/common-workflow-language/v1.0/v1.0/example_human_Illumina.pe_1.fastq:/var/lib/cwl/stg0abe3c95-ddd3-4839-a7cd-eebea180ef72/example_human_Illumina.pe_1.fastq:ro \
    --bind \
    /home/michael/common-workflow-language/v1.0/v1.0/args.py:/var/lib/cwl/stga12039e1-fbc8-4153-9f47-9718d6658866/args.py:ro \
    --bind \
    /home/michael/common-workflow-language/v1.0/v1.0/example_human_Illumina.pe_2.fastq:/var/lib/cwl/stg6064746e-b019-4159-a97c-763d36bcd3ce/example_human_Illumina.pe_2.fastq:ro \
    --bind \
    /home/michael/common-workflow-language/v1.0/v1.0/chr20.fa:/var/lib/cwl/stg7fb2017e-9a6b-4c34-902b-af916eaf8bcd/chr20.fa:ro \
    --pwd \
    /var/spool/cwl \
    /home/michael/cwltool/python-2-slim.img \
    python \
    /var/lib/cwl/stga12039e1-fbc8-4153-9f47-9718d6658866/args.py \
    bwa \
    mem \
    /var/lib/cwl/stg7fb2017e-9a6b-4c34-902b-af916eaf8bcd/chr20.fa \
    -XXX \
    -YYY \
    /var/lib/cwl/stg0abe3c95-ddd3-4839-a7cd-eebea180ef72/example_human_Illumina.pe_1.fastq \
    -YYY \
    /var/lib/cwl/stg6064746e-b019-4159-a97c-763d36bcd3ce/example_human_Illumina.pe_2.fastq

from cwltool --singularity https://github.com/common-workflow-language/common-workflow-language/raw/0c5d88392a3e7cfec8c66d02ac0ec0a1c3ddd675/v1.0/v1.0/binding-test.cwl https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/bwa-mem-job.json

I can canvass our mailing list to inquire further. How shall they indicate their interest, by 👍 on #1124 ?

[mr-c]

@dctrud Oh, is #1124 only for system wide bind paths, or would it work for the example above?

[mr-c]

@dctrud From my perspective, since udocker works everywhere this isn't urgent. For Singularity as a project (product) it may be worth the strategic investment so that sites (potential customers) who don't yet have overlayfs support on their clusters can benefit from the Singularity ecosystem.

[bauerm97]

Hey guys, I think this is solved in the 3.x series since we now support the underlay feature written by @DrDaveD

[hra0031]

@bauerm97 Yes, I think this thread is now obsolete.