created loop devices do not appear in container even when run privileged

[deitch]

Description

Mounting a loopback device in a container requires CAP=SYS_ADMIN or even better --privileged. However, the /dev/loop* do not appear in the container when created dynamically.

This is particularly a problem when manipulating disk .img files inside a container.

I am using ubuntu:16.04 as a base to reproduce, but this should be the same with any base.

Steps to reproduce the issue:

1. docker run --rm -it --privileged ubuntu:16.04 bash
2. In container: apt update && apt install -y gdisk
3. In container: make a 500MB disk filedd if=/dev/zero of=/ofile.img bs=1M count=500
4. In container: make partitions in the disksgdisk -n 1:2048:194559 -n 2:195560:300000 /ofile.img
5. In container: scan file and create loop devices for partitions:losetup -f /ofile.img -P --show
6. In container: see that partition files have not been createdls -l /dev/loop*
7. In host: see that partition files _have* been created: ls -l /dev/loop*

Describe the results you received:
In container:

/dev/loop0
/dev/loop1
...

On host:

/dev/loop0
/dev/loop0p1
/dev/loop0p2
/dev/loop1
...

Essentially, when running --privileged, docker copies the /dev/ files over to the container, but does not keep them updated.

Same problem exists if you have the files before and then run losetup -D (on host or in container): on host the /dev/loop0p1 etc. files are gone, but linger in container.

Describe the results you expected:
When in privileged mode (or cap=sys_admin), since mounting is possible, devices should be synced up where relevant.

Output of docker version:

Docker version 1.12.1, build 23cf638

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 735
Server Version: 1.12.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 365
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null bridge host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-42-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.674 GiB
Name: avi-Latitude-E6320
ID: SVT2:XSJY:6LYB:O4NB:NGZO:JTXJ:MVPZ:Z2BG:JOPO:IZXS:XD2M:5FCN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical laptop running Ubuntu.

[thaJeztah]

/cc @cpuguy83

[justincormack]

This works ok if you do docker run -v /dev:/dev --privileged .... Otherwise nothing is creating the new device nodes in the container, as the device nodes are created by one of the hotplug mechanisms on the host (usually). You can create them yourself with mknod and it will work.

(You might also be able to mount devtmpfs in the container, which is one way to get device population. Device hotpluf was not really designed for the container use case alas).

[deitch]

This works ok if you do docker run -v /dev:/dev --privileged

I thought about that, but was concerned about odd side effects

You can create them yourself with mknod and it will work.

That is exactly what I worked around. Since I am using losetup ... --show, I know the "parent" loop device, so I can do lsblk <parent>, can find the "children", and then if [[ ! -e $CHILD ]]; then mknod.... But it definitely is a tacky workaround.

Device hotpluf was not really designed for the container use case alas

Indeed, that is the crux of the problem. It all is event driven, is it not? Until such time as it is properly container-friendly at the Linux level, could the Docker daemon not listen for device events as well and update the files in the container if /dev is not volume mounted and the container has CAP_SYS_ADMIN or is privileged?

[deitch]

FYI, same problems occur if you use kpartx which, instead of building devices in /dev/loopXpY uses /dev/mapper/loopXpY, which is of course a symlink to /dev/dm-Z.

Actually, try doing this like LVM volume mounting, or cryptsetup, same device mapper problem. And bind-mounting /dev/mapper/ won't work, because those are just symlinks up to /dev/dm-Z devices (usually).

You are right: hotplug devices need to be solved across the board.

[justincormack]

I don't know if there is a better solution at present than sharing /dev. Using devtmpfs in the container would have much the same effect (ie showing all the host devices). The kernel has no current model that devices might "belong" to one container as there is currently no namespacing for device nodes, so you will just get them all anyway, and if you have privileged or mknod capability you can create them, and if you have sys_admin permission you can mount loop devices that were created in other containers.

What is your use case exactly by the way?

[deitch]

Yeah, I know, there really is no easy answer.

What is your use case exactly by the way?

I am building disk images in a container. I used to do it with packer, but that was a pain: booting a VM, no layers for minor changes, very slow, uploads, full OS install, etc.

Now I can create an image, install all of my prerequisites (like any build env), then I use dd if=/dev/zero of=somefile.img bs=... to make an image file (the directory for the image file is volume-mounted so I can get my artifacts), sgdisk to create GPT and partitions, loopback-mount them, then put on any data I want, from basic data all the way up to a full OS build.

Almost everything works fine, but once you start trying to mount loopback volumes, encrypt them, use lvm, it all gets impossible without --privileged and the devices become very difficult.

It is a pity that containers aren't built natively for this, because this is a great use case.

And I guess the LDN team is fully integrated now; @justincormack is answering the issues. :-)

[justincormack]

Yes, it is definitely a common use case that needs better tooling.

Doing the --privileged thing (or the right capabilties) is certainly the easiest route. It assumes that your Docker host has the right modules though, as effectively you are outsourcing part of your code to the kernel.

The longer term solution to this is probably lkl https://github.com/lkl/linux which lets you use Linux code as a userspace library, so you can do the whole of filesystem creation unprivileged without any kernel support (and even on Windows or OSX). They do have some examples with filesystems, but I am not sure how easy it is for your type of use case (at least at one point the support for dm was difficult to use).

An ugly, slow and complicated but working solution is to use qemu to boot a special purpose Linux that just sets up devices as you want on virtual block devices that are just files. If you cut it down you can probably boot qemu in under a minute in a container, and have it do its work and exit.

I am interested in this as we have similar issues here. Most of my use cases so far have been a bit simpler, but this is unlikely to remain so. I might be interested in working on a Go binding to lkl for example to make it easy to build images programmatically...

[deitch]

effectively you are outsourcing part of your code to the kernel.

And thus by definition losing the whole portability of the build. But I don't have a much better solution right now.

The longer term solution to this is probably lkl

Thanks for the reference, just checked it out. Yes. I really should be able to mount a block device in a directory personally mountable to me (e.g. $HOME/mnt/) working entirely in userspace. I get that there are kernel impacts, so perhaps a full mount might not work, but I certainly should be able to operate on a device as if it is mounted.

This is even more true when the "block device" is just a file .img or .iso, and the file is in space accessible to me.

use qemu to boot a special purpose Linux that just sets up devices as you want on virtual block devices that are just files

Interesting. Happen to have any examples? I kind of dread doing qemu like this, but that probably is an irrational fear. :-)

I am interested in this as we have similar issues here.

As am I. Do let me know. Wish I had more time to dig more deeply, but day jobs and all that. :-)

[justincormack]

Ah and I forgot about libguestfs, which already uses qemu to do this for you http://libguestfs.org/

They are looking at using lkl see https://rwmj.wordpress.com/2015/11/07/linux-kernel-library-backend-for-libguestfs/ as it is faster, but it looks like it might be sufficient for most use cases already.

(edit: there appear to be a couple of docker images with libguestfs, the upstream don't seem to have one, it seems to expect to use the host kernel for qemu, looks like the packages install a distro kernel. Might make a smaller alpine version).

[deitch]

That is interesting. C lib and/or CLI for mounting anything from userspace, by launching a tightly controlled VM using qemu (why does macOS always try to auto-correct "qemu" to "emu"? "gnu" is an animal, not "qemu"!). I still feel like I shouldn't have to go the VM route to get filesystem-in-a-partition-in-a-file access, but I get that I do for now.

[deitch]

@justincormack could this be a cause of filesystem corruption? I do the following (shorthand):

1. docker run -v /dev:/dev -v /path/to/image.img:/image.img --privileged ubuntu:16.04 bash (or pick your distro)
2. In the container, dd if=/dev/zero of=/image.img bs=1M count=2048 (or whatever size image you want)
3. sgdisk to create a GPT with three partitions
4. losetup -f /image.img -P --show to get the partitions mapped to loopback devices
5. mkfs.<whatever> to make filesystems
6. mount /dev/loop0p1 /mnt, etc.
7. copy lots of data in
8. umount
9. losetup -d /image.img to remove from loopbacks
10. Outside the container, losetup -f image.img -P --show.

Sometimes, when I fsck /dev/loop0p1 (or p2, etc.), I end up with filesystem corruption. It is frequent, but not consistent. Yet, with that clean list of activities - all scripted, no human process error or fat-fingers - it should not have any issue. Sometimes, it even corrupts if I do fsck -y to each before step 9.

Could it have anything to do with confusion among the attached devices and inside/outside the container?

[justincormack]

I don't think that is the reason.

Can you try adding --make-private on the first mount? It might help.
Otherwise look at lsof on host and see if something else opens the device.

On 10 Nov 2016 8:07 a.m., "Avi Deitcher" notifications@github.com wrote:

@justincormack https://github.com/justincormack could this be a cause
of filesystem corruption? I do the following (shorthand):

1. docker run -v /dev:/dev -v /path/to/image.img:/image.img
   --privileged ubuntu:16.04 bash (or pick your distro)
2. In the container, dd if=/dev/zero of=/image.img bs=1M count=2048
   (or whatever size image you want)
3. sgdisk to create a GPT with three partitions
4. losetup -f /image.img -P --show to get the partitions mapped to
   loopback devices
5. mkfs. to make filesystems
6. mount /dev/loop0p1 /mnt, etc.
7. copy lots of data in
8. umount
9. losetup -d /image.img to remove from loopbacks
10. Outside the container, losetup -f image.img -P --show.

Sometimes, when I fsck /dev/loop0p1 (or p2, etc.), I end up with
filesystem corruption. It is frequent, but not consistent. Yet, with
that clean list of activities - all scripted, no human process error or
fat-fingers - it should not have any issue. Sometimes, it even corrupts if
I do fsck -y to each before step 9.

Could it have anything to do with confusion among the attached devices and
inside/outside the container?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#27886 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdcPMzv85v67VAP8ATXl2uenTmWc8byks5q8sKcgaJpZM4KkWXU
.

[deitch]

I didn't think it was either, but I thought it might be possible. After all, /dev is mapped inside and outside, the container is --privileged, and it might be confused about what is happening.

I do not like that it has the device files in and out; too easy for corruption to occur. Clearly, containers did not (yet) plan for this use case, even though we made a reasonable argument for it. Personally, I also could say physical block devices mapped directly into a container, the same way that a physical network device might be dedicated to one. The OS does a good job with moving a network device in; block, not really.

I will try it with --make-private, but I cannot consistently reproduce it.

It is strange. To be safe, right after umounting, I do fsck -y on all of the filesystems, and sometimes I still get corruption afterwards. I also sometimes see messages about corrupt back GPT table.

[ybizeul]

My small contribution to the mknod script up there that is using partition numbers :

LOOPDEV=$(losetup --partscan --find --show "$IMAGE")
lsblk --raw --output "NAME,MAJ:MIN" --noheadings $LOOPDEV | tail -n +2 | while read dev node;
do
    MAJ=$(echo $node | cut -d: -f1)
    MIN=$(echo $node | cut -d: -f2)
    [ ! -e "/dev/$dev" ] &&  mknod "/dev/$dev" b $MAJ $MIN
done

[guysoft]

@ybizeul What is this script for? Does it help with mounting?
How does lsblk help?

[ybizeul]

It's an updated version of @tonyfahrion script to create the loop device. It creates and keeps the partition number as it is in the device.
The reason I had to do this might be specific to docker on the mac, when I mount the disk, only loopx gets created, without any of the loopxpy partitions. With the script it creates both.
Hope that makes sense.