security=insecure does not automatically provide access to devices

[EduardoRFS]

I'm trying to build a docker image using an ext4 filesystem, but it seems that even when using --allow security.insecure I cannot run some commands like mount nor access my host devices, like what --privileged allows.

# syntax = docker/dockerfile:experimental
FROM archlinux/base

RUN pacman -Sy --noconfirm wget unzip
RUN wget https://dl.google.com/android/repository/sys-img/android/x86_64-21_r05.zip
RUN wget https://dl.google.com/android/repository/android-ndk-r20-linux-x86_64.zip

RUN unzip x86_64-21_r05.zip
RUN unzip android-ndk-r20-linux-x86_64.zip

RUN mkdir system
RUN --security=insecure mount x86_64/system.img system

docker buildx build --allow security.insecure .

[tonistiigi]

@smira I tried it and it indeed did not work. Any ideas?

[smira]

not sure, does this --alow flag gets passed to the builder?

we used something like: https://github.com/talos-systems/talos/blob/master/Makefile#L52

docker buildx create --driver docker-container --name local --buildkitd-flags '--allow-insecure-entitlement security.insecure' --use

[tonistiigi]

@smira yes

#6 [1/2] FROM docker.io/tonistiigi/listcaps@sha256:86852ae6d01d2dbdd779ffb1...
#6 resolve docker.io/tonistiigi/listcaps@sha256:86852ae6d01d2dbdd779ffb1ab8e4b4cce7d9c882783ebc25f47d875b9dbca17 done
#6 CACHED

#7 [2/2] RUN --security=insecure ["/list"]
#7 0.076 /list = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,37+eip
#7 0.076                      cap_chown 1 1 1 1
#7 0.076               cap_dac_override 1 1 1 1
#7 0.076            cap_dac_read_search 1 1 1 1
#7 0.076                     cap_fowner 1 1 1 1
#7 0.076                     cap_fsetid 1 1 1 1
#7 0.076                       cap_kill 1 1 1 1
#7 0.076                     cap_setgid 1 1 1 1
#7 0.076                     cap_setuid 1 1 1 1
#7 0.076                    cap_setpcap 1 1 1 1
#7 0.076            cap_linux_immutable 1 1 1 1
#7 0.076           cap_net_bind_service 1 1 1 1
#7 0.076              cap_net_broadcast 1 1 1 1
#7 0.076                  cap_net_admin 1 1 1 1
#7 0.076                    cap_net_raw 1 1 1 1
#7 0.076                   cap_ipc_lock 1 1 1 1
#7 0.076                  cap_ipc_owner 1 1 1 1
#7 0.076                 cap_sys_module 1 1 1 1
#7 0.076                  cap_sys_rawio 1 1 1 1
#7 0.076                 cap_sys_chroot 1 1 1 1
#7 0.076                 cap_sys_ptrace 1 1 1 1
#7 0.076                  cap_sys_pacct 1 1 1 1
#7 0.076                  cap_sys_admin 1 1 1 1
#7 0.076                   cap_sys_boot 1 1 1 1
#7 0.076                   cap_sys_nice 1 1 1 1
#7 0.076               cap_sys_resource 1 1 1 1
#7 0.076                   cap_sys_time 1 1 1 1
#7 0.076             cap_sys_tty_config 1 1 1 1
#7 0.076                      cap_mknod 1 1 1 1
#7 0.076                      cap_lease 1 1 1 1
#7 0.076                cap_audit_write 1 1 1 1
#7 0.076              cap_audit_control 1 1 1 1
#7 0.076                    cap_setfcap 1 1 1 1
#7 0.076               cap_mac_override 1 1 1 1
#7 0.076                  cap_mac_admin 1 1 1 1
#7 0.076                     cap_syslog 1 1 1 1
#7 0.076                 cap_wake_alarm 1 1 1 1
#7 0.076              cap_block_suspend 1 1 1 1


root@tonis-ubuntu-s-4vcpu-8gb-sfo1-01:~/test# cat Dockerfile
# syntax = docker/dockerfile:experimental
from tonistiigi/listcaps
run --security=insecure ["/list"]

[smira]

our usecase was actually running containerd in a process launched from RUN --security=insecure ... during the build (unit-tests which depend on containerd), and containerd does tons of mount calls while it creates a container, but I never tested cases like mounting filesystem images, I can check it out.

[EduardoRFS]

@smira but shouldn't be using --security=insecure the same as running as root bypassing the container?

[smira]

So I guess the problem is that loopback mount relies on /dev/loopX devices to be available inside the build environment, while they're not there:

#15 0.183 stat("/sys/block", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
#15 0.183 stat("/dev/loop-control", 0x7ffe59f25c70) = -1 ENOENT (No such file or directory)
#15 0.183 stat("/dev/loop", 0x7ffe59f25df0)       = -1 ENOENT (No such file or directory)
#15 0.183 stat("/dev/loop0", 0x7ffe59f25d90)      = -1 ENOENT (No such file or directory)
#15 0.184 stat("/dev/loop1", 0x7ffe59f25d90)      = -1 ENOENT (No such file or directory)
#15 0.184 stat("/dev/loop2", 0x7ffe59f25d90)      = -1 ENOENT (No such file or directory)
#15 0.184 stat("/dev/loop3", 0x7ffe59f25d90)      = -1 ENOENT (No such file or directory)
#15 0.184 stat("/dev/loop4", 0x7ffe59f25d90)      = -1 ENOENT (No such file or directory)
#15 0.184 stat("/dev/loop5", 0x7ffe59f25d90)      = -1 ENOENT (No such file or directory)
#15 0.184 stat("/dev/loop6", 0x7ffe59f25d90)      = -1 ENOENT (No such file or directory)
#15 0.184 stat("/dev/loop7", 0x7ffe59f25d90)      = -1 ENOENT (No such file or directory)

And error message from mount is misleading.

[smira]

And to add to that, one can successfully do bind mount for example:

RUN --security=insecure mount --bind /path1 /path2

[tonistiigi]

@smira I forgot the --privileged adds the host devices as well. Should we do the same for build as well? Or is it weird to depend on specific devices in host and we should just add a preset list. I think we should definitely support loopbacks, fuse etc.

[smira]

@tonistiigi this is more of a product-level question, I'm not sure if mknod could be used to create the devices required while buildkit allows device passthrough from the host (haven't checked if it's enabled?)

I don't have a good answer tbh

[EduardoRFS]

@tonistiigi I saw your PR and did test using it, but unfortunately it doesn't work, sure now it's not giving a error anymore but the image is simply not mounted

# syntax = docker/dockerfile:experimental
FROM archlinux/base as system

RUN pacman -Sy --noconfirm wget unzip
RUN wget https://dl.google.com/android/repository/sys-img/android/armeabi-v7a-22_r02.zip
RUN unzip armeabi-v7a-22_r02.zip
RUN mkdir /armeabi-v7a/mnt
RUN --security=insecure mount /armeabi-v7a/system.img /armeabi-v7a/mnt
RUN ls /armeabi-v7a/mnt

Any idea?

[tonistiigi]

That PR is not in a release build yet, did you use --driver-opt image=moby/buildkit:master when creating the builder instance?

[tonistiigi]

@EduardoRFS btw, your loopback mount will not exist after RUN has completed if that wasn't clear. All the RUN commands run in isolated containers. So your ls would need to be RUN --security=insecure mount /armeabi-v7a/system.img /armeabi-v7a/mnt && ls /armeabi-v7a/mnt. If you want the files to be in the image you need to copy them out from the loopback mount to the container layers with a RUN command.

[EduardoRFS]

@tonistiigi oh nice, I did tried it with master, but didn't tought about it thx man <3