Mounting root filesystem of host crashes sysbox

[nudgegoonies]

I am running current master of Sysbox CE on Debian 10 with Backport Kernel. The issue can be reproduced with:

docker run -it -v /:/mnt:ro --rm debian:buster
Unable to find image 'debian:buster' locally
buster: Pulling from library/debian
e22122b926a1: Pull complete 
Digest: sha256:a4e852392000434b7c50b26dcf6a659a037521b17df69dd2ace5c2368efba38b
Status: Downloaded newer image for debian:buster

I tried mounting regular directories like /home and /tmp and this works. I suspect there is a problem with /dev, /proc or /sys when mounting /. I get no error message when entering the command above. Container is in the state "starting". In the log i could find this error message:

Error response from daemon: OCI runtime create failed: failed to pre-register with sysbox-fs: failed to register container with sysbox-fs: rpc error: code = DeadlineExceeded desc = context deadline exceeded: unknown.

I cannot stop the container and i cannot start new containers. Rebooting via "reboot" hangs the system. Only a hard reset helps.

[rodnymolina]

Thanks for reporting this one @nudgegoonies.

I reproduced this one in my setup and noticed a deadlock in sysbox-fs module during the container pre-registration phase. This is simply a consequence of the fact that you are setting the host entire "/" as part of the container's rootfs. Sysbox-fs was not designed with this case in mind, so i'm not sure we will ever support this corner case, specially since this goes again Sysbox's dogma of creating secure and well-isolated environments.

Now, what we really need to do though, is to identify this scenario and notify users accordingly to avoid the pain you just went through to restore the service.

[ctalledo]

To expand a bit on Rodny's comments: a key tenet of deploying containers with Sysbox is that the container is well isolated from the host. Mounting the host's root "/" into the container by definition breaks that isolation, so it's not something we can support.

Having said that, Sysbox should detect this and report an appropriate error, so thanks @nudgegoonies for reporting the issue. We will also update this section of the Sysbox user-guide to reflect this.

[nudgegoonies]

@rodnymolina @ctalledo Thank you very much for your answers. Mounting / really makes no sense regarding better isolation. It was not my idea - i stumbled on it because cadvisor was already installed on the server where i replaced runc with sysbox runc for further evaluation. And mounting root seems to be the default way to start cadvisor according to their documentation:
https://github.com/google/cadvisor
And cadvisor is configured this way in our puppet code. It took me several reboots until i found the cause and disabled cadvisor.

Thanks for taking a look on the crashes and adding a warning in the documentation. I think you should warn about cadvisor too.

[ctalledo]

Thanks again @nudgegoonies , was not aware that cAdvisor was mounting the host's root into the container (which makes sense for what it's trying to do). Sounds like for cAdvisor you'll need to use the regular OCI runc for the time being. We'll update the docs accordingly.

[ctalledo]

This no longer occurs on Sysbox v0.4.1. Closing.