Fails when docker root dir is mapped to a volume (e.g emptyDir)
#406
Comments
|
Hi @felipecrs, thanks for reporting the issue. What version of Sysbox are you using? I ask because this looks similar to issue #254 (search for unlinkat in that issue), which is supposed to be fixed.
That's strange; I noticed you are using Thanks! |
|
I installed Sysbox today through the daemonset, so I it's the newest version: $ sysbox-runc --version
sysbox-runc
edition: Community Edition (CE)
version: 0.4.1
commit: d540126188a1e8595c8f769aeb91833002c37b3a
built at: Fri Oct 1 19:33:49 UTC 2021
built by: Rodny Molina
oci-specs: 1.0.2-dev |
|
I will try to change the PVC and check. |
|
Well, I believe that using local volume works, as inside of the |
|
You are using the latest Sysbox, so the root cause must be something different from issue #254 (though it has the same symptom).
Yes, please check and confirm if this works or not. If it does work, it gives us a strong clue that the problem is related to the |
|
Ops, with $ kubectl run dind --rm -i --image ghcr.io/felipecrs/jenkins-agent-dind:latest --pod-running-timeout=3m --overrides='
{
"metadata": {
"annotations": {
"io.kubernetes.cri-o.userns-mode": "auto:size=65536"
}
},
"spec": {
"containers": [
{
"image": "ghcr.io/felipecrs/jenkins-agent-dind:latest",
"name": "dind",
"imagePullPolicy": "Always",
"tty": true,
"volumeMounts": [
{
"mountPath": "/home/jenkins/agent",
"name": "workspace-volume",
"readOnly": false
}
],
"command": ["/entrypoint.sh", "bash"]
}
],
"runtimeClassName": "sysbox-runc",
"volumes": [
{
"name": "workspace-volume",
"emptyDir": {}
}
]
}
}
'I will rephrase the Issue. |
felipecrs
changed the title
docker pull gradle inside sysbox pod fails when docker root dir is within a emptyDir
|
@ctalledo I updated the issue description, it has also more logs such as the docker version and info within the sysbox pod. Can you please take a look again? |
|
Thanks Felipe, will try to repro and debug a bit later today. |
|
On my image, But when testing with |
No that should not matter. I was thinking it's probably related to the type of storage of the PVC, though the fact that if fails with something as simple as emptyDir does not gel with that theory. |
|
@ctalledo I furthered tested here. It turns out that it actually matters. Using Notice that I delete the kubectl run dind --rm -i --image ghcr.io/felipecrs/jenkins-agent-dind:latest --pod-running-timeout=3m --overrides='
{
"metadata": {
"annotations": {
"io.kubernetes.cri-o.userns-mode": "auto:size=65536"
}
},
"spec": {
"containers": [
{
"image": "ghcr.io/felipecrs/jenkins-agent-dind:latest",
"name": "dind",
"imagePullPolicy": "Always",
"tty": true,
"volumeMounts": [
{
"mountPath": "/var/lib/docker",
"name": "workspace-volume",
"readOnly": false
}
],
"command": ["bash", "-xec", "sudo rm -f /etc/docker/daemon.json; exec /entrypoint.sh bash -xec \"df /var/lib/docker; docker version; docker info; docker pull gradle\""]
}
],
"runtimeClassName": "sysbox-runc",
"volumes": [
{
"name": "workspace-volume",
"emptyDir": {}
}
]
}
}
'
If you don't see a command prompt, try pressing enter.
INFO[2021-10-06T17:12:48.299069538Z] Loading containers: done.
WARN[2021-10-06T17:12:48.358322334Z] Not using native diff for overlay2, this may cause degraded performance for building images: running in a user namespace storage-driver=overlay2
INFO[2021-10-06T17:12:48.358539871Z] Docker daemon commit=79ea9d3 graphdriver(s)=overlay2 version=20.10.9
INFO[2021-10-06T17:12:48.358668340Z] Daemon has completed initialization
INFO[2021-10-06T17:12:48.534876604Z] API listen on /var/run/docker.sock
Client: Docker Engine - Community
Version: 20.10.9
API version: 1.41
Go version: go1.16.8
Git commit: c2ea9bc
Built: Mon Oct 4 16:08:29 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.9
API version: 1.41 (minimum version 1.12)
Go version: go1.16.8
Git commit: 79ea9d3
Built: Mon Oct 4 16:06:37 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.11
GitCommit: 5b46e404f6b9f661a205e28d59c982d3634148f8
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
[services.d] done.
+ df /var/lib/docker
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 130550852 14376736 109499480 12% /var/lib/docker
+ docker version
Client: Docker Engine - Community
Version: 20.10.9
API version: 1.41
Go version: go1.16.8
Git commit: c2ea9bc
Built: Mon Oct 4 16:08:29 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.9
API version: 1.41 (minimum version 1.12)
Go version: go1.16.8
Git commit: 79ea9d3
Built: Mon Oct 4 16:06:37 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.11
GitCommit: 5b46e404f6b9f661a205e28d59c982d3634148f8
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
+ docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.6.3)
compose: Docker Compose (Docker Inc., v2.0.1)
scan: Docker Scan (Docker Inc., v0.8.0)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 20.10.9
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
runc version: v1.0.2-0-g52b36a2
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.4.0-70-generic
Operating System: Ubuntu 20.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.34GiB
Name: dind
ID: RQFG:7SGQ:QEKF:HFJA:LIU7:RT63:TZSW:GUBS:BAZS:UG6V:KAUD:BQ6W
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
+ docker pull gradle
Using default tag: latest
latest: Pulling from gradle
f3ef4ff62e0d: Pull complete
706b9b9c1c44: Pull complete
0fffb0c672b9: Pull complete
5a54c3905797: Pull complete
830009aaff35: Pull complete
a28d173c1d5d: Pull complete
Digest: sha256:d7a3b2f32d4a78efe3f9c275d97b3c8f728f34a145511c9815b29645cf0eb854
Status: Downloaded newer image for gradle:latest
gradle:latest
[cmd] setpriv exited 0
INFO[2021-10-06T17:13:08.127827498Z] Processing signal 'terminated'
INFO[2021-10-06T17:13:08.130454646Z] stopping event stream following graceful shutdown error="<nil>" module=libcontainerd namespace=moby
INFO[2021-10-06T17:13:08.130854411Z] Daemon shutdown complete
INFO[2021-10-06T17:13:08.130896632Z] stopping event stream following graceful shutdown error="context canceled" module=libcontainerd namespace=plugins.moby
INFO[2021-10-06T17:13:08.130903359Z] stopping healthcheck following graceful shutdown module=libcontainerd
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
WARN[2021-10-06T17:13:09.131438672Z] grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout". Reconnecting... module=grpc
s6-svscanctl: fatal: unable to control /var/run/s6/services: supervisor not listening
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
Session ended, resume using 'kubectl attach dind -c dind -i -t' command when the pod is running
pod "dind" deleted |
|
Hm... when I mount $ df /home/jenkins/agent/docker
Filesystem 1K-blocks Used Available Use% Mounted on
/var/lib/sysbox/shiftfs/b3a26b5d-2d6c-40e7-a356-5e29903e7125 130550852 14374468 109501748 12% /home/jenkins/agent/dockerBut when I mount $ df /var/lib/docker
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 130550852 14376736 109499480 12% /var/lib/docker |
I see ... I understand the problem now: you are mounting an emptyDir (i.e., a host dir managed by K8s) into the container's As a result, Sysbox treats the mount normally and mounts shiftfs on the emptyDir (so that it show up with proper permissions inside the container). The problem is that now we have the Docker data root inside the container working on top of shiftfs, and this does not work (i.e., the inner Docker will try to mount overlayfs on top of shiftfs and this does not work). You've hit a limitation of Sysbox, documented here: the inner Docker data root must be at Sorry about this, glad we root caused it. This limitation should go away soon (months), as we transition Sysbox to leverage the new Linux ID-Mapped mount feature which voids the need for shiftfs. |
|
That's interesting. Thank you very much for the explanation. Currently, I use PVC to give my builds a given amount of disk space through a network storage provider (because my nodes are not capable of providing such amount of storage by theirselves). For that, I use the Jenkins Kubernetes Plugin for automatically creating a new PVC for each build, and then assigning it to the build's pod. But it's limited to the Jenkins Agent Workspace ( I understand that, as you said, it's probably going to be fixed in the future. But for now, is there any workaround that you can think of? Perhaps a |
I see ... that's a tough one. A simlink from Let me think if there is another solution. |
|
Well, I actually meant a symlink from Something that I thought, but it would require to change sysbox's internals would be a new annotation like: "metadata": {
"annotations": {
"sysbox.docker-root-dir": "/home/jenkins/agent/docker"
}
}But have in mind that I don't know much about it so I don't even know if the And of course, if the use-case is worth it. |
|
Well, my tries with symlinks or bind mounts didn't work. Which, I guess, is expected. |
I don't think that will work, because that would mean the inner Docker would store it's data in If you do the symlink the other way around (from Thinking more about this, I am not sure the idea of having the inner Docker store it's images on a mount (
I was thinking about something similar (but using env variables passed to the container rather than annotations), but in general we want to avoid this approach. Rather we want Sysbox to be smart enough to setup the container environment the proper way. I would prefer to implement the ID-mapped mount approach to this one, though it would require the host have a Linux kernel >= 5.12. |
|
@felipecrs, question: in the |
I understand your thinking 100%. But it turns out that it's a very fast network-based storage poll, it's even faster than the SSDs attached to the nodes their selves. And, I have been running the builds using such arrange since more than an year already, although I'm using the normal privileged flag (that's the reason why I want to migrate to sysbox). My nodes, as I said, suffer of low disk resources, all have an attached SSD of 128GB. I have builds which requires free 48Gi of space just for them, and it's often to run more in parallel in the same node. Not to count the fact that the kubelet in the node itself keeps downloading images on and on, and for having a decent amount of disk free, I would have to customize the k8s garbage collector on each node to make sure these dangling images gets deleted in time for allocating a pod with a requirement of 48Gi, as example. Sadly, if I let K8s do it on its own, and instead of using the network-based PVC but using a
I totally understand. Better to do the definitive fix than spending time on the workaround. But unfortunately, that won't work for me... my nodes all run Ubuntu 18.04, with Linux 5.4. The 5.12 didn't make its way even to 20.04 so I doubt it will do for 18.04 anytime soon. The cluster which I use is rented, and I have no option to ask for Ubuntu 20.04 either (despite as I said, Linux 5.12 is not even available to it yet -- HWE I mean). Having that said, the definitive fix would probably not be effective for anyone else who is using Ubuntu, as of now. I don't know about Debian, though.
Yes, see: |
|
Hi @felipecrs, thanks for the info. We've found a simpler solution: when the container starts, Sysbox will look at the container's @rodnymolina will be working on this soon. |
|
That's really cool, nice approach and it would work for me. I would just suggest to document the fact that |
Yep, makes sense ... will do. |
|
@felipecrs, the fix for this one just went in here. Please build Sysbox from sources when have a chance and let us know how it goes. |
|
Sure, I will do. I plan to: Build from source, create my own sysbox binary container and adjust the daemonset to use my custom image to install in my workers. Does it sounds like a good plan? |
|
I just realized something: I think it would be a good idea to adjust the CI pipelines to publish a tag like |
Let me give you a docker-image that you can point our manifests to. |
|
Hey @rodnymolina, I don't mind building it myself, really. I just don't want that you "waste" your time only because of me... You are already doing much! |
|
Thanks Felipe, but it's not that simple in this case as you don't have access to the code (k8s-installer) to generate these daemon-set docker images. I originally thought that you already had your setup available and that you would simply replace the old sysbox binaries with the new ones. But if that's not the case, then you need a new sysbox-k8s-deploy daemonset ... Please point our manifests to this image: Let me know if any questions. |
Let us think about this for a bit. Will get back to you. Thanks! |
|
Well, it didn't work at first, then I found that only the focal variant is updated in the image:
I went ahead and replaced the bionic binary with the focal one (and created a custom image for deploying in the daemonset). But, after installing, everything seems to be working! |
|
@felipecrs, glad to hear that it works! And yes, should have told you that I had only built the ubuntu-focal image as that's what I found first in the description of this issue -- didn't read a more recent comment where you explicitly mentioned ubuntu-bionic. Sorry for that :-( |
|
Don't worry! :D The Ubuntu 20.04 is my dind image, btw. |
|
Yes, I see that now :-) Will go ahead and close this issue. Re-open it if you see anything else related to this fix. |
|
Again, thanks a lot, I couldn't be more grateful. Now I ran into #410. |
|
Some fun pictures: Before the fix: After the fix: I'm glad I have written this pipeline for testing the capabilities of my Jenkins Agents. It allows me to find the issues I would have before I actually change the environment. |
felipecrs
changed the title
docker pull gradle inside sysbox pod fails when docker root dir is within a emptyDiremptyDir)
|
Truly a "graphical" description! Let's hope we can make it all green in the new issue :-) |
In my Jenkins setup, I use the Kubernetes Plugin to allow it to spawn a new Pod for each build. Now, I'm integrating it with Sysbox, but I found this issue.
Some notes:
docker:dindor theregistry.nestybox.com/nestybox/ubuntu-bionic-systemd-dockeremptyDirvolume the error does not happenubuntusysbox, withprivilegedit works fine/home/jenkins/agent/dockeris the docker root dir rather than/var/lib/dockerI know there must be something wrong with my image and I will probably refactor my image to make use of systemd as the sysbox sample image does, but I would like to report it here in case someone else faces the same issue.
To reproduce:
The text was updated successfully, but these errors were encountered: