Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatible with cgroup v2 for memory utils #3278

Closed
wey-gu opened this issue Nov 5, 2021 · 4 comments · Fixed by #3419
Closed

Compatible with cgroup v2 for memory utils #3278

wey-gu opened this issue Nov 5, 2021 · 4 comments · Fixed by #3419
Labels
type/enhancement Type: make the code neat or more efficient

Comments

@wey-gu
Copy link
Contributor

wey-gu commented Nov 5, 2021

It seems in a cgroup v2 host OS, graphD won't boot up due to it's calling sysfs of cgroup v1 by MemoryUtils.cpp, in production most OS now are cgroup v1 now but we should consider cgroupv2 ASAP :).

Log file created at: 2021/11/05 08:45:43
Running on machine: graphd1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E1105 08:45:43.579129     1 GraphDaemon.cpp:158] Could not open the file: /sys/fs/cgroup/memory/memory.limit_in_bytes

Minimal reproduce:

in a ubuntu 21.10 VM with cgroupv2 to docker-compose up nebula

root@lima-default:/tmp/lima/nebula-docker-compose# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 21.10
Release:	21.10
Codename:	impish

root@lima-default:/tmp/lima/nebula-docker-compose# file /sys/fs/cgroup/cgroup.controllers
/sys/fs/cgroup/cgroup.controllers: empty
@wey-gu wey-gu added type/bug Type: something is unexpected type/enhancement Type: make the code neat or more efficient and removed type/bug Type: something is unexpected labels Nov 5, 2021
@yixinglu yixinglu linked a pull request Dec 7, 2021 that will close this issue
7 tasks
@wey-gu
Copy link
Contributor Author

wey-gu commented Dec 8, 2021

It turned out in ARM CentOS 7.9.2009 latest docker desktop on M1 MacBook, it's with cgroupv2 on Docker VM hostOS by default... Thanks to @igrekun for the PR to fix it, looking forward to tomorrow's nightly docker image ~~

[root@13b3994f8360 nebula]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (AltArch)
[root@13b3994f8360 nebula]# ls /sys/fs/cgroup/cgroup.controllers
/sys/fs/cgroup/cgroup.controllers
[root@13b3994f8360 nebula]# ls /sys/fs/cgroup/memory.stat
/sys/fs/cgroup/memory.stat
[root@13b3994f8360 nebula]# ls /sys/fs/cgroup/memory/memory.stat
ls: cannot access /sys/fs/cgroup/memory/memory.stat: No such file or directory

[root@13b3994f8360 nebula]# ls /sys/fs/cgroup/memory.max
/sys/fs/cgroup/memory.max
[root@13b3994f8360 nebula]# ls /sys/fs/cgroup/memory/memory.limit_in_bytes
ls: cannot access /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory

[root@13b3994f8360 nebula]# uname -a
Linux 13b3994f8360 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
[root@13b3994f8360 nebula]# ls /sys/fs/cgroup/memory.current
/sys/fs/cgroup/memory.current
[root@13b3994f8360 nebula]# ls /sys/fs/cgroup/memory/memory.usage_in_bytes
ls: cannot access /sys/fs/cgroup/memory/memory.usage_in_bytes: No such file or directory
[root@13b3994f8360 nebula]# ls /sys/fs/cgroup/memory/memory.limit_in_bytes
ls: cannot access /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory

[root@13b3994f8360 nebula]# grep cgroup /proc/filesystems
nodev	cgroup
nodev	cgroup2

@wey-gu
Copy link
Contributor Author

wey-gu commented Dec 20, 2021

It turned out since docker desktop(macOS) 4.3.0, the host by default used Cgroup v2, thus 2.6.0 nebula graphD wont boot up due to this issue, the mitigation/work around should be falling back to previous docker desktop 4.2.0.

ref: https://docs.docker.com/desktop/mac/release-notes/#docker-desktop-430

@igrekun
Copy link
Contributor

igrekun commented Dec 20, 2021

@wey-gu does the nightly version work with your Cgroup v2 setup? GraphD booted fine on my Cgroup v2 machines.
Do you have issues with nightly version or you just provided a workaround for those running 2.6.0 and not nightly? Sorry for many questions, just want to make sure that the fix works for everyone (:

@wey-gu
Copy link
Contributor Author

wey-gu commented Dec 21, 2021

@wey-gu does the nightly version work with your Cgroup v2 setup? GraphD booted fine on my Cgroup v2 machines. Do you have issues with nightly version or you just provided a workaround for those running 2.6.0 and not nightly? Sorry for many questions, just want to make sure that the fix works for everyone (:

Dear @igrekun, It worked like a charm!!!
with your PR merged, I had booted up the next nightly build on the Cgroupv2 machine the next day:) Thanks a lot for your great work!!! :)

And yes, this is was just a workaround for the 2.6.0 tagged image(as the master nebula doesn't fit any studio for now).

Due to the docker desktop/lima-VM's default host VM migrated to Cgroup V2 recently, your PR had become more crucial to ensure the next release is bootable on many developers' machines in addition to helping on modern servers only.

Thank you so much, Igor!
BR//Wey

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Type: make the code neat or more efficient
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants