Add cgroup v2 support

[drakenclimber]

This issue is to track the addition of cgroup v2 support to the various tools in libcgroup.

Below is the list of libcgroup's current applications and their cgroup v2 status:

/usr/bin/

• cgclassify - See commit 9fe521f
• cgcreate - See commit 8fb91a0
• cgdelete - See commit c0a7606
• cgexec - See commit 9fe521f
• cgget - See commit da13073
• cgset - See commit da13073
• cgsnapshot - See commit 564e6b6
• lscgroup - See commit da13073
• lssubsys - See commit 68fef57

/usr/sbin/

• cgclear - See commit 83af70d
• cgconfigparser - See commit 8fb91a0
• cgrulesengd - See commit 3bb5a8f

[santiagorr]

Hi @drakenclimber,
Thanks for you work on libcgroup. Do you have a roadmap/timetable for the three tools that still don't support cgroup2? And maybe a new release? (and even alpha release?)

I am asking you since I am interested in keeping libcgroup in good shape in Debian (specifically since mininet depends on it), and currently, it will be removed from the next stable distribution by the lack of cgroup2 support. i.e. https://bugs.debian.org/959022. So I am keeping an eye on this.

P.S. This should be maybe another issue, do you have the means to make https://sourceforge.net/projects/libcg/ point to your git repo?

[drakenclimber]

Hi @drakenclimber,
Thanks for you work on libcgroup.

@santiagorr, Glad to help :).

Do you have a roadmap/timetable for the three tools that still don't support cgroup2?

Unfortunately, I don't. libcgroup is one of many hats that I wear, and my work on it is sporadic. As you can see, though, we have some lofty goals.

And maybe a new release? (and even alpha release?)

I agree; another release is long overdue. This is definitely on my todo list soon. I'll try to get it out in the next couple weeks.

I am asking you since I am interested in keeping libcgroup in good shape in Debian (specifically since mininet depends on it), and currently, it will be removed from the next stable distribution by the lack of cgroup2 support. i.e. https://bugs.debian.org/959022. So I am keeping an eye on this.

Thanks for the link. I'll do my best to help keep libcgroup in Debian.

P.S. This should be maybe another issue, do you have the means to make https://sourceforge.net/projects/libcg/ point to your git repo?

Agreed. I don't have admin privileges for the old sourceforge repo, but I can reach out to the previous owners. Hopefully they can make the update.

Thanks!

[santiagorr]

@drakenclimber, a couple of comments/questions about this cgroupv2 support:

• if I create a groupe like this: sudo cgcreate -g io:tmp, following the v2 logic, I got as result all the resources available for this tmp group:

/sys/fs/cgroup/tmp/cgroup.controllers      /sys/fs/cgroup/tmp/cpu.pressure           /sys/fs/cgroup/tmp/io.stat              /sys/fs/cgroup/tmp/memory.pressure
/sys/fs/cgroup/tmp/cgroup.events           /sys/fs/cgroup/tmp/cpuset.cpus            /sys/fs/cgroup/tmp/io.weight            /sys/fs/cgroup/tmp/memory.stat
/sys/fs/cgroup/tmp/cgroup.freeze           /sys/fs/cgroup/tmp/cpuset.cpus.effective  /sys/fs/cgroup/tmp/memory.current       /sys/fs/cgroup/tmp/memory.swap.current
/sys/fs/cgroup/tmp/cgroup.max.depth        /sys/fs/cgroup/tmp/cpuset.cpus.partition  /sys/fs/cgroup/tmp/memory.events        /sys/fs/cgroup/tmp/memory.swap.events
/sys/fs/cgroup/tmp/cgroup.max.descendants  /sys/fs/cgroup/tmp/cpuset.mems            /sys/fs/cgroup/tmp/memory.events.local  /sys/fs/cgroup/tmp/memory.swap.high
/sys/fs/cgroup/tmp/cgroup.procs            /sys/fs/cgroup/tmp/cpuset.mems.effective  /sys/fs/cgroup/tmp/memory.high          /sys/fs/cgroup/tmp/memory.swap.max
/sys/fs/cgroup/tmp/cgroup.stat             /sys/fs/cgroup/tmp/cpu.stat               /sys/fs/cgroup/tmp/memory.low           /sys/fs/cgroup/tmp/pids.current
/sys/fs/cgroup/tmp/cgroup.subtree_control  /sys/fs/cgroup/tmp/cpu.weight             /sys/fs/cgroup/tmp/memory.max           /sys/fs/cgroup/tmp/pids.events
/sys/fs/cgroup/tmp/cgroup.threads          /sys/fs/cgroup/tmp/cpu.weight.nice        /sys/fs/cgroup/tmp/memory.min           /sys/fs/cgroup/tmp/pids.max
/sys/fs/cgroup/tmp/cgroup.type             /sys/fs/cgroup/tmp/io.max                 /sys/fs/cgroup/tmp/memory.numa_stat
/sys/fs/cgroup/tmp/cpu.max                 /sys/fs/cgroup/tmp/io.pressure            /sys/fs/cgroup/tmp/memory.oom.group

or, using cgroup-tools:
̀ sudo lscgroup | grep tmp cpuset,cpu,io,memory,hugetlb,pids,rdma:/tmp ̀

wouldn't make sense to create a "unified" group like this: cgcreate -g tmp?

• Could you please provide an example how to use cgset? I am unable to find a correct syntax.

Thanks!

[drakenclimber]

* Could you please provide an example how to use `cgset`? I am unable to find a correct syntax.

As of right now, all of the tools (cgset, cgcreate, etc.) have the exact same usage for both cgroup v1 and cgroup v2, and I would like to keep these tools consistent with old libcgroup releases for one more release - v0.43. Once release v0.43 is out (hopefully in a month or so), then I am open to hearing ideas on how to improve these interfaces and potentially even break backward compatibility. I haven't thought about it too deeply, but at first glance I think your request of cgcreate -g tmp makes a lot of sense.

Anyway :), onto your question. cgset supports two major usages - -r and --copy-from.

I created two v2 cgroups as follows:
$ sudo cgcreate -g io:foo -g io:bar

To set a single value in foo, you can do the following:
$ sudo cgset -r io.weight=42 foo

And just to verify it worked properly:

$ sudo cgget -r io.weight foo
foo:
io.weight: default 42

Or you can copy an entire cgroup's settings to another cgroup, e.g.
$ sudo cgset --copy-from=foo bar

And let's verify the --copy-from worked:

$ sudo cgget -g io:bar
bar:
io.latency: 
io.pressure: some avg10=0.00 avg60=0.00 avg300=0.00 total=0
	full avg10=0.00 avg60=0.00 avg300=0.00 total=0
io.bfq.weight: default 100
io.max: 
io.weight: default 42
io.stat: 

[santiagorr]

* Could you please provide an example how to use `cgset`? I am unable to find a correct syntax.

As of right now, all of the tools (cgset, cgcreate, etc.) have the exact same usage for both cgroup v1 and cgroup v2, and I would like to keep these tools consistent with old libcgroup releases for one more release - v0.43. Once release v0.43 is out (hopefully in a month or so), then I am open to hearing ideas on how to improve these interfaces and potentially even break backward compatibility. I haven't thought about it too deeply, but at first glance I think your request of cgcreate -g tmp makes a lot of sense.

Understood!

Anyway :), onto your question. cgset supports two major usages - -r and --copy-from.

I created two v2 cgroups as follows:
$ sudo cgcreate -g io:foo -g io:bar

To set a single value in foo, you can do the following:
$ sudo cgset -r io.weight=42 foo
...

Stupid me... I had tried setting io.weight, but I forgot to sudo.

However, I still have a resource (io.max) that I cannot get working with cgset:

root@bartik:/sys/fs/cgroup/foo# cat io.max 
root@bartik:/sys/fs/cgroup/foo# cgset -r io.max="8:16 wbps=1024" foo
cgset: cgroup modify error: Invalid argument 
root@bartik:/sys/fs/cgroup/foo# echo "8:16 wbps=1024" > io.max 
root@bartik:/sys/fs/cgroup/foo# cgget -r io.max foo
foo:
io.max: 8:16 rbps=max wbps=1024 riops=max wiops=max

Am I doing something wrong?

Thanks!

[drakenclimber]

Stupid me... I had tried setting io.weight, but I forgot to sudo.

No worries :). For most cgroup settings, you can also change the ownership/permission bits of the setting if you would prefer to avoid using sudo. cgcreate supports this via the -d, -f, and -a flags.

However, I still have a resource (io.max) that I cannot get working with cgset:

root@bartik:/sys/fs/cgroup/foo# cat io.max 
root@bartik:/sys/fs/cgroup/foo# cgset -r io.max="8:16 wbps=1024" foo
cgset: cgroup modify error: Invalid argument 
root@bartik:/sys/fs/cgroup/foo# echo "8:16 wbps=1024" > io.max 
root@bartik:/sys/fs/cgroup/foo# cgget -r io.max foo
foo:
io.max: 8:16 rbps=max wbps=1024 riops=max wiops=max

Am I doing something wrong?

Nope, it appears that you've found a bug in cgset. cgset is using strtok to parse the name/value pair - io.max=... in this case. But the = in the value portion of the logic is confusing it.

I have hacked together a fix, but I would like to make it better and add a test around it. I'll file a bug to ensure it gets into v0.43.

Good find. Thank you!

[drakenclimber]

All tools now support cgroup v2 and have some level of automated testing.

[pxeger]

When is a release that will contain this new v2 support planned? Will it get into v0.43.0?

[giani]

@pxeger we are planning something for next week. The version number will probably be 2.0 - with expectation that the v2 cycle will have continously breaking ABI. When we are happy where it is, we will push out v3 which will then maintain ABI through its lifetime.

[drakenclimber]

When is a release that will contain this new v2 support planned? Will it get into v0.43.0?

@pxeger - v2.0.rc1 is available here:
https://github.com/libcgroup/libcgroup/releases/tag/v2.0.rc1

While doing the release, I noticed a few small things I would like to clean up. We're aiming for next week for the actual release.

Thanks!

[pevogam]

@pxeger we are planning something for next week. The version number will probably be 2.0 - with expectation that the v2 cycle will have continously breaking ABI. When we are happy where it is, we will push out v3 which will then maintain ABI through its lifetime.

Regarding versioning I would recommend eventually also releasing at least one 1.0 version with the now very stable cgroup v1 support for general usage on legacy and not so legacy systems. The 2.0 version is then ideal continuation with the v2 support.