Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CGroup abstraction and parity between V1/V2 #6305

Open
stgraber opened this issue Oct 10, 2019 · 13 comments
Assignees
Milestone

Comments

@stgraber
Copy link
Member

@stgraber stgraber commented Oct 10, 2019

So far for LXD, we've done some very minimal work to have it work on CGroupV2, but that work does not currently include support for any of the limits we support on CGroupV1.

Low level liblxc has support for setting up CgroupV2 controllers, so the bulk of this is to provide it suitable configuration and to update the logic we have driving Cgroups after startup to also work with CGroupV2.

To do so, I believe we'll want to introduce an abstraction layer, effectively a single struct likely in our new cgroup package which we can call from wherever we need cgroup changes done, it would then support detecting if a particular controller is handled by v1 or v2 and give us the key/value we need to feed to liblxc, either through configuration or through the cgroupset API.

@stgraber stgraber added this to the later milestone Oct 10, 2019
@lucindan

This comment has been minimized.

Copy link

@lucindan lucindan commented Oct 23, 2019

I would like to work on this.

1 similar comment
@Namit-Agrawal

This comment has been minimized.

Copy link

@Namit-Agrawal Namit-Agrawal commented Oct 23, 2019

I would like to work on this.

@stgraber

This comment has been minimized.

Copy link
Member Author

@stgraber stgraber commented Oct 23, 2019

Per our e-mail discussion, I suspect the main steps for this will be:

  • Build up a list of all the CGroupV1 files that we interact with today
  • Figure out the equivalent for each of those in the CGroupV2 world
  • Create a set of functions in the "cgroup" package which covers our current use of cgroups and switch over to using them
  • Add functions to detect whether a particular controller is present on the system and if so whether it's using cgroupv1 or cgroupv2
  • Using those functions, update the functions in the cgroup package to detect and handle cgroupv2 resource controllers

To find all the current uses, you can look around for lxc.cgroup (liblxc config file) or CGroupGet and CGroupSet which are the functions used to do live changes.

For testing, your best bet is a recent Ubuntu system, at least 18.04, quite possible 19.10 at this point. This will default to a hybrid v1/v2 setup where all controllers are v1 but a v2 hierarchy exists.

To switch to cgroupv2 only, you can boot the system with systemd.unified_cgroup_hierarchy=true on the kernel command line which should result in only a cgroupv2 hierarchy in /sys/fs/cgroup.

Good contacts for this work are @brauner (who did liblxc cgroupv2), @tomponline (who has quite a bit of experience with those kind of refactoring) and myself for the rest of the LXD knowledge.

@shreyassood

This comment has been minimized.

Copy link

@shreyassood shreyassood commented Oct 29, 2019

@stgraber I would like to contribute as well -- could you please assign me to the issue as well?

@lucindan

This comment has been minimized.

Copy link

@lucindan lucindan commented Oct 30, 2019

@stgraber

This comment has been minimized.

Copy link
Member Author

@stgraber stgraber commented Oct 30, 2019

Hi @lucindan the screenshot didn't make it to github I'm afraid so it's a bit hard to help :)

The mention of a segfault makes me think of a broken dqlite setup, did you make sure that the terminal you're running lxd from has those export variables that make deps gave you?

If not, it'd be missing the correct LD_LIBRARY_PATH which usually results in a segfault during DB initialization.

@aadi123

This comment has been minimized.

Copy link

@aadi123 aadi123 commented Oct 30, 2019

Here is some output that might be helpful

aaditya@pop-os:~$ sudo -E LD_LIBRARY_PATH=$LD_LIBRARY_PATH $GOPATH/bin/lxd --group sudo
WARN[10-29|21:14:19] Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored. 
WARN[10-29|21:14:19] CGroup memory swap accounting is disabled, swap limits will be ignored. 
Segmentation fault
aaditya@pop-os:~$ echo $LD_LIBRARY_PATH
/home/aaditya/go/deps/sqlite/.libs/:/home/aaditya/go/deps/dqlite/.libs/:/home/aaditya/go/deps/raft/.libs:/home/aaditya/go/deps/libco/:
aaditya@pop-os:~$ echo $CGO_CFLAGS
-I/home/aaditya/go/deps/sqlite/ -I/home/aaditya/go/deps/dqlite/include/ -I/home/aaditya/go/deps/raft/include/ -I/home/aaditya/go/deps/libco/
aaditya@pop-os:~$ echo $CGO_LDFLAGS
-L/home/aaditya/go/deps/sqlite/.libs/ -L/home/aaditya/go/deps/dqlite/.libs/ -L/home/aaditya/go/deps/raft/.libs -L/home/aaditya/go/deps/libco/
aaditya@pop-os:~$ 

We are still getting a segfault even though the LD_LIBRARY_PATH seems to be correctly set

@stgraber

This comment has been minimized.

Copy link
Member Author

@stgraber stgraber commented Oct 30, 2019

@aadi123 that's pretty odd. Can you run with --debug.

Also, @freeekanayaka has been doing some fixes around the database logic so maybe you got unlucky and built at a time things were inconsistent, you could run make deps again to update those bits and see if that helps.

@stgraber

This comment has been minimized.

Copy link
Member Author

@stgraber stgraber commented Oct 30, 2019

@aadi123 @freeekanayaka and I have now reproduced that failure on our systems too, looking into it.

@stgraber

This comment has been minimized.

Copy link
Member Author

@stgraber stgraber commented Oct 30, 2019

looks like building libraft with --enable-debug fixes it :)

@stgraber

This comment has been minimized.

Copy link
Member Author

@stgraber stgraber commented Oct 30, 2019

@aadi123 this has now been resolved, running make deps should refresh your copy of libraft and let LXD run.

@aadi123

This comment has been minimized.

Copy link

@aadi123 aadi123 commented Nov 9, 2019

@stgraber I am also part of the group from UT Austin working on this issue. Could I be assigned? Thanks.

@stgraber

This comment has been minimized.

Copy link
Member Author

@stgraber stgraber commented Nov 9, 2019

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.