Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Respect container's cgroup path #1872

Conversation

ostenbom
Copy link

Hi there!

We have a testing environment where we have multiple instances of garden/runc running, where each instance creates and destroys containers for its tests. Each of these testing nodes are run in isolation on the same host, so each have their own cgroup directories to minimise interference.

We have noticed that occasionally, when runc is trying to write properties to a cgroup, it chooses a neighbours cgroup by accident.

We think this is to do with the implementation of FindCgroupMountpointAndRoot, which finds the first occurrence of a subsystem in the mount table. When we run multiple instances in parallel, this is not always the right one.

We'd like to improve this function to respect the containers cgroup path, which we can find from the containers spec.

Looking forward to your feedback,

Cheers, the garden team.

Signed-off-by: Danail Branekov danailster@gmail.com
Signed-off-by: Oliver Stenbom ostenbom@pivotal.io

@crosbymichael
Copy link
Member

crosbymichael commented Aug 17, 2018

LGTM

Approved with PullApprove

@ostenbom
Copy link
Author

Bump, need another LGTM or some feedback here!

@BooleanCat
Copy link

@cyphar WDYT?


scanner := bufio.NewScanner(f)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main concern with this change is that it switches away from bufio to storing the entire contents of /proc/self/mountinfo in memory. From memory, we switched away from doing this a long time ago in #496 because of performance problems -- I think effectively reverting this change without a solid justification is probably less-than-ideal.

Respect the container's cgroup path when finding the container's
cgroup mount point, which is useful in multi-tenant environments, where
containers have their own unique cgroup mounts

Signed-off-by: Danail Branekov <danailster@gmail.com>
Signed-off-by: Oliver Stenbom <ostenbom@pivotal.io>
Signed-off-by: Giuseppe Capizzi <gcapizzi@pivotal.io>
@gcapizzi
Copy link
Contributor

@cyphar we've switched back to a Scanner based solution, how does it look?

danail-branekov added a commit to cloudfoundry/garden-runc-release that referenced this pull request Sep 26, 2018
Cherrypick PR opencontainers/runc#1872 to prove
that it fixes containerd hanging. We suspect that runc does not read the
mount table properly hence trying to kill the wrong process

[#159192734]

Submodule src/github.com/opencontainers/runc 9d48320..d01993b:
  > Respect container's cgroup path
@cyphar
Copy link
Member

cyphar commented Nov 13, 2018

LGTM.

Last call for 1.0 merges!

/cc @opencontainers/runc-maintainers

@cyphar
Copy link
Member

cyphar commented Nov 13, 2018

Dammit, PullApprove is broken again -- I guess they want to stop us from releasing. 😉

/cc @caniszczyk

@cyphar
Copy link
Member

cyphar commented Nov 14, 2018

LGTM

Approved with PullApprove

1 similar comment
@crosbymichael
Copy link
Member

crosbymichael commented Nov 16, 2018

LGTM

Approved with PullApprove

@crosbymichael crosbymichael merged commit 76520a4 into opencontainers:master Nov 16, 2018
for _, opt := range strings.Split(fields[len(fields)-1], ",") {
if opt == subsystem {
return fields[4], fields[3], nil
fields := strings.Fields(txt)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replacing strings.Split with strings.Fields is a noticeable performance hit. This is why we need benchmarks.

(this is no longer a problem, just looking through older changes)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants