Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply memory cgroups to the container. #44

Merged
merged 1 commit into from
Sep 26, 2020
Merged

Apply memory cgroups to the container. #44

merged 1 commit into from
Sep 26, 2020

Conversation

shishir-a412ed
Copy link
Collaborator

Fixes #8

@shishir-a412ed shishir-a412ed requested a review from a team September 25, 2020 23:42
@shishir-a412ed shishir-a412ed self-assigned this Sep 25, 2020
@chuckyz
Copy link
Collaborator

chuckyz commented Sep 25, 2020

Does this only restrict memory?

@shishir-a412ed
Copy link
Collaborator Author

Does this only restrict memory?

Yes, CPU is a soft limit in nomad.

Copy link

@roberteckert roberteckert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small concern, but there's probably an explanation :)

@@ -105,6 +105,9 @@ func (d *Driver) createContainer(image containerd.Image, containerName, containe
// Set environment variables.
opts = append(opts, oci.WithEnv(env))

// Set cgroups memory limit.
opts = append(opts, oci.WithMemoryLimit(uint64(memoryLimit)))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this a little dangerous? What if someone puts in -9223372036854775808?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This value (memoryLimit) is coming from nomad resource stanza.

Does nomad have any resource validations? as in can someone pass a junk value to nomad job and it will accept?
Let me check.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like nomad does validations on the input:

Case 1: memory = -9223372036854775808

root@ash1-lapp301:~/go/src/github.com/roblox/nomad-driver-containerd/example# nomad job run hog.nomad
Error submitting job: Unexpected response code: 500 (1 error occurred:
	* Task group hog-group validation failed: 1 error occurred:
	* Task hog-task validation failed: 1 error occurred:
	* 1 error occurred:
	* 1 error occurred:
	* minimum MemoryMB value is 10; got -9223372036854775808

Case 2: memory = 9223372036854775808

root@ash1-lapp301:~/go/src/github.com/roblox/nomad-driver-containerd/example# nomad job run hog.nomad
Error getting job struct: Error parsing job file from hog.nomad: error parsing 'job': strconv.ParseInt: parsing "9223372036854775808": value out of range

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Case 2 is not an ideal error, but at least it allows you to search for the value.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess my next concern would be -1. That representation should be 9223372036854775809 (or something like that). I suppose I can RTFnomadC to find out, but it just seems odd to mix and convert types this way without validation on the value.

@chuckyz
Copy link
Collaborator

chuckyz commented Sep 25, 2020

CPU can be a hard limit with the current Docker driver, and even with a soft limit, it's still applied as a part of the cgroup limits in docker via quota/period. I will approve this and open a new issue. 👍

@chuckyz chuckyz merged commit 22f5ed4 into master Sep 26, 2020
@github-actions github-actions bot locked and limited conversation to collaborators Sep 26, 2020
@shishir-a412ed shishir-a412ed deleted the cgroups branch September 26, 2020 00:10
@roberteckert
Copy link

To follow up on this, the bounds checking from nomad is here: https://github.com/hashicorp/nomad/blob/master/nomad/structs/structs.go#L2217. I might try to implement some sort of bounds-check the maximum next week to throw a slightly nicer error, akin to the ones thrown for minimum violations.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cgroups not getting applied on containers launched using nomad-driver-containerd
3 participants