Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

o/servicestate/quota_control.go: enforce minimum of 4K for quota groups #10346

Conversation

anonymouse64
Copy link
Member

@anonymouse64 anonymouse64 commented Jun 3, 2021

This is to support nesting and to avoid confusing situations like being able to
create empty but nested quota groups which trigger oom-killer to be invoked on
the empty quota group because newer systems require at least 4K for the
accounting of a sub-group.

This came about during investigations of the spread test failures in #10333.

Copy link
Contributor

@stolowski stolowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commit ff59d21 looks good to me, +1

This is to support nesting and to avoid confusing situations like being able to
create empty but nested quota groups which trigger oom-killer to be invoked on
the empty quota group because newer systems require at least 4K for the
accounting of a sub-group.

Signed-off-by: Ian Johnson <ian.johnson@canonical.com>
@anonymouse64 anonymouse64 force-pushed the feature/quota-groups-remastered-deluxe-diamond-edition-2.5 branch from ff59d21 to 54fcd36 Compare June 7, 2021 14:14
@anonymouse64
Copy link
Member Author

(the force push here was just to rebase on master, sorry I didn't realize it had gotten reviews before force pushing otherwise I would have done a master merge)

// to allow nesting, otherwise groups with less than 4K will trigger the
// oom killer to be invoked when a new group is added as a sub-group to the
// larger group.
if memoryLimit <= 4*quantity.SizeKiB {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm I guess a quota the size of a single page is not very useful either 🤣 but alas, if one wants to use and learn the hard way I don't see a reason to prevent that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume with this 4k you can add multiple sub-groups as well? It's not 4k for each sub-group? If so, then 👍

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah well actually funny thing you should ask ... no it's 4K per group it seems but we can't know ahead of time how many sub-groups may go into a group. We could however add accounting to ensure that if a child group is being added that the parent has at least 4096 leftover

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now the thing we will do is to just make the minimum 4K and if folks disagree or have issues we can increase it or make other adjustments later.

…mond-edition-2.5

Signed-off-by: Ian Johnson <ian.johnson@canonical.com>
We now have a minimum memory limit of 4K, so we can't use 1K anymore.

Signed-off-by: Ian Johnson <ian.johnson@canonical.com>
Signed-off-by: Ian Johnson <ian.johnson@canonical.com>
@anonymouse64
Copy link
Member Author

The spread tests are green again FWIW

…mond-edition-2.5

Signed-off-by: Ian Johnson <ian.johnson@canonical.com>
…mond-edition-2.5

Signed-off-by: Ian Johnson <ian.johnson@canonical.com>
…mond-edition-2.5

Signed-off-by: Ian Johnson <ian.johnson@canonical.com>
@anonymouse64 anonymouse64 requested a review from mvo5 June 15, 2021 14:28
This makes it more clear what the actual necessary size is, which is 4K + 1.

Thanks to Samuele for pointing this out.

Signed-off-by: Ian Johnson <ian.johnson@canonical.com>
Copy link
Collaborator

@pedronis pedronis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@mvo5
Copy link
Contributor

mvo5 commented Jun 16, 2021

One failure looks real:

2021-06-15T18:07:54.2454873Z ##[error]2021-06-15 18:07:54 Error executing google:ubuntu-core-16-64:tests/main/snap-quota-groups (jun151733-439828) : 
2021-06-15T18:07:54.2485470Z -----
... 
2021-06-15T18:07:54.2621554Z ExecMainStatus=11
2021-06-15T18:07:54.2623585Z retry: command sh -c systemctl show --property=ExecMainStatus snap.go-example-webserver.webserver.service | MATCH "ExecMainStatus=(203|9)" failed with code 1
2021-06-15T18:07:54.2625196Z retry: next attempt in 1.0 second(s) (attempt 1 of 10)
2021-06-15T18:07:54.2627073Z retry: command sh -c systemctl show --property=ExecMainStatus snap.go-example-webserver.webserver.service | MATCH "ExecMainStatus=(203|9)" failed with code 1
2021-06-15T18:07:54.2628686Z retry: next attempt in 1.0 second(s) (attempt 2 of 10)
2021-06-15T18:07:54.2630951Z retry: command sh -c systemctl show --property=ExecMainStatus snap.go-example-webserver.webserver.service | MATCH "ExecMainStatus=(203|9)" failed with code 1
2021-06-15T18:07:54.2632604Z retry: next attempt in 1.0 second(s) (attempt 3 of 10)
2021-06-15T18:07:54.2634636Z retry: command sh -c systemctl show --property=ExecMainStatus snap.go-example-webserver.webserver.service | MATCH "ExecMainStatus=(203|9)" failed with code 1
2021-06-15T18:07:54.2636566Z retry: next attempt in 1.0 second(s) (attempt 4 of 10)
2021-06-15T18:07:54.2638418Z retry: command sh -c systemctl show --property=ExecMainStatus snap.go-example-webserver.webserver.service | MATCH "ExecMainStatus=(203|9)" failed with code 1
2021-06-15T18:07:54.2640097Z retry: next attempt in 1.0 second(s) (attempt 5 of 10)
2021-06-15T18:07:54.2642000Z retry: command sh -c systemctl show --property=ExecMainStatus snap.go-example-webserver.webserver.service | MATCH "ExecMainStatus=(203|9)" failed with code 1
2021-06-15T18:07:54.2644545Z retry: next attempt in 1.0 second(s) (attempt 6 of 10)
2021-06-15T18:07:54.2646828Z retry: command sh -c systemctl show --property=ExecMainStatus snap.go-example-webserver.webserver.service | MATCH "ExecMainStatus=(203|9)" failed with code 1
2021-06-15T18:07:54.2648530Z retry: next attempt in 1.0 second(s) (attempt 7 of 10)
2021-06-15T18:07:54.2650408Z retry: command sh -c systemctl show --property=ExecMainStatus snap.go-example-webserver.webserver.service | MATCH "ExecMainStatus=(203|9)" failed with code 1
2021-06-15T18:07:54.2652005Z retry: next attempt in 1.0 second(s) (attempt 8 of 10)
2021-06-15T18:07:54.2653841Z retry: command sh -c systemctl show --property=ExecMainStatus snap.go-example-webserver.webserver.service | MATCH "ExecMainStatus=(203|9)" failed with code 1
2021-06-15T18:07:54.2655381Z retry: next attempt in 1.0 second(s) (attempt 9 of 10)
2021-06-15T18:07:54.2657897Z retry: command sh -c systemctl show --property=ExecMainStatus snap.go-example-webserver.webserver.service | MATCH "ExecMainStatus=(203|9)" failed with code 1
2021-06-15T18:07:54.2660975Z retry: command sh -c systemctl show --property=ExecMainStatus snap.go-example-webserver.webserver.service | MATCH "ExecMainStatus=(203|9)" keeps failing after 10 attempts
2021-06-15T18:07:54.2662713Z -----

unclear if it's a race or a real issue.

@anonymouse64
Copy link
Member Author

Let's land #10266 before this one, that conflicts with this and much as I would like to land this one, there is the failure that @mvo5 pointed out here and there is a pre-existing race condition that 10266 exacerbates that I would like to solve there. With any luck fixing that race condition there will make this failure go away too

…mond-edition-2.5

Signed-off-by: Ian Johnson <ian.johnson@canonical.com>
@anonymouse64
Copy link
Member Author

Okay, this is now unblocked, let's see how happy the tests are here

@pedronis pedronis merged commit 2fb355d into snapcore:master Jun 18, 2021
@anonymouse64 anonymouse64 deleted the feature/quota-groups-remastered-deluxe-diamond-edition-2.5 branch June 18, 2021 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants