8284950: CgroupV1 detection code should consider memory.swappiness #8285

xpbob · 2022-04-18T09:07:31Z

set memory.swappiness to 0,swap space will not be used
determine the value of memory.swappiness
https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

    Memory Limit: 50.00M
    Memory Soft Limit: Unlimited
    Memory & Swap Limit: 100.00M
    Maximum Processes Limit: 4194305

=>

    Memory Limit: 50.00M
    Memory Soft Limit: Unlimited
    Memory & Swap Limit: 50.00M
    Maximum Processes Limit: 4194305

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed (1 review required, with at least 1 reviewer)

Issue

JDK-8284950: CgroupV1 detection code should consider memory.swappiness

Reviewers

Severin Gehwolf (@jerboaa - Reviewer)
Ioi Lam (@iklam - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/8285/head:pull/8285
$ git checkout pull/8285

Update a local copy of the PR:
$ git checkout pull/8285
$ git pull https://git.openjdk.java.net/jdk pull/8285/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 8285

View PR using the GUI difftool:
$ git pr show -t 8285

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/8285.diff

bridgekeeper · 2022-04-18T09:08:54Z

👋 Welcome back xpbob! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2022-04-18T09:10:56Z

@xpbob The following label will be automatically applied to this pull request:

core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2022-04-18T09:15:11Z

Webrevs

iklam · 2022-04-18T16:22:27Z

src/java.base/linux/classes/jdk/internal/platform/cgroupv1/CgroupV1Subsystem.java

-         return retval > 0;
+        long memswBytes = getLongValue(controller, "memory.memsw.limit_in_bytes");
+        long swappiness = getLongValue(controller, "memory.swappiness");
+        return (memswBytes > 0 && swappiness > 0);


Does this also need to be changed in the test?

jdk/test/lib/jdk/test/lib/containers/cgroup/MetricsTesterCgroupV1.java

Lines 291 to 296 in c63fabe

oldVal = metrics.getMemoryAndSwapLimit();

newVal = getLongValueFromFile(Controller.MEMORY, "memory.memsw.limit_in_bytes");

newVal = newVal > unlimited_minimum ? CgroupSubsystem.LONG_RETVAL_UNLIMITED : newVal;

if (!CgroupMetricsTester.compareWithErrorMargin(oldVal, newVal)) {

fail(Controller.MEMORY, "memory.memsw.limit_in_bytes", oldVal, newVal);

}

There's also corresponding code in HotSpot:

jdk/src/hotspot/os/linux/cgroupV1Subsystem_linux.cpp

Lines 129 to 150 in c63fabe

jlong CgroupV1Subsystem::memory_and_swap_limit_in_bytes() {

GET_CONTAINER_INFO(julong, _memory->controller(), "/memory.memsw.limit_in_bytes",

"Memory and Swap Limit is: " JULONG_FORMAT, JULONG_FORMAT, memswlimit);

if (memswlimit >= _unlimited_memory) {

log_trace(os, container)("Non-Hierarchical Memory and Swap Limit is: Unlimited");

CgroupV1MemoryController* mem_controller = reinterpret_cast<CgroupV1MemoryController*>(_memory->controller());

if (mem_controller->is_hierarchical()) {

const char* matchline = "hierarchical_memsw_limit";

const char* format = "%s " JULONG_FORMAT;

GET_CONTAINER_INFO_LINE(julong, _memory->controller(), "/memory.stat", matchline,

"Hierarchical Memory and Swap Limit is : " JULONG_FORMAT, format, hier_memlimit)

if (hier_memlimit >= _unlimited_memory) {

log_trace(os, container)("Hierarchical Memory and Swap Limit is: Unlimited");

} else {

return (jlong)hier_memlimit;

}

}

return (jlong)-1;

} else {

return (jlong)memswlimit;

}

}

Does this also need to be changed in the test?

jdk/test/lib/jdk/test/lib/containers/cgroup/MetricsTesterCgroupV1.java

Lines 291 to 296 in c63fabe

oldVal = metrics.getMemoryAndSwapLimit();

newVal = getLongValueFromFile(Controller.MEMORY, "memory.memsw.limit_in_bytes");

newVal = newVal > unlimited_minimum ? CgroupSubsystem.LONG_RETVAL_UNLIMITED : newVal;

if (!CgroupMetricsTester.compareWithErrorMargin(oldVal, newVal)) {

fail(Controller.MEMORY, "memory.memsw.limit_in_bytes", oldVal, newVal);

}

This condition returns false,The following code is skipped

jdk/test/lib/jdk/test/lib/containers/cgroup/MetricsTesterCgroupV1.java

Line 284 in c63fabe

if (metrics.getMemoryAndSwapLimit() > metrics.getMemoryLimit()) {

iklam · 2022-04-20T06:28:42Z

Both the PR and the bug report are not clear about exactly what the problem is.

Could you provide a test case that demonstrates the relationship between memory.memsw.limit_in_bytes and memory.swappiness? It will be best if you can upload a shell script into the bug report so we know the exact steps to reproduce the problem.

Ultimately we should add a new jtreg test case.

# scenario 1
memory.memsw.limit_in_bytes = 1000M
memory.limit_in_bytes = 50M
memory.swappiness = 1

-> "java -Xms60m -XX:+AlwaysPreTouch -version" can successfully complete

# scenario 2
memory.memsw.limit_in_bytes = 1000M
memory.limit_in_bytes = 50M
memory.swappiness = 0

-> "java -Xms60m -XX:+AlwaysPreTouch -version" will be killed by cgroups

Related to your other PR (#8256), I think CgroupV1Subsystem::memory_and_swap_limit_in_bytes() also need to be changed so that it will return 50M in scenario 2.

iklam · 2022-04-20T06:53:39Z

I changed the JBS issue summary to "CgroupV1 detection code should consider memory.swappiness"

jerboaa · 2022-04-20T18:41:06Z

test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java

+            opts.addDockerOpts("--memory-swappiness", "0");
+        } else {
+            opts.addDockerOpts("--memory-swappiness", "60");


Unfortunately this breaks on a cgroups v2 system as --memory-swappiness is not supported there. I'd prefer if this wouldn't piggy back on the existing test, but actually assert that swap is properly reported as the same as the memory limit if --memory-swappiness=0. Also, this test only verifies the Java (core-libs) change, not the hotspot change. That would have to be done via some TestMisc variant which uses print_container_info().

xpbob · 2022-04-27T11:19:10Z

Thanks for review.
@iklam @jerboaa
I added test to TestMisc with print_container_info and -XshowSettings:system

xpbob · 2022-04-27T11:28:15Z

I refer to the content of this test

jdk/test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java

Line 119 in 0f81d8f

    
           .addDockerOpts("--memory", dockerMemLimit, "--memory-swappiness", "0", "--memory-swap", dockerMemLimit);

jerboaa · 2022-04-27T14:24:50Z

test/hotspot/jtreg/containers/docker/TestMisc.java

@@ -57,6 +57,7 @@ public static void main(String[] args) throws Exception {
            testIsContainerized();
            testPrintContainerInfo();
            testPrintContainerInfoActiveProcessorCount();
+            testPrintContainerMemoryInfo("100M", "150M");


Again. This test runs unconditionally. --memory-swappiness is not supported in cgroups v2. Thus, the test will fail on a cgroups v2 system. You need to only run this test on a cgroups v1 system. Have a look at test/lib/jdk/test/lib/containers/cgroup/MetricsTester.java how could could detect this and only run on cgroupv1 providers.

On my cgroups v2 system (i.e. it's using 200m memory + 200m swap):

$ sudo docker run --rm -ti --memory-swappiness=0 --memory=200m fedora:35 WARNING: Your kernel does not support memory swappiness capabilities or the cgroup is not mounted. Memory swappiness discarded. [root@fb3182c8e23c /]# cat /sys/fs/cgroup/memory.max 209715200 [root@fb3182c8e23c /]# cat /sys/fs/cgroup/memory.swap.max 209715200

xpbob · 2022-05-03T09:11:56Z

Thanks for review.
@jerboaa
I added new test cases to run only in cgroupv1 environment

xpbob · 2022-05-03T13:38:53Z

Thanks for review.
@jerboaa
The code has been updated

jerboaa

This looks much better. Thanks!

jerboaa · 2022-05-03T14:34:51Z

test/hotspot/jtreg/containers/docker/TestMemoryWithCgroupV1.java

+        Common.addWhiteBoxOpts(opts);
+
+        OutputAnalyzer out = Common.run(opts);
+        System.out.println(out.getOutput());


This looks like debug output?

test/hotspot/jtreg/containers/docker/TestMemoryWithCgroupV1.java

+            System.out.println("TEST PASSED!!!");
+            return;
+        }
+        if ("cgroupv1".equals(metrics.getProvider())) {


xpbob · 2022-05-04T01:34:26Z

Thanks for review.
@jerboaa
The code has been updated

jerboaa · 2022-05-04T09:16:43Z

src/hotspot/os/linux/cgroupV1Subsystem_linux.cpp

+    GET_CONTAINER_INFO(julong, _memory->controller(), "/memory.swappiness",
+                       "Swappiness is: " JULONG_FORMAT, JULONG_FORMAT, swappiness);
+    if (swappiness == 0) {
+      jlong memlimit = read_memory_limit_in_bytes();
+      log_trace(os, container)("Memory and Swap Limit has been reset to " JULONG_FORMAT " because swappiness is 0", memlimit);
+      return memlimit;
+    }


It now occurs to me that we'd need the same treatment of this on line 143. Perhaps extract a function and call it in both places?

xpbob · 2022-05-12T09:52:39Z

Thanks for review.
@jerboaa
I recently had a minor operation, so I couldn't get back on time.
Read values in different places, I added a method read_mem_swappiness
Is that possible?

jerboaa

This looks good. I think the only thing missing is a test for the JDK side. Perhaps write one using the OperatingSystemMXBean's getTotalSwapSpaceSize() method within a container with --memory=200m --memory-swap=250m and using --memory-swappiness=0. You could use test/hotspot/jtreg/containers/docker/TestMemoryAwareness.java and CheckOperatingSystemMXBean.java as a model. Again a cgroupsv1 specific test.

xpbob · 2022-05-12T13:34:01Z

Thanks for review.
@jerboaa
Added test in TestMemoryWithCgroupV1

jerboaa

LGTM. Consider a better name for the test :)

jerboaa · 2022-05-12T13:32:45Z

test/hotspot/jtreg/containers/docker/TestMemoryWithCgroupV1.java

+                .shouldContain("Memory & Swap Limit: " + expectedLimit);
+    }
+
+    private static void testOperatingSystemMXBeanAwareness(String memoryAllocation, String swapAllocation,


Please use a more telling name for this. Perhaps this? testOSBeanSwappinessMemory.

openjdk · 2022-05-12T13:36:34Z

@xpbob This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8284950: CgroupV1 detection code should consider memory.swappiness

Reviewed-by: sgehwolf, iklam

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 414 new commits pushed to the master branch:

e2448ce: 8286791: CLONE - ProblemList compiler/c2/irTests/TestSkeletonPredicates.java in -Xcomp mode
357f990: 8286428: AlgorithmId should understand PBES2
f4f1ddd: 8284194: Allow empty subject fields in keytool
dc94621: 8286782: Exclude vmTestbase/gc/gctests/WeakReference/weak006/weak006.java
0e4bece: 8286705: GCC 12 reports use-after-free potential bugs
63bd3b7: 8286773: cleanup @returns in sun.security classes
af24d2d: 8286771: workaround implemented for JDK-8282607 is incomplete
80cf9f3: 8286594: (zipfs) Mention paths with dot elements in ZipException and cleanups
29c4b8e: 8286444: javac errors after JDK-8251329 are not helpful enough to find root cause
9eb15c9: 8286681: ShenandoahControlThread::request_gc misses the case of GCCause::_codecache_GC_threshold
... and 404 more: https://git.openjdk.java.net/jdk/compare/21ea740e1da48054ee46efda493d0812a35d786e...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@jerboaa, @iklam) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

xpbob · 2022-05-12T13:41:53Z

Thanks for review.
@jerboaa
The expression is clearer after the change

jerboaa

LGTM.

xpbob · 2022-05-16T02:12:54Z

@iklam , are you also fine with the latest change? Thanks.

iklam

LGTM

xpbob · 2022-05-16T06:02:24Z

Thanks @jerboaa and @iklam . :)

xpbob · 2022-05-16T06:03:04Z

/integrate

openjdk · 2022-05-16T06:04:29Z

@xpbob
Your change (at version 584488f) is now ready to be sponsored by a Committer.

DamonFool · 2022-05-16T06:39:08Z

/sponsor

openjdk · 2022-05-16T06:41:04Z

Going to push as commit 46d208f.
Since your change was applied there have been 414 commits pushed to the master branch:

e2448ce: 8286791: CLONE - ProblemList compiler/c2/irTests/TestSkeletonPredicates.java in -Xcomp mode
357f990: 8286428: AlgorithmId should understand PBES2
f4f1ddd: 8284194: Allow empty subject fields in keytool
dc94621: 8286782: Exclude vmTestbase/gc/gctests/WeakReference/weak006/weak006.java
0e4bece: 8286705: GCC 12 reports use-after-free potential bugs
63bd3b7: 8286773: cleanup @returns in sun.security classes
af24d2d: 8286771: workaround implemented for JDK-8282607 is incomplete
80cf9f3: 8286594: (zipfs) Mention paths with dot elements in ZipException and cleanups
29c4b8e: 8286444: javac errors after JDK-8251329 are not helpful enough to find root cause
9eb15c9: 8286681: ShenandoahControlThread::request_gc misses the case of GCCause::_codecache_GC_threshold
... and 404 more: https://git.openjdk.java.net/jdk/compare/21ea740e1da48054ee46efda493d0812a35d786e...master

Your commit was automatically rebased without conflicts.

openjdk · 2022-05-16T06:41:22Z

@DamonFool @xpbob Pushed as commit 46d208f.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8284950: Swappiness disables swap space usage

52cd641

openjdk bot added the rfr Pull request is ready for review label Apr 18, 2022

openjdk bot added the core-libs core-libs-dev@openjdk.org label Apr 18, 2022

iklam reviewed Apr 18, 2022

View reviewed changes

add test and hotspot data

cda7f85

xpbob changed the title ~~8284950: Swappiness disables swap space usage~~ 8284950: CgroupV1 detection code should consider memory.swappiness Apr 20, 2022

jerboaa suggested changes Apr 20, 2022

View reviewed changes

bobpengxie added 4 commits April 27, 2022 18:17

add test in hotspot

2defbf9

revert test file

9bf3094

revert file

f88ad50

revert file

9dd1bbf

jerboaa suggested changes Apr 27, 2022

View reviewed changes

bobpengxie added 3 commits May 3, 2022 16:03

add cgroup v1 test

46bc33b

add test

f0d1f8a

add test

8c46be1

openjdk bot removed the rfr Pull request is ready for review label May 3, 2022

bobpengxie added 2 commits May 3, 2022 16:08

revert TestMisc

13688f1

remove whitespace

f7998d7

openjdk bot added the rfr Pull request is ready for review label May 3, 2022

bobpengxie added 2 commits May 3, 2022 16:26

remove debug info

77676ed

format code

97a547f

remove no use copy

ed2f285

jerboaa reviewed May 3, 2022

View reviewed changes

update code

abd5a2b

jerboaa reviewed May 4, 2022

View reviewed changes

add swap method

296a409

jerboaa reviewed May 12, 2022

View reviewed changes

add test

93bc46a

jerboaa approved these changes May 12, 2022

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label May 12, 2022

rename method

584488f

jerboaa approved these changes May 12, 2022

View reviewed changes

iklam approved these changes May 16, 2022

View reviewed changes

openjdk bot added the sponsor Pull request is ready to be sponsored label May 16, 2022

openjdk bot added the integrated Pull request has been integrated label May 16, 2022

openjdk bot closed this May 16, 2022

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review sponsor Pull request is ready to be sponsored labels May 16, 2022

	oldVal = metrics.getMemoryAndSwapLimit();
	newVal = getLongValueFromFile(Controller.MEMORY, "memory.memsw.limit_in_bytes");
	newVal = newVal > unlimited_minimum ? CgroupSubsystem.LONG_RETVAL_UNLIMITED : newVal;
	if (!CgroupMetricsTester.compareWithErrorMargin(oldVal, newVal)) {
	fail(Controller.MEMORY, "memory.memsw.limit_in_bytes", oldVal, newVal);
	}

	jlong CgroupV1Subsystem::memory_and_swap_limit_in_bytes() {
	GET_CONTAINER_INFO(julong, _memory->controller(), "/memory.memsw.limit_in_bytes",
	"Memory and Swap Limit is: " JULONG_FORMAT, JULONG_FORMAT, memswlimit);
	if (memswlimit >= _unlimited_memory) {
	log_trace(os, container)("Non-Hierarchical Memory and Swap Limit is: Unlimited");
	CgroupV1MemoryController* mem_controller = reinterpret_cast<CgroupV1MemoryController*>(_memory->controller());
	if (mem_controller->is_hierarchical()) {
	const char* matchline = "hierarchical_memsw_limit";
	const char* format = "%s " JULONG_FORMAT;
	GET_CONTAINER_INFO_LINE(julong, _memory->controller(), "/memory.stat", matchline,
	"Hierarchical Memory and Swap Limit is : " JULONG_FORMAT, format, hier_memlimit)
	if (hier_memlimit >= _unlimited_memory) {
	log_trace(os, container)("Hierarchical Memory and Swap Limit is: Unlimited");
	} else {
	return (jlong)hier_memlimit;
	}
	}
	return (jlong)-1;
	} else {
	return (jlong)memswlimit;
	}
	}

8284950: CgroupV1 detection code should consider memory.swappiness #8285

8284950: CgroupV1 detection code should consider memory.swappiness #8285

Conversation

xpbob commented Apr 18, 2022 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Apr 18, 2022

openjdk bot commented Apr 18, 2022

mlbridge bot commented Apr 18, 2022 • edited Loading

Webrevs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iklam commented Apr 20, 2022

iklam commented Apr 20, 2022

Choose a reason for hiding this comment

xpbob commented Apr 27, 2022

xpbob commented Apr 27, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xpbob commented May 3, 2022

xpbob commented May 3, 2022

jerboaa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

xpbob commented May 4, 2022

Choose a reason for hiding this comment

xpbob commented May 12, 2022

jerboaa left a comment

Choose a reason for hiding this comment

xpbob commented May 12, 2022

jerboaa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openjdk bot commented May 12, 2022 • edited Loading

xpbob commented May 12, 2022

jerboaa left a comment

Choose a reason for hiding this comment

xpbob commented May 16, 2022

iklam left a comment

Choose a reason for hiding this comment

xpbob commented May 16, 2022

xpbob commented May 16, 2022

openjdk bot commented May 16, 2022

DamonFool commented May 16, 2022

openjdk bot commented May 16, 2022

openjdk bot commented May 16, 2022

xpbob commented Apr 18, 2022 •

edited by openjdk bot

Loading

mlbridge bot commented Apr 18, 2022 •

edited

Loading

openjdk bot commented May 12, 2022 •

edited

Loading