CRIU support: add checkpoint/restore environment variables #486

JasonFengJ9 · 2022-03-11T14:04:26Z

Added isCRIUEnabled via CRIUProvider.getOpenJ9CRIU();
Added tracePrunedEnvVarsValue via org.eclipse.openj9.criu.TracePrunedEnvVars;
Added theOriginalUnmodifiableEnvironment to store original theUnmodifiableEnvironment;
Hard-coded immutable list: LANG, LC_ALL and LC_CTYPE;

related eclipse-openj9/openj9#13545
depends on eclipse-openj9/openj9#14730

Signed-off-by: Jason Feng fengj@ca.ibm.com

JasonFengJ9 · 2022-03-11T14:07:53Z

Jenkins test sanity.functional zlinux jdk11

src/java.base/unix/classes/java/lang/ProcessEnvironment.java

tajila · 2022-03-14T14:15:13Z

Were you able to try this patch with Toms docker script?

JasonFengJ9 · 2022-03-14T15:04:14Z

Were you able to try this patch with Toms docker script?

./open-liberty-build.sh -t pingperf-ubuntu:

Launching defaultServer (Open Liberty 22.0.0.3/wlp-1.0.62.cl220320220205-1900) on Eclipse OpenJ9 VM, version 11.0.15-internal+0-adhoc.root.openj9-openjdk-jdk11 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKT0016I: Web application available (default_host): http://1aafba4b59be:9080/pingperf/
[AUDIT   ] CWWKZ0001I: Application pingperf started in 1.911 seconds.
0
Committing container pingperf-ubuntu-checkpoint as image pingperf-ubuntu
sha256:e60ae7ed2d039f99128c535cc2ee8412124f18154991a518970adeed57e2d323
Done building image pingperf-ubuntu ... cleaning up
pingperf-ubuntu-checkpoint

./liberty-restore.sh -t pingperf-ubuntu:

Performing checkpoint --at=applications

Launching defaultServer (Open Liberty 22.0.0.3/wlp-1.0.62.cl220320220205-1900) on Eclipse OpenJ9 VM, version 11.0.15-internal+0-adhoc.root.openj9-openjdk-jdk11 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.

JasonFengJ9 · 2022-03-14T20:23:24Z

This PR appears avoid the liberty restore hang problem.

Added -Dorg.eclipse.openj9.criu.TracePrunedEnvVars=1, ./open-liberty-build.sh -t pingperf-ubuntu:

The env var (INVOKED) is not in CRIU immutable list but was set to : /opt/ol/wlp/bin/server
The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
The env var (WLP_OUTPUT_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/output
The env var (X_LOG_DIR) is not in CRIU immutable list but was set to : /logs
The env var (X_LOG_FILE) is not in CRIU immutable list but was set to : console.log
The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
Launching defaultServer (Open Liberty 22.0.0.3/wlp-1.0.62.cl220320220205-1900) on Eclipse OpenJ9 VM, version 11.0.15-internal+0-adhoc.root.openj9-openjdk-jdk11 (en_US)
The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[err] The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
[err] The env var (WLP_OUTPUT_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/output
[err] The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[err] The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
0
Committing container pingperf-ubuntu-checkpoint as image pingperf-ubuntu
sha256:51d3cdb67a392b5a4736327a404fd9ce13518520c64cf448c55f0e84450c1128
Done building image pingperf-ubuntu ... cleaning up
pingperf-ubuntu-checkpoint

./liberty-restore.sh -t pingperf-ubuntu:

Performing checkpoint --at=applications

The env var (INVOKED) is not in CRIU immutable list but was set to : /opt/ol/wlp/bin/server
The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
The env var (WLP_OUTPUT_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/output
The env var (X_LOG_DIR) is not in CRIU immutable list but was set to : /logs
The env var (X_LOG_FILE) is not in CRIU immutable list but was set to : console.log
The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
Launching defaultServer (Open Liberty 22.0.0.3/wlp-1.0.62.cl220320220205-1900) on Eclipse OpenJ9 VM, version 11.0.15-internal+0-adhoc.root.openj9-openjdk-jdk11 (en_US)
The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[err] The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
[err] The env var (WLP_OUTPUT_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/output
[err] The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[err] The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr

tajila · 2022-03-15T13:24:34Z

to know if it works you need to see CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 0.779 seconds.

src/java.base/unix/classes/java/lang/ProcessEnvironment.java

JasonFengJ9 · 2022-03-16T13:05:53Z

to know if it works you need to see CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 0.779 seconds.

There is a message [AUDIT ] CWWKE0001I: The server defaultServer has been launched.

./liberty-restore.sh -t pingperf-ubuntu
Performing checkpoint --at=applications

The env var (INVOKED) is not in CRIU immutable list but was set to : /opt/ol/wlp/bin/server
The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
The env var (WLP_OUTPUT_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/output
The env var (X_LOG_DIR) is not in CRIU immutable list but was set to : /logs
The env var (X_LOG_FILE) is not in CRIU immutable list but was set to : console.log
The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
Launching defaultServer (Open Liberty 22.0.0.3/wlp-1.0.62.cl220320220205-1900) on Eclipse OpenJ9 VM, version 11.0.15-internal+0-adhoc.root.openj9-openjdk-jdk11 (en_US)
The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[err] The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
[err] The env var (WLP_OUTPUT_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/output
[err] The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[err] The env var (WLP_USER_DIR) is not in CRIU immutable list but was set to : /opt/ol/wlp/usr

tajila · 2022-03-16T13:34:02Z

you should see

[AUDIT   ] CWWKG0017I: The server configuration was successfully updated in 0.176 seconds.
[AUDIT   ] CWWKF0012I: The server installed the following features: [cdi-2.0, checkpoint-1.0, jaxrs-2.1, jaxrsClient-2.1, jndi-1.0, jsonp-1.1, servlet-4.0].
[AUDIT   ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 0.789 seconds.

tajila · 2022-03-16T13:34:44Z

dont worry, I've confirmed that eclipse-openj9/openj9#14583 fixes it

JasonFengJ9 · 2022-03-16T15:30:19Z

So the restore wasn't successful w/ this PR (probably stopped earlier to avoid the hang), will give it a try w/ eclipse-openj9/openj9#14583

src/java.base/unix/classes/java/lang/ProcessEnvironment.java

Added isCRIUEnabled via CRIUProvider.getOpenJ9CRIU(); Added tracePrunedEnvVarsValue via org.eclipse.openj9.criu.TracePrunedEnvVars; Added theOriginalUnmodifiableEnvironment to store original theUnmodifiableEnvironment; Hard-coded immutable list: LANG, LC_ALL and LC_CTYPE; Signed-off-by: Jason Feng <fengj@ca.ibm.com>

tajila · 2022-03-22T15:55:13Z

jenkins compile xlinuxcriu jdk11

tajila · 2022-03-22T15:55:48Z

Jenkins test sanity plinux jdk8,jdk11,jdknext

tajila · 2022-03-22T16:45:28Z

Jenkins test sanity plinux jdk8,jdk11,jdknext

tajila · 2022-03-22T18:05:42Z

Jenkins test sanity plinux jdk11

tajila · 2022-03-22T18:07:13Z

Infra failure on last run

Exception: org.jenkinsci.plugins.workflow.steps.FlowInterruptedException

tajila · 2022-03-22T18:08:56Z

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_ppc64le_linux_Personal/182/console

JasonFengJ9 · 2022-03-23T13:35:28Z

Assuming there is no immediate need to port this PR to other Java levels.
@tajila is that correct?

tajila · 2022-03-23T13:49:29Z

we should port to 17 & next since long term we need to support all jdk levels.

We can wait on jdk8 until we know that we need it for sure

JasonFengJ9 requested a review from tajila March 11, 2022 14:04

tajila reviewed Mar 14, 2022

View reviewed changes

JasonFengJ9 force-pushed the criuenv branch from 149e004 to ccdda33 Compare March 14, 2022 20:16

tajila reviewed Mar 15, 2022

View reviewed changes

src/java.base/unix/classes/java/lang/ProcessEnvironment.java Show resolved Hide resolved

src/java.base/unix/classes/java/lang/ProcessEnvironment.java Show resolved Hide resolved

JasonFengJ9 force-pushed the criuenv branch from ccdda33 to 0fe7474 Compare March 15, 2022 20:41

JasonFengJ9 mentioned this pull request Mar 16, 2022

InternalCRIUSupport & APIs are available for all builds eclipse-openj9/openj9#14730

Merged

JasonFengJ9 force-pushed the criuenv branch 2 times, most recently from 2fa868e to a020153 Compare March 16, 2022 17:19

tajila reviewed Mar 21, 2022

View reviewed changes

src/java.base/unix/classes/java/lang/ProcessEnvironment.java Show resolved Hide resolved

JasonFengJ9 force-pushed the criuenv branch from a020153 to 6f41d57 Compare March 21, 2022 23:42

tajila approved these changes Mar 22, 2022

View reviewed changes

tajila merged commit 2e2a51b into ibmruntimes:openj9 Mar 23, 2022

JasonFengJ9 deleted the criuenv branch March 23, 2022 13:33

This was referenced Mar 28, 2022

CRIU: Performance regression ibmruntimes/openj9-openjdk-jdk#415

Closed

CRIU: Startup performance regression eclipse-openj9/openj9#14789

Closed

This was referenced Apr 11, 2022

CRIU: Backport CRIUSupport to JDK8 eclipse-openj9/openj9#14901

Open

Temporarily disable CRIU support on JDK8 ibmruntimes/openj9-openjdk-jdk8#558

Merged

This was referenced Apr 12, 2022

CRIU support w/ JPP : add checkpoint/restore environment variables ibmruntimes/openj9-openjdk-jdk17#87

Merged

CRIU support w/ JPP : add checkpoint/restore environment variables ibmruntimes/openj9-openjdk-jdk#423

Merged

JasonFengJ9 mentioned this pull request Dec 7, 2022

Port CRIU related PRs from openj9-openjdk-jdk17 and handle thread interrupted status at CRIU single threaded mode ibmruntimes/openj9-openjdk-jdk19#54

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CRIU support: add checkpoint/restore environment variables #486

CRIU support: add checkpoint/restore environment variables #486

JasonFengJ9 commented Mar 11, 2022 •

edited

JasonFengJ9 commented Mar 11, 2022

tajila commented Mar 14, 2022

JasonFengJ9 commented Mar 14, 2022

JasonFengJ9 commented Mar 14, 2022

tajila commented Mar 15, 2022

JasonFengJ9 commented Mar 16, 2022

tajila commented Mar 16, 2022

tajila commented Mar 16, 2022

JasonFengJ9 commented Mar 16, 2022

tajila commented Mar 22, 2022

tajila commented Mar 22, 2022

tajila commented Mar 22, 2022

tajila commented Mar 22, 2022

tajila commented Mar 22, 2022

tajila commented Mar 22, 2022

JasonFengJ9 commented Mar 23, 2022

tajila commented Mar 23, 2022

CRIU support: add checkpoint/restore environment variables #486

CRIU support: add checkpoint/restore environment variables #486

Conversation

JasonFengJ9 commented Mar 11, 2022 • edited

JasonFengJ9 commented Mar 11, 2022

tajila commented Mar 14, 2022

JasonFengJ9 commented Mar 14, 2022

JasonFengJ9 commented Mar 14, 2022

tajila commented Mar 15, 2022

JasonFengJ9 commented Mar 16, 2022

tajila commented Mar 16, 2022

tajila commented Mar 16, 2022

JasonFengJ9 commented Mar 16, 2022

tajila commented Mar 22, 2022

tajila commented Mar 22, 2022

tajila commented Mar 22, 2022

tajila commented Mar 22, 2022

tajila commented Mar 22, 2022

tajila commented Mar 22, 2022

JasonFengJ9 commented Mar 23, 2022

tajila commented Mar 23, 2022

JasonFengJ9 commented Mar 11, 2022 •

edited