8265836: OperatingSystemImpl.getCpuLoad() returns incorrect CPU load inside a container #3656

tanghaoth90 · 2021-04-23T13:33:42Z

OperatingSystemImpl.getCpuLoad() may return 1.0 in a container, even though the CPU load is obviously below 100%.

We created a 5-core container and run 4 "while (true)" loops in the container. OperatingSystemImpl.getCpuLoad() returned 1.0, which is incorrect (0.8 is correct).
"systemLoad" in getCpuLoad() is exactly 4.0 before "systemLoad = Math.min(1.0, systemLoad);". The problem is caused by using the elapsed time (specified by "cpu.cfs_period_us") instead of the total CPU time (specified by "cpu.cfs_quota_us"). Therefore, it is more reasonable to divide cpu usage time by "quotaNanos" instead of "elapsedNanos".

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8265836: OperatingSystemImpl.getCpuLoad() returns incorrect CPU load inside a container

Reviewers

Severin Gehwolf (@jerboaa - Reviewer)
Yasumasa Suenaga (@YaSuenag - Reviewer)

Contributors

Shaojun Wang <jeffery.wsj@alibaba-inc.com>
Severin Gehwolf <sgehwolf@openjdk.org>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3656/head:pull/3656
$ git checkout pull/3656

Update a local copy of the PR:
$ git checkout pull/3656
$ git pull https://git.openjdk.java.net/jdk pull/3656/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3656

View PR using the GUI difftool:
$ git pr show -t 3656

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3656.diff

Co-authored-by: Shaojun Wang <jeffery.wsj@alibaba-inc.com>

bridgekeeper · 2021-04-23T13:34:06Z

Hi @tanghaoth90, welcome to this OpenJDK project and thanks for contributing!

We do not recognize you as Contributor and need to ensure you have signed the Oracle Contributor Agreement (OCA). If you have not signed the OCA, please follow the instructions. Please fill in your GitHub username in the "Username" field of the application. Once you have signed the OCA, please let us know by writing /signed in a comment in this pull request.

If you already are an OpenJDK Author, Committer or Reviewer, please click here to open a new issue so that we can record that fact. Please use "Add GitHub user tanghaoth90" as summary for the issue.

If you are contributing this work on behalf of your employer and your employer has signed the OCA, please let us know by writing /covered in a comment in this pull request.

openjdk · 2021-04-23T13:35:22Z

@tanghaoth90 The following labels will be automatically applied to this pull request:

jmx
serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

tanghaoth90 · 2021-04-23T13:55:40Z

/signed

tanghaoth90 · 2021-04-23T13:55:56Z

/covered

bridgekeeper · 2021-04-23T13:56:30Z

Thank you! Please allow for up to two weeks to process your OCA, although it is usually done within one to two business days. Also, please note that pull requests that are pending an OCA check will not usually be evaluated, so your patience is appreciated!

bridgekeeper · 2021-04-23T13:56:36Z

Thank you! Please allow for a few business days to verify that your employer has signed the OCA. Also, please note that pull requests that are pending an OCA check will not usually be evaluated, so your patience is appreciated!

openjdk · 2021-04-23T14:06:44Z

@tanghaoth90 Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

/contributor add @openjdk-bot
/contributor add duke
/contributor add J. Duke <duke@openjdk.org>

tanghaoth90 · 2021-04-23T14:08:25Z

/contributor add Shaojun Wang jeffery.wsj@alibaba-inc.com

openjdk · 2021-04-23T14:09:05Z

@tanghaoth90
Contributor Shaojun Wang <jeffery.wsj@alibaba-inc.com> successfully added.

mlbridge · 2021-05-03T14:42:59Z

Webrevs

argha-c · 2021-05-04T04:07:52Z

src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java

-                    long elapsedNanos = TimeUnit.MICROSECONDS.toNanos(periodLength * numPeriods);
-                    double systemLoad = (double) usageNanos / elapsedNanos;
+                if (numPeriods > 0 && usageNanos > 0) {
+                    long quotaNanos = TimeUnit.MICROSECONDS.toNanos(quota * numPeriods);


We happened to hit an exactly similar problem when running on a container with openjdk15.

Given we effectively agree that the problem is elapsedNanos doesn't accurately reflect the cpu time allocated across all shares vs a single share, my proposal was to use getCpuShares as a multiplier for periodLength above.
Is there a good reason getCpuQuota is a better alternative?

Hi Argha, thanks a lot for your suggestion. I think both "quota" and "share" are worth considering. Let us look into the implementation of CgroupSubsystem::active_processor_count() in OpenJDK HotSpot (https://github.com/openjdk/jdk/blob/master/src/hotspot/os/linux/cgroupSubsystem_linux.cpp).

Thanks for linking that. It sounds reasonable to me to prefer quota in that case.

Thanks for linking that. It sounds reasonable to me to prefer quota in that case.

Yes, flag PreferContainerQuotaForCPUCount is true by default. Therefore, my current implementation might be a minimal implementation.

We can also cover the case where PreferContainerQuotaForCPUCount is false. There are two different ways.

To access the value of PreferContainerQuotaForCPUCount, so that we can decide if we should use quota or (quota & share);

Reuse CgroupSubsystem::active_processor_count. However, the function returns an integer. It is more reasonable to use a floating number.

Looking forward to your suggestion.

We happened to hit an exactly similar problem when running on a container with openjdk15.

Given we effectively agree that the problem is elapsedNanos doesn't accurately reflect the cpu time allocated across all shares vs a single share, my proposal was to use getCpuShares as a multiplier for periodLength above.
Is there a good reason getCpuQuota is a better alternative?

@argha-c The proposed fix is within the quota > 0 condition. I.e. this is code only run when CPU quotas, not shares are in effect. In docker/podman speach these are --cpu-quota=... and --cpu-period=.... switches. So no, in this case it wouldn't make sense to use cpu shares info in a branch which looks at cpu quotas ;-)

@argha-c The proposed fix is within the quota > 0 condition. I.e. this is code only run when CPU quotas, not shares are in effect. In docker/podman speach these are --cpu-quota=... and --cpu-period=.... switches. So no, in this case it wouldn't make sense to use cpu shares info in a branch which looks at cpu quotas ;-)

@jerboaa : Correct. My comment was less specific to the branch, and more to highlight that a fix here needs to consider the case for both quota and shares. I see the bug report has been updated to reflect that. Cheers.

OK. Thanks for clarifying.

@tanghaoth90 My local testing suggests that your fix addresses the issue of CPU quotas set via --cpu-quota/--cpu-period. When using --cpu-shares the CPU load calculation is wrong since it will (wrongly) report host values. Lets look at them individually, fix the quota and shares case individually (i.e. when not both are set). Once that's done, quota will be preferred in the OperatingSystemMXBean impl, which is reasonable. I don't think we need to account for the shares-preferred-over-quota at this point since that changed in HotSpot in JDK 11 time-frame (JDK-8197589) and OperatingSystemMXBean has only been made container aware in JDK 14 (yes, it got backported, but still).

@jerboaa I have updated my fix by inserting a new branch for --cpu-shares. By setting up containers with different restrictions based on your suggestion, I noticed two problems for the CPU load calculation. Take TestLoad.java as the example.

--cpu-quota=700000 --cpu-period=100000 as the restriction: the result is getting close to 6/7 slowly as time goes by, which indicates that the result of

long usageNanos = containerMetrics.getCpuUsage();

is an accumulated CPU usage rather than a real-time CPU usage.
According to the javadoc, getCpuLoad() returns the "recent cpu usage". The current fix obviously does not address this issue.

--cpu-shares=2048 as the restriction: the "cpu-share" branch only returns -1 since containerMetrics.getCpuNumPeriods() returns 0 (nr_periods of cpu.stat stays 0 when only --cpu-shares is set). By now, I have no idea to compute the elapsed time or the total available CPU time with the help of the methods of jdk.internal.platform.CgroupMetrics.

Any suggestion is appreciated. @jerboaa @argha-c

@tanghaoth90

As to your observation of 1). Yes, that's true and I'm not sure it's worth changing. We could change the javadoc.

As for 2) I've created a PR for your branch with a suggested implementation for cpu shares based accounting. It uses a similar heuristics than the getCpuLoad0() uses here:
https://github.com/openjdk/jdk/blob/master/src/jdk.management/linux/native/libmanagement_ext/UnixOperatingSystem.c#L278

That's the best I could think of. It's certainly more accurate than the current implementation.

@jerboaa Just read your comment about my observation (due to the weird format of github pull request). Sorry for making further change for the quota branch without asking for your suggestion. For now, I still hold a different opinion about the quota branch. My local test shows that

podman run --cpu-shares=4096 <image> podman run --cpu-quota=400000 --cpu-period=100000 <image>

yields the same result after the change, otherwise the second one still prints the average cpu load.

jerboaa

@tanghaoth90 There are a couple of ways how CPU restrictions can be enforced. It appears that the current patch fixes the cpuacct controller case (--cpu-quota), but doesn't yet address --cpu-shares case. I've made a note of that in the bug.

jerboaa · 2021-05-06T12:36:09Z

src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java

-                    long elapsedNanos = TimeUnit.MICROSECONDS.toNanos(periodLength * numPeriods);
-                    double systemLoad = (double) usageNanos / elapsedNanos;
+                if (numPeriods > 0 && usageNanos > 0) {
+                    long quotaNanos = TimeUnit.MICROSECONDS.toNanos(quota * numPeriods);


We happened to hit an exactly similar problem when running on a container with openjdk15.

Given we effectively agree that the problem is elapsedNanos doesn't accurately reflect the cpu time allocated across all shares vs a single share, my proposal was to use getCpuShares as a multiplier for periodLength above.
Is there a good reason getCpuQuota is a better alternative?

@argha-c The proposed fix is within the quota > 0 condition. I.e. this is code only run when CPU quotas, not shares are in effect. In docker/podman speach these are --cpu-quota=... and --cpu-period=.... switches. So no, in this case it wouldn't make sense to use cpu shares info in a branch which looks at cpu quotas ;-)

jerboaa · 2021-05-06T13:31:50Z

src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java

-                    long elapsedNanos = TimeUnit.MICROSECONDS.toNanos(periodLength * numPeriods);
-                    double systemLoad = (double) usageNanos / elapsedNanos;
+                if (numPeriods > 0 && usageNanos > 0) {
+                    long quotaNanos = TimeUnit.MICROSECONDS.toNanos(quota * numPeriods);


Hi Argha, thanks a lot for your suggestion. I think both "quota" and "share" are worth considering.

@tanghaoth90 My local testing suggests that your fix addresses the issue of CPU quotas set via --cpu-quota/--cpu-period. When using --cpu-shares the CPU load calculation is wrong since it will (wrongly) report host values. Lets look at them individually, fix the quota and shares case individually (i.e. when not both are set). Once that's done, quota will be preferred in the OperatingSystemMXBean impl, which is reasonable. I don't think we need to account for the shares-preferred-over-quota at this point since that changed in HotSpot in JDK 11 time-frame (JDK-8197589) and OperatingSystemMXBean has only been made container aware in JDK 14 (yes, it got backported, but still).

jerboaa · 2021-05-21T16:13:10Z

@tanghaoth90 I've added a comment which should address the cpu shares based cpuLoad problem. Let me know what you think.

jerboaa · 2021-05-25T09:50:36Z

/contributor add @jerboaa

jerboaa · 2021-05-25T09:52:14Z

/reviewers 2

jerboaa · 2021-05-25T09:53:02Z

Bumping number of required reviewers to 2 as I'm a contributing author of the change.

jerboaa

LGTM. Nit: Please remove PER_CPU_SHARES constant as it's no longer used.

jerboaa · 2021-05-25T09:56:50Z

src/jdk.management/unix/classes/com/sun/management/internal/OperatingSystemImpl.java

@@ -41,7 +41,10 @@
    implements com.sun.management.UnixOperatingSystemMXBean {

    private static final int MAX_ATTEMPTS_NUMBER = 10;
+    private static final int PER_CPU_SHARES = 1024;


I think this is no longer used anywhere so can get removed.

openjdk · 2021-05-25T11:48:19Z

@tanghaoth90 This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8265836: OperatingSystemImpl.getCpuLoad() returns incorrect CPU load inside a container

Co-authored-by: Shaojun Wang <jeffery.wsj@alibaba-inc.com>
Co-authored-by: Severin Gehwolf <sgehwolf@openjdk.org>
Reviewed-by: sgehwolf, ysuenaga

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 530 new commits pushed to the master branch:

bea4109: 8187649: ArrayIndexOutOfBoundsException in java.util.JapaneseImperialCalendar
ec65cf8: 8240347: remove undocumented options from jlink --help message
3623abb: 8263087: Add a MethodHandle combinator that switches over a set of MethodHandles
85f6165: 8267817: [TEST] Remove unnecessary init in test/micro/org/openjdk/bench/javax/crypto/full/AESGCMBench:setup
7278f56: 8267800: Remove the '_dirty' set in BCEscapeAnalyzer
bfa46f0: 8252476: as_Worker_thread() doesn't check what it intends
37bc4e2: 8263635: Add --servername option to jhsdb debugd
6ffa3e6: 8267754: cds/appcds/loaderConstraints/LoaderConstraintsTest.java fails on x86_32 due to customized class loader is not supported
1899f02: 8267805: Add UseVtableBasedCHA to the list of JVM flags known to jtreg
0fc7c8d: 8267751: (test) jtreg.SkippedException has no serial VersionUID
... and 520 more: https://git.openjdk.java.net/jdk/compare/191f1fc46c30f7e92ba09d04bc000256442e64ed...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@jerboaa, @YaSuenag) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

openjdk · 2021-05-25T11:49:54Z

@jerboaa Only the author (@tanghaoth90) is allowed to issue the contributor command.

openjdk · 2021-05-25T11:50:16Z

@jerboaa
The number of required reviews for this PR is now set to 2 (with at least 1 of role reviewers).

…hares-based calculation

tanghaoth90 · 2021-05-25T17:35:27Z

/contributor add @jerboaa

openjdk · 2021-05-25T17:36:03Z

@tanghaoth90
Contributor Severin Gehwolf <sgehwolf@openjdk.org> successfully added.

tanghaoth90 · 2021-05-25T17:46:21Z

@jerboaa Thank you for the cpu-shares-based load calculation! I also update the cpu-quota-based load calculation to use the same strategy as the cpu-shares-based one, because my previous implementation always reports the average load since the container starts.
Another change is to set the initial value of usageTicks/totalTicks to zero. Therefore, the first call of getCpuLoad can also report a reasonable result.

jerboaa · 2021-05-26T08:01:40Z

@jerboaa Thank you for the cpu-shares-based load calculation! I also update the cpu-quota-based load calculation to use the same strategy as the cpu-shares-based one, because my previous implementation always reports the average load since the container starts.

OK.

Another change is to set the initial value of usageTicks/totalTicks to zero. Therefore, the first call of getCpuLoad can also report a reasonable result.

Seems reasonable.

jerboaa

Still good.

jerboaa · 2021-05-26T18:00:32Z

Any other reviewers, please? @YaSuenag perhaps?

YaSuenag · 2021-05-27T00:47:23Z

I haven't followed yet all of discussions in this review, but I concern this PR changes the meaning of getCpuLoad().

getCpuLoad() has been based on total time since the start of the container, but after this PR, it is based on the ticks in earlier call. Is it ok?
IMHO it can be accepted because it is the same with load average on Linux, but I concern we may need CSR because this PR changes behavior.

tanghaoth90 · 2021-05-27T08:37:11Z

getCpuLoad() has been based on total time since the start of the container, but after this PR, it is based on the ticks in earlier call. Is it ok?
IMHO it can be accepted because it is the same with load average on Linux, but I concern we may need CSR because this PR changes behavior.

@YaSuenag Thanks for your comment. I can give two reasons for that.

The javadoc of OperatingSystemMXBean.getCpuLoad indicates the method should return "recent cpu usage".

jdk/src/jdk.management/share/classes/com/sun/management/OperatingSystemMXBean.java

Line 156 in 739769c

* Returns the "recent cpu usage" for the operating environment. This value
The initial (container-unaware) implementation getCpuLoad0 does reports recent CPU load, which the container-aware implementation must be consistent.

but after this PR, it is based on the ticks in earlier call. Is it ok?

The result might be too coarse/inaccurate, if the time between two calls is too long/short. Any comments for that?

jerboaa · 2021-05-27T08:53:12Z

I haven't followed yet all of discussions in this review, but I concern this PR changes the meaning of getCpuLoad().

Does it? Here is the javadoc:
https://docs.oracle.com/en/java/javase/16/docs/api/jdk.management/com/sun/management/OperatingSystemMXBean.html#getCpuLoad()

It says:

Returns the "recent cpu usage" for the operating environment. This value is a double in the [0.0,1.0] interval.

If I run outside a container, getCpuLoad0() is being used which uses a similar calculation. So in my book this fixes the bug (earlier reporting is just plain wrong, see bug for details) and just brings it in line with the non-container values.

Also note that the cpu quota-based limits implementation as been changed to also report recent load values. But I'm not sure it's worth discussing this further as the previous implementation (supposedly reporting an average) was wrong.

getCpuLoad() has been based on total time since the start of the container, but after this PR, it is based on the ticks in earlier call. Is it ok?

IMHO, yes as the previous implementation was incorrect. I suspect nobody noticed until now.

IMHO it can be accepted because it is the same with load average on Linux, but I concern we may need CSR because this PR changes behavior.

I disagree. This change makes getCpuLoad() behave similar in all cases: non-container case[1], cpu affinity-based[2], cpu-quota-based[3] and cpu-shares based[3] cpu-limits. That wasn't previously the case.

[1] https://bugs.openjdk.java.net/browse/JDK-8265836?focusedCommentId=14423632&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14423632
[2] https://bugs.openjdk.java.net/browse/JDK-8265836?focusedCommentId=14423633&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14423633
[3] https://bugs.openjdk.java.net/browse/JDK-8265836?focusedCommentId=14423371&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14423371

YaSuenag

@tanghaoth90 @jerboaa Thank you for explanation!

The result might be too coarse/inaccurate, if the time between two calls is too long/short. Any comments for that?

I concerned about that too. Javadoc says "recent cpu usage" about this - it is ambiguous. In other words, our concerns are tolerated IMHO.
Of course it is better if we can get sampling value, but it is difficult now. Maybe we can implement sampling thread, but we need to think some things (e.g. sampling frequency). I'm not sure it is worth to work for it.

I think it is important that getCpuLoad() behaves similar in all cases, and also I can't see any problems in your change. So I give +1 to your change 😉

jerboaa · 2021-05-27T09:34:38Z

@tanghaoth90 @jerboaa Thank you for explanation!

Thanks very much for the review!

tanghaoth90 · 2021-05-27T13:41:49Z

/integrate

openjdk · 2021-05-27T13:43:14Z

@tanghaoth90
Your change (at version eba3bc1) is now ready to be sponsored by a Committer.

jerboaa · 2021-05-27T14:38:54Z

/sponsor

openjdk · 2021-05-27T14:40:53Z

@jerboaa @tanghaoth90 Since your change was applied there have been 531 commits pushed to the master branch:

10a6f5d: 8230623: Extract command-line help for -Xlint sub-options to new --help-lint
bea4109: 8187649: ArrayIndexOutOfBoundsException in java.util.JapaneseImperialCalendar
ec65cf8: 8240347: remove undocumented options from jlink --help message
3623abb: 8263087: Add a MethodHandle combinator that switches over a set of MethodHandles
85f6165: 8267817: [TEST] Remove unnecessary init in test/micro/org/openjdk/bench/javax/crypto/full/AESGCMBench:setup
7278f56: 8267800: Remove the '_dirty' set in BCEscapeAnalyzer
bfa46f0: 8252476: as_Worker_thread() doesn't check what it intends
37bc4e2: 8263635: Add --servername option to jhsdb debugd
6ffa3e6: 8267754: cds/appcds/loaderConstraints/LoaderConstraintsTest.java fails on x86_32 due to customized class loader is not supported
1899f02: 8267805: Add UseVtableBasedCHA to the list of JVM flags known to jtreg
... and 521 more: https://git.openjdk.java.net/jdk/compare/191f1fc46c30f7e92ba09d04bc000256442e64ed...master

Your commit was automatically rebased without conflicts.

Pushed as commit ef368b3.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8265836: OperatingSystemImpl.getCpuLoad() returns incorrect CPU load

b052b62

Co-authored-by: Shaojun Wang <jeffery.wsj@alibaba-inc.com>

bridgekeeper bot added the oca Needs verification of OCA signatory status label Apr 23, 2021

openjdk bot added serviceability serviceability-dev@openjdk.org jmx jmx-dev@openjdk.org labels Apr 23, 2021

bridgekeeper bot added the oca-verify Needs verification of OCA signatory status label Apr 23, 2021

bridgekeeper bot removed oca Needs verification of OCA signatory status oca-verify Needs verification of OCA signatory status labels May 3, 2021

openjdk bot added the rfr Pull request is ready for review label May 3, 2021

argha-c reviewed May 4, 2021

View reviewed changes

jerboaa suggested changes May 6, 2021

View reviewed changes

cpu.shares for OperatingSystemImpl.getCpuLoad()

81867f2

openjdk bot changed the title ~~8265836: OperatingSystemImpl.getCpuLoad() returns incorrect CPU load~~ 8265836: OperatingSystemImpl.getCpuLoad() returns incorrect CPU load inside a container May 11, 2021

jerboaa mentioned this pull request May 21, 2021

Add cpuLoad implementation for cpu shares quota tanghaoth90/jdk#1

Merged

jerboaa and others added 2 commits May 23, 2021 14:53

Add cpuLoad implementation for cpu shares quota

d76fd44

Fix compile errors on MacOS

066083e

jerboaa approved these changes May 25, 2021

View reviewed changes

remove PER_CPU_SHARES

c7abec0

openjdk bot added the ready Pull request is ready to be integrated label May 25, 2021

openjdk bot removed the ready Pull request is ready to be integrated label May 25, 2021

Hao Tang added 2 commits May 25, 2021 21:42

rename usageTicks and totalTicks

663478b

Use historical-value-based formula for both cpu-quota-based and cpu-s…

eba3bc1

…hares-based calculation

jerboaa approved these changes May 26, 2021

View reviewed changes

YaSuenag approved these changes May 27, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label May 27, 2021

openjdk bot added the sponsor Pull request is ready to be sponsored label May 27, 2021

openjdk bot closed this May 27, 2021

openjdk bot added integrated Pull request has been integrated and removed sponsor Pull request is ready to be sponsored ready Pull request is ready to be integrated rfr Pull request is ready for review labels May 27, 2021

8265836: OperatingSystemImpl.getCpuLoad() returns incorrect CPU load inside a container #3656

8265836: OperatingSystemImpl.getCpuLoad() returns incorrect CPU load inside a container #3656

Conversation

tanghaoth90 commented Apr 23, 2021 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Contributors

Reviewing

bridgekeeper bot commented Apr 23, 2021

openjdk bot commented Apr 23, 2021

tanghaoth90 commented Apr 23, 2021

tanghaoth90 commented Apr 23, 2021

bridgekeeper bot commented Apr 23, 2021

bridgekeeper bot commented Apr 23, 2021

openjdk bot commented Apr 23, 2021

tanghaoth90 commented Apr 23, 2021

openjdk bot commented Apr 23, 2021

mlbridge bot commented May 3, 2021 • edited Loading

Webrevs

argha-c May 4, 2021 • edited by bridgekeeper bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerboaa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerboaa commented May 21, 2021

jerboaa commented May 25, 2021

jerboaa commented May 25, 2021

jerboaa commented May 25, 2021

jerboaa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openjdk bot commented May 25, 2021 • edited Loading

openjdk bot commented May 25, 2021

openjdk bot commented May 25, 2021

tanghaoth90 commented May 25, 2021

openjdk bot commented May 25, 2021

tanghaoth90 commented May 25, 2021

jerboaa commented May 26, 2021

jerboaa left a comment

Choose a reason for hiding this comment

jerboaa commented May 26, 2021

YaSuenag commented May 27, 2021

tanghaoth90 commented May 27, 2021 • edited Loading

jerboaa commented May 27, 2021

YaSuenag left a comment

Choose a reason for hiding this comment

jerboaa commented May 27, 2021

tanghaoth90 commented May 27, 2021

openjdk bot commented May 27, 2021

jerboaa commented May 27, 2021

openjdk bot commented May 27, 2021

tanghaoth90 commented Apr 23, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented May 3, 2021 •

edited

Loading

argha-c May 4, 2021 •

edited by bridgekeeper bot

Loading

openjdk bot commented May 25, 2021 •

edited

Loading

tanghaoth90 commented May 27, 2021 •

edited

Loading