JDK-8268893: jcmd to trim the glibc heap #4510

tstuefe · 2021-06-16T12:57:44Z

Proposal to add a Linux+glibc-only jcmd to manually induce malloc_trim(3) in the VM process.

The glibc is somewhat notorious for retaining released C Heap memory: calling free(3) returns memory to the glibc, and most libc variants will return at least a portion of it back to the Operating System, but the glibc often does not.

This depends on the granularity of the allocations and a number of other factors, but we found that many small allocations in particular may cause the process heap segment (hence RSS) to get bloaty. This can cause the VM to not recover from C-heap usage spikes.

The glibc offers an API, "malloc_trim", which can be used to cause the glibc to return free'd memory back to the Operating System.

This may cost performance, however, and therefore I hesitate to call malloc_trim automatically. That may be an idea for another day.

Instead of an automatic trim I propose to add a jcmd which allows to manually trigger a libc heap trim. Such a command would have two purposes:

when analyzing cases of high memory footprint, it allows to distinguish "real" footprint, e.g. leaks, from a cases where the glibc just holds on to memory
as a stop gap measure it allows to release pressure from a high footprint scenario.

Note that this command also helps with analyzing libc peaks which had nothing to do with the VM - e.g. peaks created by customer code which just happens to share the same process as the VM. Such memory does not even have to show up in NMT.

I propose to introduce this command for Linux only. Other OSes (apart maybe AIX) do not seem to have this problem, but Linux is arguably important enough in itself to justify a Linux specific jcmd.

CSR for this command: https://bugs.openjdk.java.net/browse/JDK-8269345

Note that an alternative to a Linux-only jcmd would be a command which would trim the C-heap on all platforms, with implementations to be filled out later.

=========

This patch:

introduces a new jcmd, "VM.trim_libc_heap", no arguments, which trims the glibc heap on glibc platforms.
includes a (rather basic) test
the command calls malloc_trim(3), and additionally prints out its effect (changes caused in virt size, rss and swap space)
I refactored some code in os_linux.cpp to factor out scanning /proc/self/status to get kernel memory information.

=========

Example:

A programm causes a temporary peak in C-heap usage (in this case, triggered via Unsafe.allocateMemory), right away frees the memory again, so its not leaky. The peak in RSS was ~8G (even though the user allocation was way smaller - glibc has a lot of overhead). The effects of this peak linger even after returning that memory to the glibc:

thomas@starfish:~$ jjjcmd AllocCHeap VM.info | grep Resident
Resident Set Size: 8685896K (peak: 8685896K) (anon: 8648680K, file: 37216K, shmem: 0K)
                   ^^^^^^^^

We execute the new trim command via jcmd:

thomas@starfish:~$ jjjcmd AllocCHeap VM.trim_libc_heap
18770:
Attempting trim...
Done.
Virtual size before: 28849744k, after: 28849724k, (-20k)
RSS before: 8685896k, after: 920740k, (-7765156k)  <<<<
Swap before: 0k, after: 0k, (0k)

It prints out reduction in virtual size, rss and swap. The virtual size did not decrease since no mappings had been unmapped by the glibc. However, the process heap was shrunk heavily by the glibc, resulting in a large drop in RSS (8.5G->900M), freeing >7G of memory:

thomas@starfish:~$ jjjcmd AllocCHeap VM.info | grep Resident
Resident Set Size: 920740K (peak: 8686004K) (anon: 883460K, file: 37280K, shmem: 0K)
                   ^^^^^^^

When the VM is started with -Xlog:os, this is also logged:

[139,068s][info][os] malloc_trim:
[139,068s][info][os] Virtual size before: 28849744k, after: 28849724k, (-20k)
RSS before: 8685896k, after: 920740k, (-7765156k)
Swap before: 0k, after: 0k, (0k)

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8268893: jcmd to trim the glibc heap

Reviewers

Volker Simonis (@simonis - Reviewer) ⚠️ Review applies to 29807b2
David Holmes (@dholmes-ora - Reviewer) ⚠️ Review applies to c3f055a

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/4510/head:pull/4510
$ git checkout pull/4510

Update a local copy of the PR:
$ git checkout pull/4510
$ git pull https://git.openjdk.java.net/jdk pull/4510/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 4510

View PR using the GUI difftool:
$ git pr show -t 4510

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/4510.diff

bridgekeeper · 2021-06-16T12:58:49Z

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-06-16T13:00:23Z

@tstuefe The following labels will be automatically applied to this pull request:

hotspot-runtime
serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-06-17T06:41:32Z

Webrevs

src/hotspot/os/linux/os_linux.hpp

mlbridge · 2021-06-21T06:23:59Z

Mailing list message from David Holmes on hotspot-runtime-dev:

Hi Thomas,

On 17/06/2021 4:41 pm, Thomas Stuefe wrote:

The glibc is somewhat notorious for retaining released C Heap memory: calling free(3) returns memory to the glibc, and most libc variants will return at least a portion of it back to the Operating System, but the glibc often does not.

This depends on the granularity of the allocations and a number of other factors, but we found that many small allocations in particular may cause the process heap segment (hence RSS) to get bloaty. This can cause the VM to not recover from C-heap usage spikes.

The glibc offers an API, "malloc_trim", which can be used to cause the glibc to return free'd memory back to the Operating System.

This may cost performance, however, and therefore I hesitate to call malloc_trim automatically. That may be an idea for another day.

Instead of an automatic trim I propose to add a jcmd which allows to manually trigger a libc heap trim. Such a command would have two purposes:
- when analyzing cases of high memory footprint, it allows to distinguish "real" footprint, e.g. leaks, from a cases where the glibc just holds on to memory
- as a stop gap measure it allows to release pressure from a high footprint scenario.

Note that this command also helps with analyzing libc peaks which had nothing to do with the VM - e.g. peaks created by customer code which just happens to share the same process as the VM. Such memory does not even have to show up in NMT.

I propose to introduce this command for Linux only. Other OSes (apart maybe AIX) do not seem to have this problem, but Linux is arguably important enough in itself to justify a Linux specific jcmd.

Is it perhaps worthwhile trying to generalize this to a jcmd to request
an attempt to release system resources and then each platform can do
whatever may be available to assist in that - including doing nothing,
or in this case trimming the glibc heap ?

Thanks,
David
-----

mlbridge · 2021-06-21T06:24:01Z

Mailing list message from David Holmes on hotspot-runtime-dev:

Hi Thomas,

On 17/06/2021 4:41 pm, Thomas Stuefe wrote:

The glibc is somewhat notorious for retaining released C Heap memory: calling free(3) returns memory to the glibc, and most libc variants will return at least a portion of it back to the Operating System, but the glibc often does not.

This depends on the granularity of the allocations and a number of other factors, but we found that many small allocations in particular may cause the process heap segment (hence RSS) to get bloaty. This can cause the VM to not recover from C-heap usage spikes.

The glibc offers an API, "malloc_trim", which can be used to cause the glibc to return free'd memory back to the Operating System.

This may cost performance, however, and therefore I hesitate to call malloc_trim automatically. That may be an idea for another day.

Instead of an automatic trim I propose to add a jcmd which allows to manually trigger a libc heap trim. Such a command would have two purposes:
- when analyzing cases of high memory footprint, it allows to distinguish "real" footprint, e.g. leaks, from a cases where the glibc just holds on to memory
- as a stop gap measure it allows to release pressure from a high footprint scenario.

Note that this command also helps with analyzing libc peaks which had nothing to do with the VM - e.g. peaks created by customer code which just happens to share the same process as the VM. Such memory does not even have to show up in NMT.

I propose to introduce this command for Linux only. Other OSes (apart maybe AIX) do not seem to have this problem, but Linux is arguably important enough in itself to justify a Linux specific jcmd.

Is it perhaps worthwhile trying to generalize this to a jcmd to request
an attempt to release system resources and then each platform can do
whatever may be available to assist in that - including doing nothing,
or in this case trimming the glibc heap ?

Thanks,
David
-----

tstuefe · 2021-06-21T07:07:31Z

Hi David,

thanks for looking into this!

Is it perhaps worthwhile trying to generalize this to a jcmd to request
an attempt to release system resources and then each platform can do
whatever may be available to assist in that - including doing nothing,
or in this case trimming the glibc heap ?

I thought about this too. It would certainly make the jcmd look nicer.

But I think that this would be one of the cases where the supporter needs to know the details of what the command does under the hood, so by hiding the complexity behind the platform nothing is gained; you need to know what happens to gauge the influence it has on the system (eg performance loss by dropping caches) and to analyze the precise after-effects. I

I am undecided on this. I will ask around offline.

..Thomas

simonis

I like this little new diagnostic command which I think can be quite useful in specific situations.

However, in contrast to other reviewers, I'd rather keep this simple and Glibc specific instead of extending it to a more general but mostly empty command.

I'd therefore propose to rename this command to glibc_trim_heap to make it evident from the command name already that it is Glibc-specific.

Besides that, just cosmetic changes and suggestions.

src/hotspot/os/linux/os_linux.cpp

src/hotspot/os/linux/trimCHeapDCmd.cpp

src/hotspot/share/services/diagnosticCommand.cpp

src/hotspot/os/linux/trimCHeapDCmd.cpp

test/hotspot/jtreg/serviceability/dcmd/vm/TrimLibcHeapTest.java

tstuefe · 2021-06-21T09:55:18Z

I like this little new diagnostic command which I think can be quite useful in specific situations.

Thanks a lot Volker!

However, in contrast to other reviewers, I'd rather keep this simple and Glibc specific instead of extending it to a more general but mostly empty command.

Yes, that was my thought too. Lets wait for @dholmes-ora to chime in, whether we can all agree on a glibc specific variant. I also preferred that one.

I'd therefore propose to rename this command to glibc_trim_heap to make it evident from the command name already that it is Glibc-specific.

Besides that, just cosmetic changes and suggestions.

All of which make sense, I'll work them in.

Thanks!

..Thomas

mlbridge · 2021-06-21T12:14:00Z

Mailing list message from David Holmes on hotspot-runtime-dev:

On 21/06/2021 7:58 pm, Thomas Stuefe wrote:

On Mon, 21 Jun 2021 09:15:05 GMT, Volker Simonis <simonis at openjdk.org> wrote:

I like this little new diagnostic command which I think can be quite useful in specific situations.

Thanks a lot Volker!

However, in contrast to other reviewers, I'd rather keep this simple and Glibc specific instead of extending it to a more general but mostly empty command.

Yes, that was my thought too. Lets wait for @dholmes-ora to chime in, whether we can all agree on a glibc specific variant. I also preferred that one.

I don't totally oppose the specialised variant, but it certainly isn't
my ideal solution. Not sure if we already have platform specific dcmds?
This will need a CSR request anyway.

Cheers,
David

mlbridge · 2021-06-21T12:14:02Z

Mailing list message from David Holmes on hotspot-runtime-dev:

On 21/06/2021 7:58 pm, Thomas Stuefe wrote:

On Mon, 21 Jun 2021 09:15:05 GMT, Volker Simonis <simonis at openjdk.org> wrote:

I like this little new diagnostic command which I think can be quite useful in specific situations.

Thanks a lot Volker!

However, in contrast to other reviewers, I'd rather keep this simple and Glibc specific instead of extending it to a more general but mostly empty command.

Yes, that was my thought too. Lets wait for @dholmes-ora to chime in, whether we can all agree on a glibc specific variant. I also preferred that one.

I don't totally oppose the specialised variant, but it certainly isn't
my ideal solution. Not sure if we already have platform specific dcmds?
This will need a CSR request anyway.

Cheers,
David

tstuefe · 2021-06-21T12:26:20Z

Mailing list message from David Holmes on hotspot-runtime-dev:

On 21/06/2021 7:58 pm, Thomas Stuefe wrote:

On Mon, 21 Jun 2021 09:15:05 GMT, Volker Simonis wrote:

I like this little new diagnostic command which I think can be quite useful in specific situations.

Thanks a lot Volker!

However, in contrast to other reviewers, I'd rather keep this simple and Glibc specific instead of extending it to a more general but mostly empty command.

Yes, that was my thought too. Lets wait for @dholmes-ora to chime in, whether we can all agree on a glibc specific variant. I also preferred that one.

I don't totally oppose the specialised variant, but it certainly isn't
my ideal solution. Not sure if we already have platform specific dcmds?
This will need a CSR request anyway.

Cheers,
David

I'll prepare the CSR and bring it to review.

Cheers, Thomas

simonis · 2021-06-21T15:23:59Z

Mailing list message from David Holmes on hotspot-runtime-dev:

On 21/06/2021 7:58 pm, Thomas Stuefe wrote:

On Mon, 21 Jun 2021 09:15:05 GMT, Volker Simonis wrote:

I like this little new diagnostic command which I think can be quite useful in specific situations.

Thanks a lot Volker!

However, in contrast to other reviewers, I'd rather keep this simple and Glibc specific instead of extending it to a more general but mostly empty command.

Yes, that was my thought too. Lets wait for @dholmes-ora to chime in, whether we can all agree on a glibc specific variant. I also preferred that one.

I don't totally oppose the specialised variant, but it certainly isn't
my ideal solution. Not sure if we already have platform specific dcmds?

Yes, we have. E.g.:

#ifdef LINUX
class PerfMapDCmd : public DCmd {
public:
  PerfMapDCmd(outputStream* output, bool heap) : DCmd(output, heap) {}
  static const char* name() {
    return "Compiler.perfmap";
  }
  static const char* description() {
    return "Write map file for Linux perf tool.";
  }
...

This will need a CSR request anyway.

Cheers,
David

simonis · 2021-06-22T07:36:44Z

src/hotspot/os/linux/trimCHeapDCmd.cpp

+  }
+  _output->print_raw(ss_report.base());
+  log_info(os)("malloc_trim: ");
+  log_info(os)("%s", ss_report.base());


Might also make sense to put this into a single log line like log_info(os)("malloc_trim:\n%s", ss_report.base()); to get something like:

[139,068s][info][os] malloc_trim: Virtual size before: 28849744k, after: 28849724k, (-20k) RSS before: 8685896k, after: 920740k, (-7765156k) Swap before: 0k, after: 0k, (0k)

instead of:

[139,068s][info][os] malloc_trim: [139,068s][info][os] Virtual size before: 28849744k, after: 28849724k, (-20k) RSS before: 8685896k, after: 920740k, (-7765156k) Swap before: 0k, after: 0k, (0k)

tstuefe · 2021-06-25T05:33:33Z

Hi,

I created the CSR for this command: https://bugs.openjdk.java.net/browse/JDK-8269345

When you have time, please take a look. Thank you.

..Thomas

simonis

Thanks for the cleanups. Looks good to me now.

openjdk · 2021-06-25T15:36:16Z

@tstuefe This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8268893: jcmd to trim the glibc heap

Reviewed-by: simonis, dholmes

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 92 new commits pushed to the master branch:

c36755d: Merge
89f7998: 8266347: assert(Dependencies::is_concrete_root_method(fm, ctxk) == Dependencies::is_concrete_method(m, ctxk)) failed: mismatch
e1051ae: 8264066: Enhance compiler validation
add995b: 8265201: JarFile.getInputStream not validating invalid signed jars
ca6b222: 8258432: Improve File Transfers
a3acce5: 8264079: Improve abstractions
138f598: 8262380: Enhance XML processing passes
0e6c011: 8262967: Improve Zip file support
e48af00: 8264460: Improve NTLM support
b87e526: 8256491: Better HTTP transport
... and 82 more: https://git.openjdk.java.net/jdk/compare/4a7ccf36e9a3978c437db3efe892dd23e8a0b772...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

jerboaa · 2021-06-25T17:16:45Z

/csr

openjdk · 2021-06-25T17:17:24Z

@jerboaa this pull request will not be integrated until the CSR request JDK-8269345 for issue JDK-8268893 has been approved.

tstuefe · 2021-07-09T04:53:36Z

I renamed the command as agreed upon in the CSR discussion.

tstuefe · 2021-07-14T04:35:17Z

Gentle ping. Need a second review. @dholmes-ora ?

dholmes-ora

Hi Thomas,

This seems okay to me.

Thanks,
David

tstuefe · 2021-07-14T09:43:26Z

Hi Thomas,

This seems okay to me.

Thanks,
David

Thank you David and Volker!

tstuefe · 2021-07-14T12:10:27Z

/integrate

openjdk · 2021-07-14T12:11:31Z

@tstuefe This PR has not yet been marked as ready for integration.

jerboaa · 2021-07-14T13:49:52Z

The CSR https://bugs.openjdk.java.net/browse/JDK-8269345 is in Provisional. Once it's approved it should become ready for integration.

tstuefe · 2021-07-14T15:52:14Z

The CSR https://bugs.openjdk.java.net/browse/JDK-8269345 is in Provisional. Once it's approved it should become ready for integration.

Oh, you are right. Good that skara caught that.

tstuefe · 2021-07-21T05:15:29Z

Not sure what to do here honestly.

The CSR seems stuck in "provisional" without any indication if any action is required from my side. This PR has gathered enough reviews meanwhile, but I cannot integrate it without the CSR moving forward.

Someone from Oracle, please advise?

simonis · 2021-07-21T18:14:53Z

You have to move the CSR to "Proposed" or "Finalized" otherwise it probably won't appear on Joe's work list.

I've moved it to "Finalized" now because I think all the issues have been resolved. Please feel free to change the state again if you think that's not appropriate.

tstuefe · 2021-07-21T18:38:10Z

You have to move the CSR to "Proposed" or "Finalized" otherwise it probably won't appear on Joe's work list.

I've moved it to "Finalized" now because I think all the issues have been resolved. Please feel free to change the state again if you think that's not appropriate.

Thanks a lot Volker!

tstuefe · 2021-07-22T04:06:18Z

/integrate

openjdk · 2021-07-22T04:07:20Z

Going to push as commit 6096dd9.
Since your change was applied there have been 92 commits pushed to the master branch:

c36755d: Merge
89f7998: 8266347: assert(Dependencies::is_concrete_root_method(fm, ctxk) == Dependencies::is_concrete_method(m, ctxk)) failed: mismatch
e1051ae: 8264066: Enhance compiler validation
add995b: 8265201: JarFile.getInputStream not validating invalid signed jars
ca6b222: 8258432: Improve File Transfers
a3acce5: 8264079: Improve abstractions
138f598: 8262380: Enhance XML processing passes
0e6c011: 8262967: Improve Zip file support
e48af00: 8264460: Improve NTLM support
b87e526: 8256491: Better HTTP transport
... and 82 more: https://git.openjdk.java.net/jdk/compare/4a7ccf36e9a3978c437db3efe892dd23e8a0b772...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-07-22T04:07:35Z

@tstuefe Pushed as commit 6096dd9.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

openjdk bot added serviceability serviceability-dev@openjdk.org hotspot-runtime hotspot-runtime-dev@openjdk.org labels Jun 16, 2021

start

97d71ea

tstuefe force-pushed the jcmd-for-malloc-trim branch from 815e076 to 97d71ea Compare June 16, 2021 13:51

tstuefe changed the title ~~JDK-8268893: jcmd command to trim the glibc heap~~ JDK-8268893: jcmd to trim the glibc heap Jun 17, 2021

tstuefe marked this pull request as ready for review June 17, 2021 06:36

openjdk bot added the rfr Pull request is ready for review label Jun 17, 2021

jerboaa reviewed Jun 17, 2021

View reviewed changes

src/hotspot/os/linux/os_linux.hpp Outdated Show resolved Hide resolved

Feedback Severin; renamed query function

a48791e

simonis suggested changes Jun 21, 2021

View reviewed changes

simonis reviewed Jun 22, 2021

View reviewed changes

tstuefe added 2 commits June 25, 2021 07:47

Merge

9a9f0ec

Volker feedback

29807b2

simonis approved these changes Jun 25, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Jun 25, 2021

openjdk bot added the csr Pull request needs approved CSR before integration label Jun 25, 2021

openjdk bot removed the ready Pull request is ready to be integrated label Jun 25, 2021

tstuefe added 2 commits July 9, 2021 06:27

Merge

88c7bf1

Rename command (see CSR)

c3f055a

Merge

c9e7355

dholmes-ora approved these changes Jul 14, 2021

View reviewed changes

tstuefe mentioned this pull request Jul 19, 2021

JDK-8270308: Amalloc aligns size but not return value #4784

Closed

3 tasks

openjdk bot added ready Pull request is ready to be integrated and removed csr Pull request needs approved CSR before integration labels Jul 22, 2021

openjdk bot closed this Jul 22, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Jul 22, 2021

tstuefe deleted the jcmd-for-malloc-trim branch August 23, 2021 12:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JDK-8268893: jcmd to trim the glibc heap #4510

JDK-8268893: jcmd to trim the glibc heap #4510

tstuefe commented Jun 16, 2021 •

edited by openjdk bot

bridgekeeper bot commented Jun 16, 2021

openjdk bot commented Jun 16, 2021

mlbridge bot commented Jun 17, 2021 •

edited

mlbridge bot commented Jun 21, 2021

mlbridge bot commented Jun 21, 2021

tstuefe commented Jun 21, 2021

simonis left a comment

tstuefe commented Jun 21, 2021

mlbridge bot commented Jun 21, 2021

mlbridge bot commented Jun 21, 2021

tstuefe commented Jun 21, 2021

simonis commented Jun 21, 2021

simonis Jun 22, 2021

tstuefe commented Jun 25, 2021

simonis left a comment

openjdk bot commented Jun 25, 2021 •

edited

jerboaa commented Jun 25, 2021

openjdk bot commented Jun 25, 2021

tstuefe commented Jul 9, 2021

tstuefe commented Jul 14, 2021

dholmes-ora left a comment

tstuefe commented Jul 14, 2021

tstuefe commented Jul 14, 2021

openjdk bot commented Jul 14, 2021

jerboaa commented Jul 14, 2021

tstuefe commented Jul 14, 2021

tstuefe commented Jul 21, 2021

simonis commented Jul 21, 2021

tstuefe commented Jul 21, 2021

tstuefe commented Jul 22, 2021

openjdk bot commented Jul 22, 2021

openjdk bot commented Jul 22, 2021

JDK-8268893: jcmd to trim the glibc heap #4510

JDK-8268893: jcmd to trim the glibc heap #4510

Conversation

tstuefe commented Jun 16, 2021 • edited by openjdk bot

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Jun 16, 2021

openjdk bot commented Jun 16, 2021

mlbridge bot commented Jun 17, 2021 • edited

Webrevs

mlbridge bot commented Jun 21, 2021

mlbridge bot commented Jun 21, 2021

tstuefe commented Jun 21, 2021

simonis left a comment

Choose a reason for hiding this comment

tstuefe commented Jun 21, 2021

mlbridge bot commented Jun 21, 2021

mlbridge bot commented Jun 21, 2021

tstuefe commented Jun 21, 2021

simonis commented Jun 21, 2021

simonis Jun 22, 2021

Choose a reason for hiding this comment

tstuefe commented Jun 25, 2021

simonis left a comment

Choose a reason for hiding this comment

openjdk bot commented Jun 25, 2021 • edited

jerboaa commented Jun 25, 2021

openjdk bot commented Jun 25, 2021

tstuefe commented Jul 9, 2021

tstuefe commented Jul 14, 2021

dholmes-ora left a comment

Choose a reason for hiding this comment

tstuefe commented Jul 14, 2021

tstuefe commented Jul 14, 2021

openjdk bot commented Jul 14, 2021

jerboaa commented Jul 14, 2021

tstuefe commented Jul 14, 2021

tstuefe commented Jul 21, 2021

simonis commented Jul 21, 2021

tstuefe commented Jul 21, 2021

tstuefe commented Jul 22, 2021

openjdk bot commented Jul 22, 2021

openjdk bot commented Jul 22, 2021

tstuefe commented Jun 16, 2021 •

edited by openjdk bot

mlbridge bot commented Jun 17, 2021 •

edited

openjdk bot commented Jun 25, 2021 •

edited