8252221: Use multiple workers for Parallel GC pre-touching

[amitdpawar]

Please review this change.

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed

Issue

• JDK-8252221: Use multiple workers for Parallel GC pre-touching

Reviewers

• Kim Barrett (@kimbarrett - Reviewer)
• Thomas Schatzl (@tschatzl - Reviewer)

Download

$ git fetch https://git.openjdk.java.net/jdk pull/180/head:pull/180
$ git checkout pull/180

[bridgekeeper]

Hi @amitdpawar, welcome to this OpenJDK project and thanks for contributing!

We do not recognize you as Contributor and need to ensure you have signed the Oracle Contributor Agreement (OCA). If you have not signed the OCA, please follow the instructions. Please fill in your GitHub username in the "Username" field of the application. Once you have signed the OCA, please let us know by writing /signed in a comment in this pull request.

If you already are an OpenJDK Author, Committer or Reviewer, please click here to open a new issue so that we can record that fact. Please use "Add GitHub user amitdpawar" as summary for the issue.

If you are contributing this work on behalf of your employer and your employer has signed the OCA, please let us know by writing /covered in a comment in this pull request.

[openjdk]

@amitdpawar The following label will be automatically applied to this pull request: hotspot-gc.

When this pull request is ready to be reviewed, an RFR email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label (add|remove) "label" command.

[amitdpawar]

/signed

[bridgekeeper]

Thank you! Please allow for up to two weeks to process your OCA, although it is usually done within one to two business days. Also, please note that pull requests that are pending an OCA check will not usually be evaluated, so your patience is appreciated!

[mlbridge]

Webrevs

• 04: Full - Incremental (82fbbe4)
• 03: Full - Incremental (e6c072c)
• 02: Full - Incremental (d81c3b9)
• 01: Full - Incremental (232adc9)
• 00: Full (f4d0cef)

[mlbridge]

Mailing list message from Thomas Schatzl on hotspot-gc-dev:

Hi Amit,

thanks for your contribution.

On 16.09.20 18:18, Amit Pawar wrote:

Please review this change.

^^--- please consider putting a few lines of explanation in there like
in a regular (old-style) RFR email. People may respond more eagerly if
they do not have to search around what you want to accomplish.

Thanks.

- I am not sure why you want some callers of MutableSpace::initialize()
not use the workgang for parallel pretouch. Particularly after resizing
the old gen it might be very good to use it.

Just drop all optional parameters to intialize then, as I would prefer
to have the Workgang as the last parameter.

- mutableSpace.cpp:166

if (pretouch_gang) {

coding guidelines prohibit relying on implicit conversion of integers to
booleans. Please add != NULL. See below too.

- mutableSpace.cpp:170/184:

the optimization to not call pretouching code just complicates the code:
you do not save anything interesting as pretouching takes much longer
than that check.

- I am not completely clear why the change only parallelizes the "head"
portion of the MutableSpace in the initialize() method.

- afaict the current pretouch code does not pretouch the tail of the
requested memory range if it is not evenly divisible by #threads and
page size. Is this intentional?

- there are some other issues in the new gangtask code, but instead of
rolling your own pretouch task in this change, please move the code in
G1PageBasedVirtualSpace::pretouch() (changing parameters to use
start_addr/end_addr and a page_size parameter in addition to the
workgang) to a new class with that method as public static in the
gc/shared directory and use it in both places.

Remove the "G1" prefixes in the shared code of course.

That pretouch gang task code one has been tested and if changes are
required, they do not need to be done in multiple places.

Thanks,
Thomas

[amitdpawar]

Hi Thomas,

Thanks for your suggestions and please see my inline reply

Mailing list message from Thomas Schatzl on hotspot-gc-dev:

Hi Amit,

thanks for your contribution.

On 16.09.20 18:18, Amit Pawar wrote:

Please review this change.

^^--- please consider putting a few lines of explanation in there like
in a regular (old-style) RFR email. People may respond more eagerly if
they do not have to search around what you want to accomplish.

OK.

Thanks.

• I am not sure why you want some callers of MutableSpace::initialize()
  not use the workgang for parallel pretouch. Particularly after resizing
  the old gen it might be very good to use it.

It does not work correctly for old gen after resizing. One of the tier1 test is running for a long time and not sure why ? Initially thought to handle the startup case first and then resizing next based on feedback. I will come back with more info on this.

Just drop all optional parameters to intialize then, as I would prefer
to have the Workgang as the last parameter.

• mutableSpace.cpp:166

if (pretouch_gang) {

coding guidelines prohibit relying on implicit conversion of integers to
booleans. Please add != NULL. See below too.

• mutableSpace.cpp:170/184:

OK.

the optimization to not call pretouching code just complicates the code:
you do not save anything interesting as pretouching takes much longer
than that check.

• I am not completely clear why the change only parallelizes the "head"
  portion of the MutableSpace in the initialize() method.

• afaict the current pretouch code does not pretouch the tail of the
  requested memory range if it is not evenly divisible by #threads and
  page size. Is this intentional?

Currently with single thread, pretouch is done separately for both "head" and "tail" part. I didnt get this why ? so thought of following the same and then can be changed based on feedback. I guess both "head" and "tail" can be merged. Please suggest.

• there are some other issues in the new gangtask code, but instead of
  rolling your own pretouch task in this change, please move the code in
  G1PageBasedVirtualSpace::pretouch() (changing parameters to use
  start_addr/end_addr and a page_size parameter in addition to the
  workgang) to a new class with that method as public static in the
  gc/shared directory and use it in both places.

Remove the "G1" prefixes in the shared code of course.

That pretouch gang task code one has been tested and if changes are
required, they do not need to be done in multiple places.

Will do the change as suggested.

Thanks,
Thomas

Thanks,
Amit

[mlbridge]

Mailing list message from Thomas Schatzl on hotspot-gc-dev:

Hi Amit,

On 18.09.20 18:59, Amit Pawar wrote:

On Tue, 15 Sep 2020 13:39:14 GMT, Amit Pawar <github.com+71302734+amitdpawar at openjdk.org> wrote:

Please review this change.

Hi Thomas,

Thanks for your suggestions and please see my inline reply

[...]

- I am not sure why you want some callers of MutableSpace::initialize()
not use the workgang for parallel pretouch. Particularly after resizing
the old gen it might be very good to use it.

It does not work correctly for old gen after resizing. One of the tier1 test is running for a long time and not sure
why ? Initially thought to handle the startup case first and then resizing next based on feedback. I will come back
with more info on this.

Okay.

Just drop all optional parameters to intialize then, as I would prefer
to have the Workgang as the last parameter.

- mutableSpace.cpp:166

if (pretouch_gang) {

coding guidelines prohibit relying on implicit conversion of integers to
booleans. Please add != NULL. See below too.

- mutableSpace.cpp:170/184:

OK.

- I am not completely clear why the change only parallelizes the "head"
portion of the MutableSpace in the initialize() method.

- afaict the current pretouch code does not pretouch the tail of the
requested memory range if it is not evenly divisible by #threads and
page size. Is this intentional?

Currently with single thread, pretouch is done separately for both "head" and "tail" part. I didnt get this why ? so
thought of following the same and then can be changed based on feedback. I guess both "head" and "tail" can be merged.
Please suggest.

The code tries to avoid virtual memory operation during resizes that
occur at both ends of the existing committed memory area. Head contains
work "to the left" of the current committed memory area, tail to the right.

Most often if not all the time one of these areas will be empty (idk
when there would be resizes at both ends), but let's keep it. Both areas
can benefit from the pre-touch improvement though.

- there are some other issues in the new gangtask code, but instead of
rolling your own pretouch task in this change, please move the code in
G1PageBasedVirtualSpace::pretouch() (changing parameters to use
start_addr/end_addr and a page_size parameter in addition to the
workgang) to a new class with that method as public static in the
gc/shared directory and use it in both places.

Remove the "G1" prefixes in the shared code of course.

That pretouch gang task code one has been tested and if changes are
required, they do not need to be done in multiple places.

Will do the change as suggested.

Thanks,
Thomas

[amitdpawar]

Hi Thomas,

Thanks for replying again and will do as per your suggestion.

Regarding using multiple threads to pre-touch old gen post resize.
Initial debugging showed that GC threads do reach here to initialize old gen memory post resizing. This limits to single threaded only as GC thread is already executing a task and needs special handling. Please see generated log messages below that included thread name.

2.116s][debug][gc ] GC(9) Expanding ParOldGen thread name VM Thread from 5632K by 3584K to 9216K
2.249s][debug][gc ] GC(12) Expanding ParOldGen thread name VM Thread from 9216K by 6144K to 15360K
5.768s][debug][gc ] GC(33) Expanding ParOldGen thread name GC Thread#139 from 15360K by 512K to 15872K
5.769s][debug][gc ] GC(33) Expanding ParOldGen thread name GC Thread#92 from 15872K by 512K to 16384K
5.769s][debug][gc ] GC(33) Expanding ParOldGen thread name GC Thread#13 from 16384K by 512K to 16896K

I would like to fix this in two separate patches.

1st Patch: Current PR fixes JVM startup time only. Pre-touching post resize do it multi-threaded with VM thread and single threaded with non VM threads. This will be a partial fix but not complete. Complete fix can be done in second patch.

2nd Patch: Please check log messages from 33th GC. Old gen size was changed three times in the same GC by three different GC threads. This makes separate Pre-touching needs to be done post every resize. In the second patch this can be done once by combining after all the resizes. I am interested in fixing this but need more time.

Please suggest.

Thanks,
Amit

[mlbridge]

Mailing list message from Thomas Schatzl on hotspot-gc-dev:

Hi,

On 25.09.20 14:27, Amit Pawar wrote:

On Fri, 18 Sep 2020 16:56:42 GMT, Amit Pawar <github.com+71302734+amitdpawar at openjdk.org> wrote:

Hi Thomas,

Thanks for replying again and will do as per your suggestion.

Regarding using multiple threads to pre-touch old gen post resize.
Initial debugging showed that GC threads do reach here to initialize old gen memory post resizing. This limits to
single threaded only as GC thread is already executing a task and needs special handling. Please see generated log
messages below that included thread name.

2.116s][debug][gc ] GC(9) Expanding ParOldGen thread name VM Thread from 5632K by 3584K to 9216K
2.249s][debug][gc ] GC(12) Expanding ParOldGen thread name VM Thread from 9216K by 6144K to 15360K
5.768s][debug][gc ] GC(33) Expanding ParOldGen thread name GC Thread#139 from 15360K by 512K to 15872K
5.769s][debug][gc ] GC(33) Expanding ParOldGen thread name GC Thread#92 from 15872K by 512K to 16384K
5.769s][debug][gc ] GC(33) Expanding ParOldGen thread name GC Thread#13 from 16384K by 512K to 16896K

So basically the threads themselves do expansion themselves during gc if
they are out of memory. Makes sense.

I would like to fix this in two separate patches.

1st Patch: Current PR fixes JVM startup time only. Pre-touching post resize do it multi-threaded with VM thread and
single threaded with non VM threads. This will be a partial fix but not complete. Complete fix can be done in second
patch.

2nd Patch: Please check log messages from 33th GC. Old gen size was changed three times in the same GC by three
different GC threads. This makes separate Pre-touching needs to be done post every resize. In the second patch this can
be done once by combining after all the resizes. I am interested in fixing this but need more time.

Please suggest.

This looks like expansion of old gen during promotion. I did not think
of that when replying earlier.

I do not think this can be delayed as the thread is simply out of
uncommitted memory and needs it right now to continue evacuation.

Let's at least split this out into a separate change then as you
suggested. If you can find something to improve here do it in another
change.

Thanks,
Thomas

[amitdpawar]

Thanks for agreeing and will update the PR with suggested changes for final review.

Thanks,
Amit

[amitdpawar]

Hi Thomas,

PR is updated with suggested changes. Please review.

Thanks,
Amit

[mlbridge]

Mailing list message from Thomas Schatzl on hotspot-gc-dev:

Hi Amit,

On 06.10.20 20:14, Amit Pawar wrote:

On Mon, 28 Sep 2020 04:11:16 GMT, Amit Pawar <github.com+71302734+amitdpawar at openjdk.org> wrote:

Thanks for agreeing and will update the PR with suggested changes for final review.

Thanks,
Amit

Hi Thomas,

PR is updated with suggested changes. Please review.

Thanks,
Amit

looks really good now, some minor nits:

- g1PageBasedVirtualSpace.cpp/mutableSpace.cpp: please keep the include
declarations sorted

- mutableSpace.cpp:118+

PretouchTask::pretouch("ParallelGC PreTouch", (char*)head.start(),
(char*)head.end(), page_size, pretouch_gang);

PretouchTask::pretouch("ParallelGC PreTouch", (char*)tail.start(),
(char*)tail.end(), page_size, pretouch_gang);

- Please move the end_address parameter to the previous line even if
it crosses 80 chars.
- the two tasks could be named differently, i.e. "ParallelGC PreTouch
head" and "... tail"

- pretouch.hpp: s/2018/2020 in the header; maybe update the end dates of
the other files too.

I'll let it pass through testing over the weekend. I will report back if
there is anything problematic.

Thanks,
Thomas

[mlbridge]

Mailing list message from Thomas Schatzl on hotspot-gc-dev:

Hi again,

some more nits:

- the code does not compiler without precompiled headers because of
undeclared WorkGang.

hotspot/share/gc/parallel/mutableSpace.hpp:90:27: error: 'WorkGang' has
not been declared
[2020-10-09T11:44:35,851Z] 90 | WorkGang*
pretouch_gang = NULL);
[2020-10-09T11:44:35,851Z] | ^~~~~~~~

Please fix and check by building with --disable-precompiled_headers
configure parameters.

Same in mutableNUMASpace.hpp.

The cpp files also need the workgroup.hpp include.

- please rename pretouch.hpp to pretouchTask.hpp.

- includes are completely missing in pretouch.hpp. Needs at least
workgroup.hpp, globals.hpp (for the UseTransparentHugePages global),
atomic.hpp, os.hpp

- please move the implementation of the various methods of PretouchTask
into a .cpp file. Try to avoid includes in the .hpp file.

Thanks,
Thomas

On 09.10.20 12:53, Thomas Schatzl wrote:

Hi Amit,

On 06.10.20 20:14, Amit Pawar wrote:

On Mon, 28 Sep 2020 04:11:16 GMT, Amit Pawar
<github.com+71302734+amitdpawar at openjdk.org> wrote:

Thanks for agreeing and will update the PR with suggested changes for
final review.

Thanks,
Amit

Hi Thomas,

PR is updated with suggested changes. Please review.

Thanks,
Amit

? looks really good now, some minor nits:

- g1PageBasedVirtualSpace.cpp/mutableSpace.cpp: please keep the include
declarations sorted

- mutableSpace.cpp:118+

?PretouchTask::pretouch("ParallelGC PreTouch", (char*)head.start(),
??????????????????????? (char*)head.end(), page_size, pretouch_gang);

?PretouchTask::pretouch("ParallelGC PreTouch", (char*)tail.start(),
??????????????????????? (char*)tail.end(), page_size, pretouch_gang);

?- Please move the end_address parameter to the previous line even if
it crosses 80 chars.
?- the two tasks could be named differently, i.e. "ParallelGC PreTouch
head" and "... tail"

- pretouch.hpp: s/2018/2020 in the header; maybe update the end dates of
the other files too.

I'll let it pass through testing over the weekend. I will report back if
there is anything problematic.

Thanks,
? Thomas

[amitdpawar]

Hi Thomas,

I have fixed the build issue and also updated the PR with suggested changes. Thanks for testing and please check now.

Thanks,
Amit

[mlbridge]

Mailing list message from Thomas Schatzl on hotspot-gc-dev:

Hi Amit,

On 09.10.20 19:08, Amit Pawar wrote:

On Tue, 6 Oct 2020 18:11:52 GMT, Amit Pawar <github.com+71302734+amitdpawar at openjdk.org> wrote:

Thanks for agreeing and will update the PR with suggested changes for final review.

Thanks,
Amit

Hi Thomas,

PR is updated with suggested changes. Please review.

Thanks,
Amit

Hi Thomas,

I have fixed the build issue and also updated the PR with suggested changes. Thanks for testing and please check now.

Thanks,
Amit

thanks for your changes. Some more, final minor nits:

- mutableSpace.cpp:115: please remove the newline after the "if"

- psYoungGen.cpp:641: I do not think PSYoungGen::resize_spaces() can be
called by anything but the VMThread, so code to distinguish there seems
superfluous.

I.e. ultimately by either PSParallelCompact::invoke_no_policy() or
PSScavenge::invoke_no_policy.

Same with the code in PSYoungGen::reset_survivors_after_shrink().

(I did not actually test this change).

- preTouchTask.hpp: please move all implementation to the cpp file, then
you can also move all includes to that file. (Do not forget the forward
declaration of WorkGang).

- preTouchTask.hpp: please drop the first "private:" in the PretouchTask
class.

- preTouchTask.hpp: please move the "size_t page_size" for the static
pretouch method to the next line.

- preTouchTask.hpp:56: please drop the extra spaces between function
declaration and definition of chunk_size().

- preTouchTask.cpp: please add "#include "precompiled.hpp"" as first
includes. Otherwise it will not build on Windows.

Other than that testing tiers 1-5 seems good so far (70% done) after
fixing the precompiled.hpp issue.

Thanks,
Thomas

[amitdpawar]

Hi Thomas,

Thanks for the your feedback and testing and good to hear that it passed 70%.

PR is updated as per your suggestions and please check.

I have another suggestion regarding "PreTouchParallelChunkSize" flag.
The default value looks huge and reducing this value to 128MB improved startup time for both G1GC and ParallelGC.
Tested same with 1TB memory and please see the results below.

ParallelGC = 0.5-1 sec.
G1GC = 2-3 sec

Also, the minimum value allowed is '1' and it should be similar to page size right ?

Please check.

Thanks,
Amit

[mlbridge]

Mailing list message from Thomas Schatzl on hotspot-gc-dev:

Hi,

On 13.10.20 15:53, Amit Pawar wrote:

On Fri, 9 Oct 2020 17:05:51 GMT, Amit Pawar <github.com+71302734+amitdpawar at openjdk.org> wrote:

Hi Thomas,

PR is updated with suggested changes. Please review.

Thanks,
Amit

Hi Thomas,

I have fixed the build issue and also updated the PR with suggested changes. Thanks for testing and please check now.

Thanks,
Amit

Hi Thomas,

Thanks for the your feedback and testing and good to hear that it passed 70%.

PR is updated as per your suggestions and please check.

I have another suggestion regarding "PreTouchParallelChunkSize" flag.
The default value looks huge and reducing this value to 128MB improved startup time for both G1GC and ParallelGC.
The following test are done with 1TB memory. Please see below.

ParallelGC = 0.5-1 sec.
G1GC = 2-3 sec

I filed https://bugs.openjdk.java.net/browse/JDK-8254699, although
I'm not sure what this means since the comparison is lacking.

Also, the minimum value allowed is '1' and it should be similar to page size right ?

It should be at least a page size. We can change the CR as required.

Please check.

Thanks,
Thomas

[amitdpawar]

Sorry for the confusion and here is how I tested

Command:
time ./jdk/bin/java -XX:+AlwaysPreTouch -XX:-UseParallelGC -Xmx900g -Xms900g -Xmn800g -XX:SurvivorRatio=400 -Xlog:gc*=debug:file=gc.log -XX:ParallelGCThreads=128 -XX:PreTouchParallelChunkSize=1g -version

G1GC Test
ChunkSize= 1GB : Real time taken = 17.68s
ChunkSize= 128MB : Real time ttaken = 14.8s

ParallelGC Test
ChunkSize= 1GB : Real time taken = 13.54s
ChunkSize= 128MB : Real time taken = 12.8s

I hope this helps.

Thanks,
Amit

[kimbarrett]

new files "preTouchTask.[ch]pp" should be "pretouchTask.[ch]pp"

[amitdpawar]

Thanks for the suggestions and PR is updated with suggested changes.

Thanks,
Amit

[tschatzl]

lgtm. Thanks. Please wait on Kim's approval and I'll do a final testing round - I think you need somebody sponsoring this change anyway. I can do that.

[openjdk]

@amitdpawar This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8252221: Use multiple workers for Parallel GC pre-touching

Reviewed-by: kbarrett, tschatzl

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 419 new commits pushed to the master branch:

• 7e5eb49: 8253402: Convert vmSymbols::SID to enum class
• 038f58d: 8226236: [TESTBUG] win32: gc/metaspace/TestCapacityUntilGCWrapAround.java fails
• 5194f11: 8254792: Disable intrinsic StringLatin1.indexOf until 8254790 is fixed
• 55d760d: 8254263: Remove special_runtime_exit_condition() check from ~ThreadInVMForHandshake()
• 03fa733: 8254777: Remove unimplemented Management::get_loaded_classes
• 8fb294a: 8254781: Remove unimplemented ClassFieldMap::compute_field_count
• da2f5ab: 8254780: EnterInterpOnlyModeClosure::completed() always returns true
• 0c99b19: 8223347: Integration of Vector API (Incubator)
• 386e7e8: 8254789: ProblemList compiler/graalunit/HotspotTest.java
• cd33abb: 8249623: test @Ignore-d due to 7013634 should be returned back to execution
• ... and 409 more: https://git.openjdk.java.net/jdk/compare/d219d8b9871be037815413e6f52bc85ebac8db2a...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@kimbarrett, @tschatzl) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

[tschatzl]

Yes. I also changed it so that you need to wait for a second reviewer (i.e. Kim to be satisfied) before you can do that. It would be a bit rude to not wait until Kim okay'ed it after he started the review.

/reviewers 2

[openjdk]

@tschatzl
The number of required reviews for this PR is now set to 2 (with at least 1 of role reviewers).

[kimbarrett]

Looks good.

[amitdpawar]

Thanks Kim and Thomas for approving the PR.

[amitdpawar]

/integrate

[openjdk]

@amitdpawar
Your change (at version 82fbbe4) is now ready to be sponsored by a Committer.

[tschatzl]

Another round of tier1-5 testing seems to be good.

/sponsor

[tschatzl]

Thanks @amitdpawar for your contribution! :)

[openjdk]

@tschatzl @amitdpawar Since your change was applied there have been 423 commits pushed to the master branch:

• f44fc6d: 8254734: "dead loop detected" assert failure with patch from 8223051
• 7f73474: 8254773: Remove unimplemented ciReplay::is_loaded(Klass* klass)
• 167c192: 8254771: Remove unimplemented ciSignature::get_all_klasses
• 81a8ff1: 8254769: Remove unimplemented BCEscapeAnalyzer::{add_dependence, propagate_dependencies}
• 7e5eb49: 8253402: Convert vmSymbols::SID to enum class
• 038f58d: 8226236: [TESTBUG] win32: gc/metaspace/TestCapacityUntilGCWrapAround.java fails
• 5194f11: 8254792: Disable intrinsic StringLatin1.indexOf until 8254790 is fixed
• 55d760d: 8254263: Remove special_runtime_exit_condition() check from ~ThreadInVMForHandshake()
• 03fa733: 8254777: Remove unimplemented Management::get_loaded_classes
• 8fb294a: 8254781: Remove unimplemented ClassFieldMap::compute_field_count
• ... and 413 more: https://git.openjdk.java.net/jdk/compare/d219d8b9871be037815413e6f52bc85ebac8db2a...master

Your commit was automatically rebased without conflicts.

Pushed as commit 9359ff0.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.