JDK-8293313: NMT: Add "Fake OOM" mode to MallocLimit

[tstuefe]

• still work in progress *

This RFE adds the optional ability for -XX:MallocLimit to, instead of exiting the VM with a fatal error, fake a native OOM on this and all future malloc calls.

The user controls this by appending the option "oom" to the malloc limit argument:

-XX:MallocLimit=2g # will cause fatal error upon reaching limit
-XX:MallocLimit=2g,oom # will cause malloc to return NULL, mimicking a real malloc OOM

That makes it possible to test hotspot robustness in the face of real malloc OOMs, and is the prerequisite for removing the old MallocMaxTestWords machinery.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8293313: NMT: Add "Fake OOM" mode to MallocLimit

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk pull/10144/head:pull/10144
$ git checkout pull/10144

Update a local copy of the PR:
$ git checkout pull/10144
$ git pull https://git.openjdk.org/jdk pull/10144/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 10144

View PR using the GUI difftool:
$ git pr show -t 10144

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/10144.diff

[bridgekeeper]

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@tstuefe The following label will be automatically applied to this pull request:

• hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (24644027)

[bridgekeeper]

@tstuefe This pull request has been inactive for more than 8 weeks and will be automatically closed if another 8 weeks passes without any activity. To avoid this, simply add a new comment to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[tstuefe]

Not yet bot