8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_current_scale #11514

afshin-zafari · 2022-12-05T13:54:57Z

Description

MemReporterBase::diff_in_current_scale is defined as follows:

inline long diff_in_current_scale(size_t s1, size_t s2) const {
long amount = (long)(s1 - s2);
long scale = (long)_scale;
amount = (amount > 0) ? (amount + scale / 2) : (amount - scale / 2);
return amount / scale;
}

Long and size_t can have different sizes: 4 bytes and 8 bytes (LLP64). The result of 's1 - s2' might not fit into long. It might not fit into int64_t. For example: s1 is SIZE_MAX and s2 is SIZE_MAX-MAX_INT64-1.

Size_t should be used instead of long. Assertions must be added to check:
s1 >= s2 and (amount - scale/2) >= 0 and (amount + scale/2) <= SIZE_MAX.

Patch

long is replaced by size_t. Comparison to 0 is implemented accordingly since size_t is always >= 0.
Since s1 can be less than s2 in some invocations of this method, no assert is written for (s1 >= s2) case.

Test

local: runtime/NMT/Jcmd*
mach5: tier1

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_current_scale

Reviewers

Evgeny Astigeevich (@eastig - Committer)
Kim Barrett (@kimbarrett - Reviewer) ⚠️ Review applies to 91ba8d4c

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk pull/11514/head:pull/11514
$ git checkout pull/11514

Update a local copy of the PR:
$ git checkout pull/11514
$ git pull https://git.openjdk.org/jdk pull/11514/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 11514

View PR using the GUI difftool:
$ git pr show -t 11514

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/11514.diff

…urrent_scale

bridgekeeper · 2022-12-05T13:56:04Z

👋 Welcome back afshin-zafari! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2022-12-05T13:59:49Z

@afshin-zafari The following label will be automatically applied to this pull request:

hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2022-12-05T14:03:41Z

Webrevs

src/hotspot/share/services/memReporter.hpp

tstuefe · 2022-12-07T14:08:38Z

I would return ssize_t instead.

For values >SSIZE_MAX and <SSIZE_MIN I would assert in debug (because we should never see such high numbers) and cap in release builds.

And of course, the print format has to be adapted to use ssize_t format

tstuefe · 2022-12-07T14:27:54Z

I would return ssize_t instead.

For values >SSIZE_MAX and <SSIZE_MIN I would assert in debug (because we should never see such high numbers) and cap in release builds.

And of course, the print format has to be adapted to use ssize_t format

Correcting myself:

I would return int64_t (signed 64-bit).

On 32-bit platforms, where we could conceivably surpass SSIZE_MAX and SSIZE_MIN, that is large enough to hold positive and negative deltas.

On 64-bit, int64_t is the same as ssize_t. There, as I proposed, I would consider any delta > SSIZE_MIN or SSIZE_MAX to be an error. Because that indicates a negative overflow in a malloc counter.

I would actually consider any input value > 1000 TB an error as well, certainly any input > SSIZE_MAX.

afshin-zafari · 2022-12-08T09:38:30Z

int64_t is typedef'ed as long. To be portable to Windows, it should be as __int64.
What to do?

tstuefe · 2022-12-08T09:51:55Z

int64_t is typedef'ed as long. To be portable to Windows, it should be as __int64. What to do?

Its not. Hotspot defines its own variants. int/uint64 are always 64-bit, on all platforms.

jdk/src/hotspot/share/utilities/globalDefinitions_visCPP.hpp

Line 83 in 073897c

typedef int64_t ssize_t;

afshin-zafari · 2022-12-09T12:27:00Z

I would return ssize_t instead.
For values >SSIZE_MAX and <SSIZE_MIN I would assert in debug (because we should never see such high numbers) and cap in release builds.
And of course, the print format has to be adapted to use ssize_t format

Correcting myself:

I would return int64_t (signed 64-bit).

On 32-bit platforms, where we could conceivably surpass SSIZE_MAX and SSIZE_MIN, that is large enough to hold positive and negative deltas.

On 64-bit, int64_t is the same as ssize_t. There, as I proposed, I would consider any delta > SSIZE_MIN or SSIZE_MAX to be an error. Because that indicates a negative overflow in a malloc counter.

I would actually consider any input value > 1000 TB an error as well, certainly any input > SSIZE_MAX.

So, I will do these: the return value is of type int64_t; Then, there is no need to change the printing format for the returned value in places where this function is called. Correct?
What other changes need to be done?

tstuefe · 2022-12-09T13:30:27Z

So, I will do these: the return value is of type int64_t; Then, there is no need to change the printing format for the returned value in places where this function is called. Correct? What other changes need to be done?

See this PR: #11568 - solves the same issue as yours, but for the counters. The only difference to your patch is that with counters, I can use ssize_t, since I know that on 32-bit these counters cannot overflow SSIZE_MAX (2g, -2g). But with memory sizes this can happen (think: a 32-bit VM mallocing 2.1GB, then freeing it again), therefore we need to use int64_t.

You need to change the print formats too. Currently, the printout uses %+ld, which on Windows and on 32-bit platforms in general would print with 32-bit. You need to use INT64_FORMAT, but since the printout wants to have a leading '+' for positive numbers, you need something like INT64_PLUS_FORMAT.

INT64_PLUS_FORMAT does not exist yet, you need to add it. Just do this:

--- a/src/hotspot/share/utilities/globalDefinitions.hpp
+++ b/src/hotspot/share/utilities/globalDefinitions.hpp
@@ -120,6 +120,7 @@ class oopDesc;
 
 // Format 64-bit quantities.
 #define INT64_FORMAT             "%"          PRId64
+#define INT64_PLUS_FORMAT        "%+"         PRId64
 #define INT64_FORMAT_X           "0x%"        PRIx64
 #define INT64_FORMAT_X_0         "0x%016"     PRIx64
 #define INT64_FORMAT_W(width)    "%"   #width PRId64

Cheers, Thomas

…urrent_scale

afshin-zafari · 2022-12-14T13:06:47Z

Thanks Thomas. I implemented the changes as you guided.
Tests:
runtime/NMT/* , tier1-5 passed.

src/hotspot/share/services/memReporter.hpp

…urrent_scale

eastig · 2022-12-19T14:01:34Z

src/hotspot/share/services/memReporter.hpp

+    size_t amount = s1 - s2;
+    assert(amount <= SIZE_MAX - _scale / 2, "size_t overflow");
+    amount = (amount + _scale / 2) / _scale;
+    // We assume the valid range for deltas [INT64_MIN, INT64_MAX] to simplify the code.


In the assert the comment should be:

We assume the valid range for deltas [-INT64_MAX, INT64_MAX] to simplify the code.

I excluded the INT64_MIN case to simplify the code.
The INT64_MIN case is when sizeof(size_t) == sizeof(int64_t) && amount == INT64_MAX + 1 && is_negative == true.
For this case -(int64_t)amount is UB which needs proper handling.
The code instead of

return (is_negative) ? -(int64_t)amount : (int64_t)amount;

would be more complex:

if (!is_negative) { return (int64_t)amount; } else { return (amount != INT64_MAX+1) ? -(int64_t)amount : INT64_MIN; }

This is why the assert part is sizeof(size_t) == sizeof(int64_t) && amount <= INT64_MAX)

Maybe the better comment is

// We assume the valid range for deltas [-INT64_MAX, INT64_MAX]. // We excluded the `INT64_MIN` case to simplify the code.

I couldn't follow your idea.
The INT64_MAX + 1 results in compiler overflow error. Isn't it equal to INT64_MIN?

I did not know there was a defect report #1313 fixed in C++11.
Now the standard prohibits to use constant expressions like INT64_MAX + 1 in code.

I couldn't follow your idea.

The idea is to show that if you want to handle -(2^63), you need more code. Your comment is not correct. The code does not handle INT64_MIN.

If we limit cases to [-(2^63 - 1), 2^63 - 1] the code would be simpler. IMHO one byte does not worth to increase the code complexity. In most cases -(2^63) would be a result of some wrong calculations.

What is the replacement for INT64_MAX + 1 in our code?

Ok, if we want to support [INT64_MIN, INT64_MAX] here is the solution:

inline int64_t diff_in_current_scale(size_t s1, size_t s2) const { // _scale should not be 0, otherwise division by zero at return. assert(_scale != 0, "wrong scale"); bool is_negative = false; if (s1 < s2) { is_negative = true; swap(s1, s2); } size_t amount = s1 - s2; assert(amount <= SIZE_MAX - _scale / 2, "size_t overflow"); amount = (amount + _scale / 2) / _scale; // We assume the valid range for deltas [INT64_MIN, INT64_MAX]. assert(sizeof(size_t) <= sizeof(int64_t) && amount - (int)is_negative <= INT64_MAX), "cannot fit scaled diff into int64_t"); if (is_negative) { return (sizeof(size_t) == sizeof(int64_t) && amount - 1 == INT64_MAX) ? INT64_MIN : -(int64_t)amount; } else { return (int64_t)amount; } }

If sizeof(size_t) < sizeof(int64_t), a compiler will optimise (sizeof(size_t) == sizeof(int64_t) && amount - 1 == INT64_MAX) ? INT64_MIN : -(int64_t)amount into -(int64_t)amount.

If sizeof(size_t) == sizeof(int64_t), a compiler will optimise (sizeof(size_t) == sizeof(int64_t) && amount - 1 == INT64_MAX) ? INT64_MIN : -(int64_t)amount into (amount - 1 == INT64_MAX) ? INT64_MIN : -(int64_t)amount.

The C++ standard guarantees (int)is_negative to be either 0 or 1. So the check amount - (int)is_negative <= INT64_MAX will be amount <= INT64_MAX for positive diffs and amount - 1 <= INT64_MAX for negative diffs.

…urrent_scale

src/hotspot/share/services/memReporter.hpp

kimbarrett · 2022-12-24T06:35:50Z

src/hotspot/share/services/memReporter.hpp

+    assert(amount <= SIZE_MAX - _scale / 2, "size_t overflow");
+    amount = (amount + _scale / 2) / _scale;
+    // We assume the valid range for deltas [INT64_MIN, INT64_MAX] to simplify the code.
+    assert((sizeof(size_t) < sizeof(int64_t)) || (sizeof(size_t) == sizeof(int64_t) && amount <= INT64_MAX), "cannot fit scaled diff into size_t");


The sizeof comparisons are pointless or even wrong. The only thing we care
about here is that the scaled value <= INT64_MAX.
If sizeof(size_t) < sizeof(int64_t) then amount <= INT64_MAX is always true,
so there's no need to do the size comparison. Otherwise, we don't care if
size_t is bigger than int64_t, we only care about the value range for the
result. So just

assert(amount <= INT64_MAX, "overflow");

is sufficient.

kimbarrett · 2022-12-24T06:36:20Z

src/hotspot/share/services/memReporter.hpp

+    amount = (amount + _scale / 2) / _scale;
+    // We assume the valid range for deltas [INT64_MIN, INT64_MAX] to simplify the code.
+    assert((sizeof(size_t) < sizeof(int64_t)) || (sizeof(size_t) == sizeof(int64_t) && amount <= INT64_MAX), "cannot fit scaled diff into size_t");
+    return (is_negative) ? -(int64_t)amount : (int64_t)amount;


I'd prefer C++-style casts, so

int64_t result = static_cast<int64_t>(amount); return is_negative ? -result : result;

src/hotspot/share/services/memReporter.hpp

…urrent_scale

src/hotspot/share/services/memReporter.hpp

tstuefe

@afshin-zafari, thanks a lot for taking my suggestion about using in64_t. Some minor nits remain, mainly the diff function can be simplified.

Thanks, Thomas

src/hotspot/share/services/memReporter.hpp

…urrent_scale

src/hotspot/share/services/memReporter.hpp

eastig

LGTM, just minor suggestions.

src/hotspot/share/services/memReporter.hpp

…urrent_scale

kimbarrett

Just one tiny nit remaining. Otherwise, looks good.

src/hotspot/share/services/memReporter.hpp

openjdk · 2023-01-16T19:49:53Z

@afshin-zafari This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_current_scale

Reviewed-by: eastigeevich, kbarrett

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 334 new commits pushed to the master branch:

eca6479: 8300087: Replace NULL with nullptr in share/cds/
49d60fe: 8300172: java/net/httpclient/MappingResponseSubscriber.java failed with java.net.ConnectException
e189397: 8296403: [TESTBUG] IR test runner methods in TestLongRangeChecks.java invoke wrong test methods
77f2d20: 8287873: Add test for using -XX:+AutoCreateSharedArchive with different JDK versions
93a933d: 8300400: Update --release 20 symbol information for JDK 20 build 32
9b97699: 8300586: Refactor code examples to use @snippet in java.text.Collator
fbbb27e: 8300356: Refactor code examples to use @snippet in java.text.CollationElementIterator
dfcd65c: Merge
1c84050: 8298400: Virtual thread instability when stack overflows
62a2f23: 8300490: Spaces in name of MacOS Code Signing Identity are not correctly handled after JDK-8293550
... and 324 more: https://git.openjdk.org/jdk/compare/50120396b6cca1219fb5dd42a11e4b29b79bd3bd...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@dholmes-ora, @eastig, @kimbarrett, @tstuefe) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

…urrent_scale

afshin-zafari · 2023-01-20T09:26:46Z

/integrate

openjdk · 2023-01-20T09:28:32Z

@afshin-zafari
Your change (at version a1f9930) is now ready to be sponsored by a Committer.

eastig · 2023-01-20T10:14:47Z

/sponsor

openjdk · 2023-01-20T10:17:18Z

Going to push as commit 26410c1.
Since your change was applied there have been 334 commits pushed to the master branch:

eca6479: 8300087: Replace NULL with nullptr in share/cds/
49d60fe: 8300172: java/net/httpclient/MappingResponseSubscriber.java failed with java.net.ConnectException
e189397: 8296403: [TESTBUG] IR test runner methods in TestLongRangeChecks.java invoke wrong test methods
77f2d20: 8287873: Add test for using -XX:+AutoCreateSharedArchive with different JDK versions
93a933d: 8300400: Update --release 20 symbol information for JDK 20 build 32
9b97699: 8300586: Refactor code examples to use @snippet in java.text.Collator
fbbb27e: 8300356: Refactor code examples to use @snippet in java.text.CollationElementIterator
dfcd65c: Merge
1c84050: 8298400: Virtual thread instability when stack overflows
62a2f23: 8300490: Spaces in name of MacOS Code Signing Identity are not correctly handled after JDK-8293550
... and 324 more: https://git.openjdk.org/jdk/compare/50120396b6cca1219fb5dd42a11e4b29b79bd3bd...master

Your commit was automatically rebased without conflicts.

openjdk · 2023-01-20T10:17:41Z

@eastig @afshin-zafari Pushed as commit 26410c1.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

afshin-zafari added 2 commits December 2, 2022 13:52

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_c…

dae5acc

…urrent_scale

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_c…

89bf10d

…urrent_scale

openjdk bot added the rfr Pull request is ready for review label Dec 5, 2022

openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label Dec 5, 2022

dholmes-ora suggested changes Dec 6, 2022

View reviewed changes

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

eastig suggested changes Dec 6, 2022

View reviewed changes

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_c…

e147bee

…urrent_scale

eastig reviewed Dec 14, 2022

View reviewed changes

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

eastig reviewed Dec 14, 2022

View reviewed changes

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_c…

d952848

…urrent_scale

eastig reviewed Dec 19, 2022

View reviewed changes

afshin-zafari added 2 commits December 22, 2022 10:10

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_c…

5f552ed

…urrent_scale

Merge branch 'master' into _8281213

68d43c8

kimbarrett suggested changes Dec 24, 2022

View reviewed changes

kimbarrett reviewed Dec 24, 2022

View reviewed changes

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_c…

6053de5

…urrent_scale

kimbarrett reviewed Jan 11, 2023

View reviewed changes

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

eastig reviewed Jan 11, 2023

View reviewed changes

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

eastig reviewed Jan 11, 2023

View reviewed changes

src/hotspot/share/services/memReporter.hpp Show resolved Hide resolved

eastig reviewed Jan 11, 2023

View reviewed changes

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

tstuefe reviewed Jan 12, 2023

View reviewed changes

src/hotspot/share/services/memReporter.hpp Show resolved Hide resolved

src/hotspot/share/services/memReporter.hpp Show resolved Hide resolved

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_c…

8aba62f

…urrent_scale

eastig reviewed Jan 13, 2023

View reviewed changes

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

eastig suggested changes Jan 13, 2023

View reviewed changes

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_c…

91ba8d4

…urrent_scale

kimbarrett approved these changes Jan 16, 2023

View reviewed changes

src/hotspot/share/services/memReporter.hpp Outdated Show resolved Hide resolved

openjdk bot added the ready Pull request is ready to be integrated label Jan 16, 2023

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_c…

a1f9930

…urrent_scale

openjdk bot added the sponsor Pull request is ready to be sponsored label Jan 20, 2023

eastig approved these changes Jan 20, 2023

View reviewed changes

openjdk bot added the integrated Pull request has been integrated label Jan 20, 2023

openjdk bot closed this Jan 20, 2023

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review sponsor Pull request is ready to be sponsored labels Jan 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_current_scale #11514

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_current_scale #11514

afshin-zafari commented Dec 5, 2022 •

edited by openjdk bot

Loading

bridgekeeper bot commented Dec 5, 2022

openjdk bot commented Dec 5, 2022

mlbridge bot commented Dec 5, 2022 •

edited

Loading

tstuefe commented Dec 7, 2022

tstuefe commented Dec 7, 2022

afshin-zafari commented Dec 8, 2022

tstuefe commented Dec 8, 2022

afshin-zafari commented Dec 9, 2022

tstuefe commented Dec 9, 2022 •

edited

Loading

afshin-zafari commented Dec 14, 2022

eastig Dec 19, 2022 •

edited

Loading

afshin-zafari Dec 19, 2022

eastig Dec 19, 2022

afshin-zafari Dec 20, 2022

eastig Dec 20, 2022

kimbarrett Dec 24, 2022

kimbarrett Dec 24, 2022

tstuefe left a comment

eastig left a comment

kimbarrett left a comment

openjdk bot commented Jan 16, 2023 •

edited

Loading

afshin-zafari commented Jan 20, 2023

openjdk bot commented Jan 20, 2023

eastig commented Jan 20, 2023

openjdk bot commented Jan 20, 2023

openjdk bot commented Jan 20, 2023

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_current_scale #11514

8281213: Unsafe uses of long and size_t in MemReporterBase::diff_in_current_scale #11514

Conversation

afshin-zafari commented Dec 5, 2022 • edited by openjdk bot Loading

Description

Patch

Test

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Dec 5, 2022

openjdk bot commented Dec 5, 2022

mlbridge bot commented Dec 5, 2022 • edited Loading

Webrevs

tstuefe commented Dec 7, 2022

tstuefe commented Dec 7, 2022

afshin-zafari commented Dec 8, 2022

tstuefe commented Dec 8, 2022

afshin-zafari commented Dec 9, 2022

tstuefe commented Dec 9, 2022 • edited Loading

afshin-zafari commented Dec 14, 2022

eastig Dec 19, 2022 • edited Loading

Choose a reason for hiding this comment

afshin-zafari Dec 19, 2022

Choose a reason for hiding this comment

eastig Dec 19, 2022

Choose a reason for hiding this comment

afshin-zafari Dec 20, 2022

Choose a reason for hiding this comment

eastig Dec 20, 2022

Choose a reason for hiding this comment

kimbarrett Dec 24, 2022

Choose a reason for hiding this comment

kimbarrett Dec 24, 2022

Choose a reason for hiding this comment

tstuefe left a comment

Choose a reason for hiding this comment

eastig left a comment

Choose a reason for hiding this comment

kimbarrett left a comment

Choose a reason for hiding this comment

openjdk bot commented Jan 16, 2023 • edited Loading

afshin-zafari commented Jan 20, 2023

openjdk bot commented Jan 20, 2023

eastig commented Jan 20, 2023

openjdk bot commented Jan 20, 2023

openjdk bot commented Jan 20, 2023

afshin-zafari commented Dec 5, 2022 •

edited by openjdk bot

Loading

mlbridge bot commented Dec 5, 2022 •

edited

Loading

tstuefe commented Dec 9, 2022 •

edited

Loading

eastig Dec 19, 2022 •

edited

Loading

openjdk bot commented Jan 16, 2023 •

edited

Loading