8346714: [ASAN] compressedKlass.cpp reported applying non-zero offset to null pointer #22848

sendaoYan · 2024-12-20T12:56:08Z

Hi all,
CompressedKlassPointers::sanity_check_after_initialization() src/hotspot/share/oops/compressedKlass.cpp:104:38 reported runtime error: applying non-zero offset 4294967296 to null pointer by clang17 UndefinedBehaviorSanitizer.

The _base initial as nullptr in function CompressedKlassPointers::initialize(address addr, size_t len) shows as below. In C/C++, offsetting a null pointer is undefined behavior. This PR do not change the original logic but eliminate the undefined behaviour in code, the risk is low.

    address const end = addr + len;
    if (end <= (address)unscaled_max) {
      _base = nullptr;
      _shift = 0;

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8346714: [ASAN] compressedKlass.cpp reported applying non-zero offset to null pointer (Bug - P3)

Reviewers

Martin Doerr (@TheRealMDoerr - Reviewer)
Coleen Phillimore (@coleenp - Reviewer) Review applies to 86f00f87

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/22848/head:pull/22848
$ git checkout pull/22848

Update a local copy of the PR:
$ git checkout pull/22848
$ git pull https://git.openjdk.org/jdk.git pull/22848/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 22848

View PR using the GUI difftool:
$ git pr show -t 22848

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/22848.diff

Using Webrev

Link to Webrev Comment

… to null pointer

bridgekeeper · 2024-12-20T12:56:56Z

👋 Welcome back syan! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2024-12-20T12:57:53Z

@sendaoYan This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8346714: [ASAN] compressedKlass.cpp reported applying non-zero offset to null pointer

Reviewed-by: mdoerr, coleenp

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 7 new commits pushed to the master branch:

c1b868d: 8346602: Remove unused macro parameters in jni.cpp
43b7e9f: 8346713: [testsuite] NeverActAsServerClassMachine breaks TestPLABAdaptToMinTLABSize.java TestPinnedHumongousFragmentation.java TestPinnedObjectContents.java
249f141: 8346737: GenShen: Generational memory pools should not report zero for maximum capacity
d562d3c: 8343882: BasicAnnoTests doesn't handle multiple annotations at the same position
7ba969a: 8346739: jpackage tests failed after JDK-8345259
b8e40b9: 8346688: GenShen: Missing metadata trigger log message
d2a4863: 8346690: Shenandoah: Fix log message for end of GC usage report

Please see this link for an up-to-date comparison between the source branch of this pull request and the master branch.
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

openjdk · 2024-12-20T12:58:31Z

@sendaoYan The following label will be automatically applied to this pull request:

hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2024-12-20T13:02:15Z

Webrevs

TheRealMDoerr

"encoding" is the correct spelling. If you cast _base and encoding_offset to uintptr_t you can do the addition without null check. (Then cast the sum back to address.)

sendaoYan · 2024-12-20T15:13:06Z

"encoding" is the correct spelling. If you cast _base and encoding_offset to uintptr_t you can do the addition without null check. (Then cast the sum back to address.)

Thanks for the advice. PR has been updated.

TheRealMDoerr

Thanks for fixing the issue! This should work.
In general, I still prefer using uintptr_t because intptr_t has undefined behavior on overflow. Probably not in this case, here.

sendaoYan · 2024-12-20T16:05:19Z

I still prefer using uintptr_t because intptr_t has undefined behavior on overflow.

Sorry, got it now. The _base and offset has been cast to uintptr_t.

TheRealMDoerr

Thanks! LGTM.

coleenp

This looks better than the conditional on _addr. Thanks for fixing these errors.

sendaoYan · 2024-12-21T02:27:45Z

GHA report 1 failure:

linux-x64 / test - Test (tier1) job report test gc/shenandoah/TestSmallHeap.java#generational timed out, the shenandoah-generational interminnet timed out failure has been recorded by JDK-8345958, it's unrelated to this PR.

tstuefe · 2024-12-21T10:57:21Z

Thanks for fixing the issue! This should work. In general, I still prefer using uintptr_t because intptr_t has undefined behavior on overflow. Probably not in this case, here.

+1. Also, instead of casting manually, please use p2u (and if there is none, we should add it like we have a p2i for intptr_t).

I am somewhat unenthusiastic about changes like these. We seem to add a lot of casting boilerplate to satisfy UBsan for the sake of cleanliness that only matters on special hardware that treats pointers differently from numeric. Like old-style IBM AS/400s. But if we ever build for that kind of hardware, nothing will work anyway. I am happy to be corrected, though, but I think all our compilers treat NULL as numeric 0, right?

sendaoYan · 2024-12-21T15:52:39Z

instead of casting manually, please use p2u (and if there is none, we should add it like we have a p2i for intptr_t).

Casting manually has been replaced as use p2u.

theRealAph · 2024-12-22T17:46:52Z

Thanks for fixing the issue! This should work. In general, I still prefer using uintptr_t because intptr_t has undefined behavior on overflow. Probably not in this case, here.

+1. Also, instead of casting manually, please use p2u (and if there is none, we should add it like we have a p2i for intptr_t).

I am somewhat unenthusiastic about changes like these. We seem to add a lot of casting boilerplate to satisfy UBsan for the sake of cleanliness that only matters on special hardware that treats pointers differently from numeric.

True, but we can very effectively make the compiler and the reader happy by using uintptr_t for all of the arithmetic. After all, we are not using null-based compressed pointers, we are using zero-based compressed pointers, and the distinction is important. uintptr_t does not have null as a member, it has zero; conversely address does not have zero, it has null. Therefore, when we are using zero-based compressed pointers, better to to use uintptr_t for the arithmetic.

tstuefe · 2024-12-23T07:23:38Z

Thanks for fixing the issue! This should work. In general, I still prefer using uintptr_t because intptr_t has undefined behavior on overflow. Probably not in this case, here.

+1. Also, instead of casting manually, please use p2u (and if there is none, we should add it like we have a p2i for intptr_t).
I am somewhat unenthusiastic about changes like these. We seem to add a lot of casting boilerplate to satisfy UBsan for the sake of cleanliness that only matters on special hardware that treats pointers differently from numeric.

True, but we can very effectively make the compiler and the reader happy by using uintptr_t for all of the arithmetic. After all, we are not using null-based compressed pointers, we are using zero-based compressed pointers, and the distinction is important. uintptr_t does not have null as a member, it has zero; conversely address does not have zero, it has null. Therefore, when we are using zero-based compressed pointers, better to to use uintptr_t for the arithmetic.

Okay, but this is just for the sake of the reader, right? Because a cast can be just as invalid as adding to NULL. I worked on hardware that marked pointers as invalid if they resulted from an integer cast. I assume that "don't add to NULL" warnings exist to preserve portability for these platforms. My thought is: since we cannot hope to achieve portability for these platforms anyway, we might just as well allow adding to NULL. Casting feels like just switching off these warnings. Note that even if I convert the whole of CompressedKlassPointers to uintptr_t, we will need a cast at the latest when we use the base as input to mmap.

But I agree that UBSAN cleanliness is valuable since most of its warnings are good, and maybe its just simpler to fix these issues than to argue about them. So, I am fine with this patch.

tstuefe · 2024-12-23T08:11:26Z

src/hotspot/share/oops/compressedKlass.cpp

@@ -96,7 +96,7 @@ void CompressedKlassPointers::sanity_check_after_initialization() {

  // Check that Klass range is fully engulfed in the encoding range
  const address encoding_start = _base;
-  const address encoding_end = _base + nth_bit(narrow_klass_pointer_bits() + _shift);
+  const address encoding_end = (address)(p2u(_base) + (uintptr_t)nth_bit(narrow_klass_pointer_bits() + _shift));


nth_bit should already give us a 64-bit value, why the second cast?
I see that nth_bit returns an intptr_t - is the sign the problem? We may want to change that to uintptr_t...

Prefer using uintptr_t because intptr_t has undefined behavior on overflow. Probably not in this case, here.

It should work without the second cast, but there may be compiler warnings like "warning: implicit conversion changes signedness: 'long' to 'unsigned long' [-Wsign-conversion]".

It should work without the second cast, but there may be compiler warnings like "warning: implicit conversion changes signedness: 'long' to 'unsigned long' [-Wsign-conversion]".

Yes, below example can reproduce the gcc/clang compile warning.

#include <stdio.h> int main() { unsigned long a = 1; signed long b = 2; printf("%llu\n", (long long unsigned int)(a + b)); return 0; }

TheRealMDoerr · 2024-12-23T10:17:26Z

Thanks for fixing the issue! This should work. In general, I still prefer using uintptr_t because intptr_t has undefined behavior on overflow. Probably not in this case, here.

+1. Also, instead of casting manually, please use p2u (and if there is none, we should add it like we have a p2i for intptr_t).
I am somewhat unenthusiastic about changes like these. We seem to add a lot of casting boilerplate to satisfy UBsan for the sake of cleanliness that only matters on special hardware that treats pointers differently from numeric.

True, but we can very effectively make the compiler and the reader happy by using uintptr_t for all of the arithmetic. After all, we are not using null-based compressed pointers, we are using zero-based compressed pointers, and the distinction is important. uintptr_t does not have null as a member, it has zero; conversely address does not have zero, it has null. Therefore, when we are using zero-based compressed pointers, better to to use uintptr_t for the arithmetic.

Okay, but this is just for the sake of the reader, right? Because a cast can be just as invalid as adding to NULL. I worked on hardware that marked pointers as invalid if they resulted from an integer cast. I assume that "don't add to NULL" warnings exist to preserve portability for these platforms. My thought is: since we cannot hope to achieve portability for these platforms anyway, we might just as well allow adding to NULL. Casting feels like just switching off these warnings. Note that even if I convert the whole of CompressedKlassPointers to uintptr_t, we will need a cast at the latest when we use the base as input to mmap.

But I agree that UBSAN cleanliness is valuable since most of its warnings are good, and maybe its just simpler to fix these issues than to argue about them. So, I am fine with this patch.

Undefined behavior is a risk. Compilers currently seem to do what we want, but there's no guarantee. They could optimize the code in a way which breaks it. That would be legal since it is undefined behavior. uintptr_t arithmetics are defined as well as the casts. "Any value of type std::nullptr_t, including nullptr can be converted to any integral type as if it were (void*)0" [https://en.cppreference.com/w/cpp/language/reinterpret_cast]. I guess compilers need to handle the conversion during the casts on platforms on which nullptr is not the zero address.

sendaoYan · 2024-12-23T11:26:11Z

Thank all for the detailed discussion, explanation and reviews.

/integrate

openjdk · 2024-12-23T11:27:13Z

Going to push as commit bffa77b.
Since your change was applied there have been 7 commits pushed to the master branch:

c1b868d: 8346602: Remove unused macro parameters in jni.cpp
43b7e9f: 8346713: [testsuite] NeverActAsServerClassMachine breaks TestPLABAdaptToMinTLABSize.java TestPinnedHumongousFragmentation.java TestPinnedObjectContents.java
249f141: 8346737: GenShen: Generational memory pools should not report zero for maximum capacity
d562d3c: 8343882: BasicAnnoTests doesn't handle multiple annotations at the same position
7ba969a: 8346739: jpackage tests failed after JDK-8345259
b8e40b9: 8346688: GenShen: Missing metadata trigger log message
d2a4863: 8346690: Shenandoah: Fix log message for end of GC usage report

Your commit was automatically rebased without conflicts.

openjdk · 2024-12-23T11:27:23Z

@sendaoYan Pushed as commit bffa77b.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

mlbridge · 2024-12-23T19:13:28Z

Mailing list message from Andrew Haley on hotspot-dev:

On 12/23/24 07:26, Thomas Stuefe wrote:

On Sun, 22 Dec 2024 17:46:52 GMT, Andrew Haley <aph at openjdk.org> wrote:

Thanks for fixing the issue! This should work. In general, I still prefer using `uintptr_t` because `intptr_t` has undefined behavior on overflow. Probably not in this case, here.

+1. Also, instead of casting manually, please use p2u (and if there is none, we should add it like we have a p2i for intptr_t).
I am somewhat unenthusiastic about changes like these. We seem to add a lot of casting boilerplate to satisfy UBsan for the sake of cleanliness that only matters on special hardware that treats pointers differently from numeric.

True, but we can very effectively make the compiler and the reader happy by using `uintptr_t` for all of the arithmetic. After all, we are not using null-based compressed pointers, we are using zero-based compressed pointers, and the distinction is important. `uintptr_t` does not have null as a member, it has zero; conversely `address` does not have zero, it has null. Therefore, when we are using zero-based compressed pointers, better to to use `uintptr_t` for the arithmetic.

Okay, but this is just for the sake of the reader, right? Because a cast can be just as invalid as adding to NULL.

I don't agree. Adding to NULL is explicitly UB. C++ doesn't have to generate
any code for it.

I worked on hardware that marked pointers as invalid if they resulted from an integer cast. I assume that "don't add to NULL" warnings exist to preserve portability for these platforms.

That's not so. Current C++ compilers can, and some do, correctly generate no code for anything following UB.

If you were to convert a pointer to uintptr_t, discover that it fits in an int, convert it to an int, and
then convert it back to a pointer, that is well-defined C++, and must be executed. You can even write a
pointer to a file, read it back later, and use that pointer. And it must work.

My thought is: since we cannot hope to achieve portability for these platforms anyway, we might just as well allow adding to NULL.

No. That is UB.

Casting feels like just switching off these warnings.

That's how it feels, sure, but you're moving from UB to implementation-defined behaviour.
The latter we can cope with.

Note that even if I convert the whole of CompressedKlassPointers to uintptr_t, we will need a cast at the latest when we use the base as input to mmap.

Yes. Good.

But I agree that UBSAN cleanliness is valuable since most of its warnings are good, and maybe its just simpler to fix these issues than to argue about them. So, I am fine with this patch.

tstuefe · 2024-12-25T07:05:55Z

@theRealAph @TheRealMDoerr Thank you for your points! You have convinced me.

8346714: [ASAN] compressedKlass.cpp reported applying non-zero offset…

766e09b

… to null pointer

sendaoYan mentioned this pull request Dec 20, 2024

[ASAN][upstream][25][clang]compressedKlass.cpp:99:38: runtime error: applying non-zero offset 4294967296 to null pointer dragonwell-project/dragonwell21#143

Closed

openjdk bot added the rfr Pull request is ready for review label Dec 20, 2024

openjdk bot added the hotspot hotspot-dev@openjdk.org label Dec 20, 2024

TheRealMDoerr reviewed Dec 20, 2024

View reviewed changes

case _base to inrptr_t and add the offset, to avoid null check

6abd26e

TheRealMDoerr reviewed Dec 20, 2024

View reviewed changes

sendaoYan added 2 commits December 21, 2024 00:01

cast offset to intptr_t to avoid overflow

4e54333

Use uintptr_t instead intptr_t

86f00f8

TheRealMDoerr approved these changes Dec 20, 2024

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Dec 20, 2024

coleenp approved these changes Dec 20, 2024

View reviewed changes

add function p2u and use function p2u instead of cast manually

f9b9bb3

openjdk bot removed the ready Pull request is ready to be integrated label Dec 21, 2024

tstuefe reviewed Dec 23, 2024

View reviewed changes

TheRealMDoerr approved these changes Dec 23, 2024

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Dec 23, 2024

openjdk bot added the integrated Pull request has been integrated label Dec 23, 2024

openjdk bot closed this Dec 23, 2024

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Dec 23, 2024

sendaoYan deleted the jbs8346714 branch December 23, 2024 12:01

sendaoYan mentioned this pull request Feb 5, 2025

8349465: [UBSAN] test_os_reserve_between.cpp reported applying non-zero offset to null pointer #23462

Closed

3 tasks

8346714: [ASAN] compressedKlass.cpp reported applying non-zero offset to null pointer #22848

8346714: [ASAN] compressedKlass.cpp reported applying non-zero offset to null pointer #22848

Uh oh!

Conversation

sendaoYan commented Dec 20, 2024 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Reviewing

Uh oh!

bridgekeeper bot commented Dec 20, 2024

Uh oh!

openjdk bot commented Dec 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openjdk bot commented Dec 20, 2024

Uh oh!

mlbridge bot commented Dec 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

TheRealMDoerr left a comment

Choose a reason for hiding this comment

Uh oh!

sendaoYan commented Dec 20, 2024

Uh oh!

TheRealMDoerr left a comment

Choose a reason for hiding this comment

Uh oh!

sendaoYan commented Dec 20, 2024

Uh oh!

TheRealMDoerr left a comment

Choose a reason for hiding this comment

Uh oh!

coleenp left a comment

Choose a reason for hiding this comment

Uh oh!

sendaoYan commented Dec 21, 2024

Uh oh!

tstuefe commented Dec 21, 2024

Uh oh!

sendaoYan commented Dec 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theRealAph commented Dec 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tstuefe commented Dec 23, 2024

Uh oh!

tstuefe Dec 23, 2024

Choose a reason for hiding this comment

Uh oh!

sendaoYan Dec 23, 2024

Choose a reason for hiding this comment

Uh oh!

TheRealMDoerr Dec 23, 2024

Choose a reason for hiding this comment

Uh oh!

sendaoYan Dec 23, 2024

Choose a reason for hiding this comment

Uh oh!

TheRealMDoerr commented Dec 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sendaoYan commented Dec 23, 2024

Uh oh!

openjdk bot commented Dec 23, 2024

Uh oh!

openjdk bot commented Dec 23, 2024

Uh oh!

mlbridge bot commented Dec 23, 2024

Uh oh!

tstuefe commented Dec 25, 2024

Uh oh!

Uh oh!

sendaoYan commented Dec 20, 2024 •

edited by openjdk bot

Loading

openjdk bot commented Dec 20, 2024 •

edited

Loading

mlbridge bot commented Dec 20, 2024 •

edited

Loading

sendaoYan commented Dec 21, 2024 •

edited

Loading

theRealAph commented Dec 22, 2024 •

edited

Loading

TheRealMDoerr commented Dec 23, 2024 •

edited

Loading