8277299: STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIconBits #6473

azuev-java · 2021-11-19T06:34:30Z

Made colorBits and maskBits arrays dynamic so they are allocated on heap instead of stack.
Added regression test.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8277299: STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIconBits

Reviewers

Alexey Ivanov (@aivanov-jdk - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/6473/head:pull/6473
$ git checkout pull/6473

Update a local copy of the PR:
$ git checkout pull/6473
$ git pull https://git.openjdk.java.net/jdk pull/6473/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 6473

View PR using the GUI difftool:
$ git pr show -t 6473

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/6473.diff

…onBits Made colorBits and maskBits arraus dynamic so they allocated on heap instead of stack. Added regression test.

bridgekeeper · 2021-11-19T06:35:58Z

👋 Welcome back kizune! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-11-19T06:39:32Z

@azuev-java The following label will be automatically applied to this pull request:

client

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-11-19T06:42:52Z

Webrevs

mrserb · 2021-11-19T20:21:27Z

src/java.desktop/windows/native/libawt/windows/ShellFolder2.cpp

@@ -1056,7 +1056,8 @@ JNIEXPORT jintArray JNICALL Java_sun_awt_shell_Win32ShellFolder2_getIconBits
            bmi.bmiHeader.biCompression = BI_RGB;
            // Extract the color bitmap
            int nBits = iconSize * iconSize;
-            long colorBits[MAX_ICON_SIZE * MAX_ICON_SIZE];
+            long * colorBits;
+            colorBits = (long*)safe_Malloc(MAX_ICON_SIZE * MAX_ICON_SIZE * sizeof(long));


I am not sure that the bad_alloc will be properly handled in this Java_sun_awt_shell_Win32ShellFolder2_getIconBits method.
+Probably it will be better to merge assigning into one line.

I am not sure that the bad_alloc will be properly handled in this Java_sun_awt_shell_Win32ShellFolder2_getIconBits method.

I can't see any try-catch.
Is it better to use malloc and check for NULL?

Probably it will be better to merge assigning into one line.

I agree. The line doesn't get too long.

Probably TRY/CATCH_BAD_ALLOC_RET(NULL); could be used. It depends on how we will clean the stuff on exception.

aivanov-jdk · 2021-11-19T20:42:13Z

src/java.desktop/windows/native/libawt/windows/ShellFolder2.cpp

@@ -1056,7 +1056,8 @@ JNIEXPORT jintArray JNICALL Java_sun_awt_shell_Win32ShellFolder2_getIconBits
            bmi.bmiHeader.biCompression = BI_RGB;
            // Extract the color bitmap
            int nBits = iconSize * iconSize;
-            long colorBits[MAX_ICON_SIZE * MAX_ICON_SIZE];
+            long * colorBits;


Pointer declarations aren't consistent in the file, the function parameter is declared as JNIEnv* env. However, in the majority of cases the asterisk is near the variable name as in const char *pLibName. In either case, there's only one space:

long *maskBits;

or

long* maskBits;

aivanov-jdk · 2021-11-19T20:47:01Z

src/java.desktop/windows/native/libawt/windows/ShellFolder2.cpp

@@ -1056,7 +1056,8 @@ JNIEXPORT jintArray JNICALL Java_sun_awt_shell_Win32ShellFolder2_getIconBits
            bmi.bmiHeader.biCompression = BI_RGB;
            // Extract the color bitmap
            int nBits = iconSize * iconSize;
-            long colorBits[MAX_ICON_SIZE * MAX_ICON_SIZE];
+            long * colorBits;
+            colorBits = (long*)safe_Malloc(MAX_ICON_SIZE * MAX_ICON_SIZE * sizeof(long));


I am not sure that the bad_alloc will be properly handled in this Java_sun_awt_shell_Win32ShellFolder2_getIconBits method.

I can't see any try-catch.
Is it better to use malloc and check for NULL?

Probably it will be better to merge assigning into one line.

I agree. The line doesn't get too long.

aivanov-jdk · 2021-11-19T20:49:58Z

test/jdk/javax/swing/JFileChooser/FileSystemView/ShellFolderStackOverflow.java

+ * @test
+ * @bug 8277299
+ * @requires (os.family == "windows")
+ * @summary STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIconBits


Maybe this could be spelt with regular case rather than caps?

Maybe this could be spelt with regular case rather than caps?

Well, it is just copy-paste from the bug synopsis which is copy-paste from the error message of the VM crash.

Fixed insets Declaration and assignment are now joined

aivanov-jdk · 2021-11-22T19:13:11Z

src/java.desktop/windows/native/libawt/windows/ShellFolder2.cpp

            free(colorBits);
-        }
+
+            CATCH_BAD_ALLOC_RET(NULL);


I believe we leak dc as well as iconInfo.hbmColor and iconInfo.hbmMask if std::bad_alloc is thrown.

Fixed. Now we will release these resources on exit in case of bad_alloc.

aivanov-jdk · 2021-12-03T13:26:57Z

src/java.desktop/windows/native/libawt/windows/ShellFolder2.cpp

+            try {
+                entry_point();
+                colorBits = (long*)safe_Malloc(MAX_ICON_SIZE * MAX_ICON_SIZE * sizeof(long));
+            } catch(std::bad_alloc&) {
+                handle_bad_alloc();
+            }
+
+            if (colorBits != NULL) {


I think this way defeats the use of exception to handle allocation error. You could use:

long *colorBits = (long*)malloc(MAX_ICON_SIZE * MAX_ICON_SIZE * sizeof(long)); if (colorBits != NULL) {

to achieve the same effect, which is shorter and clearer.

Yes, but then handle_bad_alloc() will not be called which will not raise the OutOfMemoryError on the java side.

Yes, but then handle_bad_alloc() will not be called which will not raise the OutOfMemoryError on the java side.

Can't we call it explicitly in the case where colorBits == NULL?

Alternatively, since you don't use the macros for try and catch, you can re-arrange code and perform cleanup after the catch-block. Just an idea… It may not work.

Can't we call it explicitly in the case where colorBits == NULL?

I would use the default handlers for bad_alloc since they can be changed in feature and keeping code that is intended to be used in two places is usually not a good idea.

Alternatively, since you don't use the macros for try and catch, you can re-arrange code and perform cleanup after the catch-block. Just an idea… It may not work.

That might require a lot of code reformatting and the benefits are unclear.

Alternatively, since you don't use the macros for try and catch, you can re-arrange code and perform cleanup after the catch-block. Just an idea… It may not work.

That might require a lot of code reformatting and the benefits are unclear.

I tried it, it doesn't, see this diff. It's close to your latest code, however, there's only one catch that handles both allocations and there are no if's inside the try-block.

What about this approach?
Compared to master
Compared to your latest code

I think it's cleaner in the use of exceptions. Each try-block allocates memory that it uses and frees it before the catch-block. There's no need for if's: if exception is thrown, the memory wasn't allocated, therefore the memory is freed only if no exception is thrown.

What do you think?

I like the singular try/catch block approach better so i did that in the latest commit.

aivanov-jdk · 2021-12-03T13:39:58Z

src/java.desktop/windows/native/libawt/windows/ShellFolder2.cpp

+                // Release DC
+                ReleaseDC(NULL, dc);


The DC has to be released even if colorBits allocation failed, so this needs to be after if (colorBits != NULL).

Probably, you hasn't pushed the change for releasing the DC.

aivanov-jdk

I admit I prefer the 2nd version with two try-catch blocks better because it avoids using if's completely.

Anyway, this version looks good to me. Thanks.

aivanov-jdk · 2021-12-10T17:18:15Z

src/java.desktop/windows/native/libawt/windows/ShellFolder2.cpp

+                    // Extract the mask bitmap
+
+                    maskBits = (long*)safe_Malloc(MAX_ICON_SIZE * MAX_ICON_SIZE * sizeof(long));
+
+                    GetDIBits(dc, iconInfo.hbmMask, 0, iconSize, maskBits, &bmi, DIB_RGB_COLORS);


Probably the empty lines here aren't necessary.

openjdk · 2021-12-10T17:29:08Z

@azuev-java This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8277299: STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIconBits

Reviewed-by: aivanov

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 345 new commits pushed to the master branch:

8eb453b: 8277072: ObjectStreamClass caches keep ClassLoaders alive
3e0b083: 8278533: Remove some unused methods in c1_Instruction and c1_ValueMap
aed3ea2: 8277634: Incorrect method name in invokedynamic
539fbbf: 8278532: Fix some typos in compiler comments
0113322: 8277997: Intrinsic creation for VectorMask.fromLong API
066b348: 8277807: Increase default initial concurrent refinement threshold
ec0a5ac: Merge
d40e90b: 8278521: ProblemList java/lang/management/ThreadMXBean/ThreadLists.java
918b350: 8278516: Typos in snippet for java.compiler
fcd67a5: 8278185: Custom JRE cannot find non-ASCII named module inside
... and 335 more: https://git.openjdk.java.net/jdk/compare/91607436b79126ccb999deaf38a98209dbfe6ec1...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

8277299: STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIc…

fa8a4ed

…onBits Made colorBits and maskBits arraus dynamic so they allocated on heap instead of stack. Added regression test.

openjdk bot added the rfr Pull request is ready for review label Nov 19, 2021

openjdk bot added the client client-libs-dev@openjdk.org label Nov 19, 2021

mrserb reviewed Nov 19, 2021

View reviewed changes

aivanov-jdk reviewed Nov 19, 2021

View reviewed changes

Added bad_malloc handling

16d3ead

Fixed insets Declaration and assignment are now joined

aivanov-jdk reviewed Nov 22, 2021

View reviewed changes

Eliminate potential resource leak.

c8d8ee2

aivanov-jdk suggested changes Dec 3, 2021

View reviewed changes

azuev-java added 2 commits December 3, 2021 12:15

elease device context even if colorBits buffer can not be allocated.

d72abd9

Reformat the code getting rid of the sectnd try block

39a36a1

aivanov-jdk approved these changes Dec 10, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Dec 10, 2021

aivanov-jdk mentioned this pull request Dec 10, 2021

8277299: STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIconBits openjdk/jdk18#12

Closed

3 tasks

azuev-java closed this Dec 11, 2021

azuev-java deleted the JDK-8277299 branch January 31, 2022 09:38

8277299: STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIconBits #6473

8277299: STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIconBits #6473

azuev-java commented Nov 19, 2021 •

edited by openjdk bot

bridgekeeper bot commented Nov 19, 2021

openjdk bot commented Nov 19, 2021

mlbridge bot commented Nov 19, 2021 •

edited

mrserb Nov 19, 2021

aivanov-jdk Nov 19, 2021

mrserb Nov 19, 2021

aivanov-jdk Nov 19, 2021

aivanov-jdk Nov 19, 2021

aivanov-jdk Nov 19, 2021

azuev-java Nov 20, 2021

aivanov-jdk Nov 22, 2021

azuev-java Dec 2, 2021

aivanov-jdk Dec 3, 2021

azuev-java Dec 3, 2021

aivanov-jdk Dec 3, 2021

azuev-java Dec 4, 2021

aivanov-jdk Dec 4, 2021

aivanov-jdk Dec 6, 2021

azuev-java Dec 7, 2021

aivanov-jdk Dec 3, 2021

azuev-java Dec 3, 2021

aivanov-jdk Dec 4, 2021

azuev-java Dec 6, 2021

aivanov-jdk left a comment

aivanov-jdk Dec 10, 2021

openjdk bot commented Dec 10, 2021 •

edited

8277299: STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIconBits #6473

8277299: STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIconBits #6473

Conversation

azuev-java commented Nov 19, 2021 • edited by openjdk bot

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Nov 19, 2021

openjdk bot commented Nov 19, 2021

mlbridge bot commented Nov 19, 2021 • edited

Webrevs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aivanov-jdk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openjdk bot commented Dec 10, 2021 • edited

azuev-java commented Nov 19, 2021 •

edited by openjdk bot

mlbridge bot commented Nov 19, 2021 •

edited

openjdk bot commented Dec 10, 2021 •

edited