8271419: Refactor test code for modifying CDS archive contents #4945

yminqi · 2021-07-30T15:33:14Z

The code for modifying CDS archive content could be used by other tests so the common code is refactored into a util class CDSArchiveUtils.

Tests: tier1,tier4

Thanks
Yumin

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8271419: Refactor test code for modifying CDS archive contents

Reviewers

Ioi Lam (@iklam - Reviewer)
Calvin Cheung (@calvinccheung - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/4945/head:pull/4945
$ git checkout pull/4945

Update a local copy of the PR:
$ git checkout pull/4945
$ git pull https://git.openjdk.java.net/jdk pull/4945/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 4945

View PR using the GUI difftool:
$ git pr show -t 4945

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/4945.diff

bridgekeeper · 2021-07-30T15:33:44Z

👋 Welcome back minqi! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

yminqi · 2021-07-30T15:33:59Z

/label add hotspot-runtime

openjdk · 2021-07-30T15:36:05Z

@yminqi
The hotspot-runtime label was successfully added.

mlbridge · 2021-07-30T15:39:49Z

Webrevs

iklam · 2021-07-30T18:50:47Z

test/lib/jdk/test/lib/cds/CDSArchiveUtils.java

+        alignment = wb.metaspaceSharedRegionAlignment();
+        // fileHeaderSize may not be available
+        // fileHeaderSize = (int)alignUpWithPageSize(wb.getOffsetForName("file_header_size"));
+    }


This should always be available. file_header_size is a constant value (sizeof(FileMapHeader)) stored inside cdsoffsets.cpp. It doesn't depend on the contents of the archive file.

This is same as original code, I will investigate more of this part. Thanks.

iklam · 2021-07-30T18:50:47Z

test/hotspot/jtreg/runtime/cds/appcds/SharedArchiveConsistency.java

-        if (fc.isOpen()) {
-            fc.close();
-        }
+        System.out.println("    offset_jvm_ident " + CDSArchiveUtils.offsetJvmIdent);


I think the modifyXXX functions should also be moved to CDSArchiveUtils so that they can be called by other test cases.

Let me double check this part.

calvinccheung · 2021-07-30T21:02:18Z

test/hotspot/jtreg/runtime/cds/appcds/SharedArchiveConsistency.java

-        if (fc.isOpen()) {
-            fc.close();
-        }
+        byte[] buf = { (byte)(value >> 24), (byte)(value >> 16), (byte)(value >> 8), (byte)(value)};


Isn't this the same as the original code which does a ByteBuffer.wrap()?
If so, why changing it.

I make CDSARchiveUtils take byte[] as input, not ByteBuffer. Here just simply convert int to byte[].

calvinccheung · 2021-07-30T21:02:24Z

test/hotspot/jtreg/runtime/cds/appcds/SharedArchiveConsistency.java

+        WhiteBox box = WhiteBox.getWhiteBox();
+        CDSArchiveUtils.initialize(box);  // all offsets available


After this refactoring, I don't think WhiteBox is required in this file.
So I think CDSArchiveUtils could create a WhiteBox instance instead of requiring the caller to supply one.

The util class is just a utility class, I have no idea if we can do this, let me try --- it might work.

calvinccheung

I have 2 comments.

iklam · 2021-08-02T06:49:41Z

test/hotspot/jtreg/runtime/cds/appcds/SharedArchiveConsistency.java

@@ -89,261 +62,21 @@
    public static int num_regions = shared_region_name.length;
    public static String[] matchMessages = {
        "Unable to use shared archive",
+        "The shared archive file has a bad magic number",
+        "Unable to map shared spaces",


Why are these new errors needed? Did you change any behavior of the tests?

Thanks for mentioning this. There is an issue to use FileChannel:
outputChannel.transferFrom(inputChannel, 0, offset);
+ outputChannel.position(offset); // <---- need explicitly positioned or the following write to overwrite from offset 0.
outputChannel.write(ByteBuffer.wrap(bytes));
outputChannel.transferFrom(inputChannel, offset + bytes.length, orgSize - bytes.length);

The added lines should be removed.

iklam · 2021-08-02T06:49:41Z

test/lib/jdk/test/lib/cds/CDSArchiveUtils.java

+    public static int num_regions = shared_region_name.length;
+
+    public static void initialize() throws Exception {
+        WhiteBox wb = WhiteBox.getWhiteBox();


This method can be changed to a static { ....} block, so that you don't need to explicitly call CDSArchiveUtils.initialize().

Will move to static block.

calvinccheung · 2021-08-02T18:43:51Z

test/hotspot/jtreg/runtime/cds/appcds/SharedArchiveConsistency.java

@@ -34,42 +34,15 @@
 * @run main/othervm -Xbootclasspath/a:. -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI SharedArchiveConsistency on
 * @run main/othervm -Xbootclasspath/a:. -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI SharedArchiveConsistency auto
 */
+import jdk.test.lib.cds.CDSArchiveUtils;
 import jdk.test.lib.process.OutputAnalyzer;
 import jdk.test.lib.Utils;


The above import of Utils is no longer needed.

This one is new added util class, it is needed here. Do you think it is CDSTestUtils ?

I was referring to import jdk.test.lib.Utils;. It was for Utils.getRandomInstance(); and the code has been moved to the new class CDSArchiveUtils.

I see, thanks. Will remove.

calvinccheung · 2021-08-02T18:45:51Z

test/hotspot/jtreg/runtime/cds/appcds/SharedArchiveConsistency.java

 import java.util.Random;
 import sun.hotspot.WhiteBox;


The import of Random is no longer needed.
I think the import of WhiteBox can also be removed since it is not used in this file.

Yes, they no longer needed. thanks.

calvinccheung

Just couple of nits.

openjdk · 2021-08-02T20:20:34Z

@yminqi This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8271419: Refactor test code for modifying CDS archive contents

Reviewed-by: iklam, ccheung

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 28 new commits pushed to the master branch:

0a85236: 8193559: ugly DO_JAVA_THREADS macro should be replaced
db950ca: 8271348: Add stronger sanity check of thread state when polling for safepoint/handshakes
3e3051e: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside
7a4c754: 8271611: Use SecurityConstants.ACCESS_PERMISSION in MethodHandles
e74537f: 8271605: Update JMH devkit to 1.32
249d641: 8263561: Re-examine uses of LinkedList
6a3f834: 8268113: Re-use Long.hashCode() where possible
2536e43: 8270160: Remove redundant bounds check from AbstractStringBuilder.charAt()
6c4c48f: 8266972: Use String.concat() in j.l.Class where invokedynamic-based String concatenation is not available
72145f3: 8269665: Clean-up toString() methods of some primitive wrappers
... and 18 more: https://git.openjdk.java.net/jdk/compare/d09b028407ff9d0e8c2dfd9cc5d0dca19c4497e3...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

iklam

LGTM

yminqi · 2021-08-02T23:06:50Z

@calvinccheung @iklam Thanks for review!
/integrate

openjdk · 2021-08-02T23:07:55Z

Going to push as commit 84f0231.
Since your change was applied there have been 30 commits pushed to the master branch:

0b95394: 8271624: Avoid unnecessary ThreadGroup.checkAccess calls when creating Threads
e621cff: 8271627: Use local field access in favor of Class.getClassLoader0
0a85236: 8193559: ugly DO_JAVA_THREADS macro should be replaced
db950ca: 8271348: Add stronger sanity check of thread state when polling for safepoint/handshakes
3e3051e: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside
7a4c754: 8271611: Use SecurityConstants.ACCESS_PERMISSION in MethodHandles
e74537f: 8271605: Update JMH devkit to 1.32
249d641: 8263561: Re-examine uses of LinkedList
6a3f834: 8268113: Re-use Long.hashCode() where possible
2536e43: 8270160: Remove redundant bounds check from AbstractStringBuilder.charAt()
... and 20 more: https://git.openjdk.java.net/jdk/compare/d09b028407ff9d0e8c2dfd9cc5d0dca19c4497e3...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-08-02T23:08:11Z

@yminqi Pushed as commit 84f0231.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8271419: Refactor test code for modifying CDS archive contents

d95f25f

openjdk bot added rfr hotspot-runtime labels Jul 30, 2021

iklam suggested changes Jul 30, 2021

View changes

calvinccheung reviewed Jul 30, 2021

View changes

Move more function to CDSArchiveUtils, fixed review comments

1010490

iklam suggested changes Aug 2, 2021

View changes

Yumin Qi added 2 commits Aug 2, 2021

Fix the FileChannel issue, remove added checkes and no-need imports

9dc84ec

Remove Utils from import

2479f43

calvinccheung reviewed Aug 2, 2021

View changes

calvinccheung suggested changes Aug 2, 2021

View changes

calvinccheung approved these changes Aug 2, 2021

View changes

openjdk bot added the ready label Aug 2, 2021

iklam approved these changes Aug 2, 2021

View changes

openjdk bot closed this Aug 2, 2021

openjdk bot added integrated and removed ready rfr labels Aug 2, 2021

yminqi deleted the jdk-8271419 branch Aug 2, 2021

8271419: Refactor test code for modifying CDS archive contents #4945

8271419: Refactor test code for modifying CDS archive contents #4945

yminqi commented Jul 30, 2021 •

edited by openjdk[bot] bot

bridgekeeper bot commented Jul 30, 2021

yminqi commented Jul 30, 2021

openjdk bot commented Jul 30, 2021

mlbridge bot commented Jul 30, 2021 •

edited

iklam Jul 30, 2021

yminqi Jul 30, 2021

iklam Jul 30, 2021

yminqi Jul 30, 2021

calvinccheung Jul 30, 2021

yminqi Jul 30, 2021

calvinccheung Jul 30, 2021

yminqi Jul 30, 2021

calvinccheung left a comment

iklam Aug 2, 2021

yminqi Aug 2, 2021

iklam Aug 2, 2021

yminqi Aug 2, 2021

calvinccheung Aug 2, 2021 •

edited

yminqi Aug 2, 2021

calvinccheung Aug 2, 2021

yminqi Aug 2, 2021

calvinccheung Aug 2, 2021

yminqi Aug 2, 2021

calvinccheung left a comment

openjdk bot commented Aug 2, 2021 •

edited

iklam left a comment

yminqi commented Aug 2, 2021

openjdk bot commented Aug 2, 2021

openjdk bot commented Aug 2, 2021

		WhiteBox box = WhiteBox.getWhiteBox();
		CDSArchiveUtils.initialize(box); // all offsets available

8271419: Refactor test code for modifying CDS archive contents #4945

8271419: Refactor test code for modifying CDS archive contents #4945

Conversation

yminqi commented Jul 30, 2021 • edited by openjdk[bot] bot

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Jul 30, 2021

yminqi commented Jul 30, 2021

openjdk bot commented Jul 30, 2021

mlbridge bot commented Jul 30, 2021 • edited

Webrevs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

calvinccheung left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

calvinccheung Aug 2, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

calvinccheung left a comment

openjdk bot commented Aug 2, 2021 • edited

iklam left a comment

yminqi commented Aug 2, 2021

openjdk bot commented Aug 2, 2021

openjdk bot commented Aug 2, 2021

yminqi commented Jul 30, 2021 •

edited by openjdk[bot] bot

mlbridge bot commented Jul 30, 2021 •

edited

calvinccheung Aug 2, 2021 •

edited

openjdk bot commented Aug 2, 2021 •

edited