8303891: Speed up Zip64SizeTest using a small ZIP64 file #12948

eirbjo · 2023-03-09T12:06:52Z

Please review this PR which suggests we speed up the Zip64SizeTest using a small-sized ZIP64 ZIP file specifically created to reproduce the issue being tested.

The disk space requirement of this test is known to cause problems in some builds, see JDK-8259866

By using a sparse file, we reduce consumed disk space from 5GB to 266 bytes and also reduce the runtime from ~35 seconds to ~1 seconds on my Macbook Pro.

The PR also fixes the @summary tag, which seems to have been copied from an unrelated test.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issues

JDK-8303891: Speed up Zip64SizeTest using a small ZIP64 file (Enhancement - P4)
JDK-8259866: two java.util tests failed with "IOException: There is not enough space on the disk" (Bug - P3)

Reviewers

Lance Andersen (@LanceAndersen - Reviewer)
Jaikiran Pai (@jaikiran - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/12948/head:pull/12948
$ git checkout pull/12948

Update a local copy of the PR:
$ git checkout pull/12948
$ git pull https://git.openjdk.org/jdk.git pull/12948/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 12948

View PR using the GUI difftool:
$ git pr show -t 12948

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/12948.diff

Webrev

Link to Webrev Comment

bridgekeeper · 2023-03-09T12:08:33Z

👋 Welcome back eirbjo! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2023-03-09T12:11:17Z

@eirbjo The following label will be automatically applied to this pull request:

core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2023-03-09T12:22:22Z

Webrevs

bridgekeeper · 2023-04-07T23:35:28Z

@eirbjo This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply add a new comment to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

bridgekeeper · 2023-05-06T00:43:13Z

@eirbjo This pull request has been inactive for more than 8 weeks and will now be automatically closed. If you would like to continue working on this pull request in the future, feel free to reopen it! This can be done using the /open pull request command.

…tries in the ZIP file

eirbjo · 2024-02-08T14:06:07Z

/open

openjdk · 2024-02-08T14:06:34Z

@eirbjo This pull request is now open

eirbjo · 2024-02-08T14:11:00Z

Reopening this PR which was closed without review in May 2023.

Initially, this PR suggested to use a sparse file when creating the large ZIP file. After consideration, I have found it simpler to instead doctor a small-sized ZIP64-entry with the specific structure required to trigger the regression being tested.

I have verified that the test actually fails if ZipFile is updated to call ZipEntry.setExtra0 with isLoc: true, meaning it still catches the regression.

eirbjo · 2024-02-08T14:19:59Z

/issue add 8259866

openjdk · 2024-02-08T14:20:52Z

@eirbjo
Adding additional issue to issue list: 8259866: two java.util tests failed with "IOException: There is not enough space on the disk".

test/jdk/java/util/zip/ZipFile/Zip64SizeTest.java

jaikiran · 2024-02-09T07:16:14Z

test/jdk/java/util/zip/ZipFile/Zip64SizeTest.java

+
+            ZipEntry e1 = new ZipEntry("first");
+            // Make room for an 8-byte ZIP64 extra field
+            e1.setExtra(createOpaqueExtra((short) Long.BYTES));


Hello Eirik, I couldn't understand why we first add a opaque extra field first and then update it to be a zip64 extra field. Why do we do this?

Hello Eirik, I couldn't understand why we first add a opaque extra field first and then update it to be a zip64 extra field. Why do we do this?

ZipEntry.setExtra processes the byte array argument, looking for Zip64 extended fields which it can extract the size fields from. To prevent this parsing from happening, we temporarily use the unknown tag.

In this particular case, ZipExtra.setExtra actually ends up skipping this processing (because isLOC == true and it has a guard for the block size being >= 16).

However, I prefer the test to not depend too much on the details of setExtra Zip64 processing. This trick is used in other tests as well and may be copied over to a test where the conditions are not the same.

I have refactored a bit and added some code comments to help explain the use of the 'unknown' tag.

Do you think this makes sense?

Hello Eirik, thank you for that detail. Yes, what you note and the updated comment, looks good to me.

…revent ZipEntry.setExtra0 from processing the extra field.

LanceAndersen

thank you for the latest updates Eirik.

Overall it is good a few more comment suggestions for extra clarity.

LanceAndersen · 2024-02-09T11:55:16Z

test/jdk/java/util/zip/ZipFile/Zip64SizeTest.java

+            ZipEntry e1 = new ZipEntry("first");
+
+            // Make an extra field with the correct size for an 8-byte 'uncompressed size'
+            // Zip64 field. Temporarily use the 'unknown' tag 0x9902 to make


I would suggest adding a reference to the following in the APPNOTE.TXT to make clearer where that value came from

4.6.1 Third party mappings commonly used are:

another suggestion would be to show the CEN here with this change I think would make it easier for someone who is not as familiar with APPNOTE.TXT

The CEN structure is already descibed in the method-level documentation. Would you prefer I move it into the code? Not sure I would like to repeat it, and I think perhaps it's better to have the whole structure in once place, instead of splitting it (since the Zip64 extra corresponds to the CEN header fields).

What do you think?

I would suggest adding a reference to the following in the APPNOTE.TXT to make clearer where that value came from

Added a reference to "Third Party Mappings". (It's not super-important that we chose 'unknown' here, it just seemed a reasonable choice for something ZipEntry cannot see into)

LanceAndersen · 2024-02-09T11:59:13Z

test/jdk/java/util/zip/ZipFile/Zip64SizeTest.java

-                }
-            }
-        }
+    private static void updateCENHeaderToZip64(byte[] zip) {


Again minor but perhaps articulate that after this call, the extra data will be(obvious to you and I but probably not to others...)

00B4 Extra ID Azure Pipelines continuous integration (without running tests, for now) #1 0001 'ZIP64'

00B6 Length 0008

00B8 Uncompressed Size 0000000000000005

Again minor but perhaps articulate that after this call, the extra data will be(obvious to you and I but probably not to others...)

Added zipdetails describing the resulting ZIP64 structure to the updateCENHeaderToZip64 method documentation.

openjdk · 2024-02-09T12:05:27Z

@eirbjo This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8303891: Speed up Zip64SizeTest using a small ZIP64 file
8259866: two java.util tests failed with "IOException: There is not enough space on the disk"

Reviewed-by: lancea, jpai

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 115 new commits pushed to the master branch:

c266800: 8325558: Add jcheck whitespace checking for properties files
ec20b0a: 8325626: Allow selection of non-matching configurations using CONF=!string
618af39: 8325633: Use stricter assertion in callers of Space::is_aligned
5dbf137: 8319797: Recursive lightweight locking: Runtime implementation
4513da9: 8325470: [AIX] use fclose after fopen in read_psinfo
62a4be0: 8325635: Serial: Inline verify_used_region_at_save_marks
b356fee: 8325458: Rename mlvmJvmtiUtils.h
2ed889b: 8323628: Update license on "pass-through" files
7c69712: 8325570: Update to Graphviz 9.0.0
482c100: 8322865: JavaDoc fails on aggregator modules
... and 105 more: https://git.openjdk.org/jdk/compare/692c9f8821e220560927dd6bbedfea9ddfe312f6...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

… Mappings)

jaikiran

These test-only changes look good to me.

eirbjo · 2024-02-13T16:17:49Z

/integrate

openjdk · 2024-02-13T16:18:51Z

Going to push as commit 842b895.
Since your change was applied there have been 127 commits pushed to the master branch:

243fb46: 8325750: Fix spelling of ForceTranslateFailure help message
74b90aa: 8325672: C2: allocate PhaseIdealLoop::_loop_or_ctrl from C->comp_arena()
6dfa7f3: 8325541: C2 SuperWord: refactor filter / split
6b7c971: 8325382: (fc) FileChannel.transferTo throws IOException when position equals size
13d9e8f: 8325590: Regression in round-tripping UTF-16 strings after JDK-8311906
57b04e1: 8325748: Serial: Move Generation::promote to TenuredGeneration
7cd25ed: 8322854: Incorrect rematerialization of scalar replaced objects in C2
7ec2bad: 8323520: Drop unnecessary virtual specifier in Space
71ff2d7: 8325616: JFR ZGC Allocation Stall events should record stack traces
c3c1cdd: 8325731: Installation instructions for Debian/Ubuntu don't mention autoconf
... and 117 more: https://git.openjdk.org/jdk/compare/692c9f8821e220560927dd6bbedfea9ddfe312f6...master

Your commit was automatically rebased without conflicts.

openjdk · 2024-02-13T16:18:58Z

@eirbjo Pushed as commit 842b895.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Rewrite Zip64SizeTest to write to a sparse file

d530607

eirbjo added 2 commits March 9, 2023 13:10

Remove double whitespace

2acf085

Remove introduced blank line

9b97bea

openjdk bot added the core-libs core-libs-dev@openjdk.org label Mar 9, 2023

eirbjo changed the title ~~8303891: Zip64SizeTest should use a sparse file~~ 8303891: Zip64SizeTest could use a sparse file Mar 9, 2023

Add a missing "when" in Javadocs for SparseOutputStream

7f4643a

eirbjo marked this pull request as ready for review March 9, 2023 12:14

openjdk bot added the rfr Pull request is ready for review label Mar 9, 2023

eirbjo added 4 commits March 9, 2023 14:57

Make test method public

d3a8d78

Merge branch 'master' into zip64-size-test-sparse

883f543

Merge branch 'master' into zip64-size-test-sparse

ef22460

Sparse files must be created explicitly on NTFS

adcca52

eirbjo mentioned this pull request Mar 10, 2023

8303972: (zipfs) Make test/jdk/jdk/nio/zipfs/TestLocOffsetFromZip64EF.java independent of the zip command line #12979

Closed

3 tasks

bridgekeeper bot closed this May 6, 2023

eirbjo changed the title ~~8303891: Zip64SizeTest could use a sparse file~~ 8303891: Speed up Zip64SizeTest using a sparse file Dec 28, 2023

eirbjo added 5 commits December 28, 2023 14:14

Merge branch 'master' into zip64-size-test-sparse

6e50061

Merge branch 'master' into zip64-size-test-sparse

2aee964

Use ENTRY instead of FILE when refering to names and sizes of file en…

8920abc

…tries in the ZIP file

Use a small ZIP64 file to reproduce the issue. Convert test to JUnit

41b2ba5

eirbjo changed the title ~~8303891: Speed up Zip64SizeTest using a sparse file~~ 8303891: Speed up Zip64SizeTest using a small ZIP64 file Feb 8, 2024

openjdk bot reopened this Feb 8, 2024

jaikiran reviewed Feb 9, 2024

View reviewed changes

test/jdk/java/util/zip/ZipFile/Zip64SizeTest.java Outdated Show resolved Hide resolved

jaikiran reviewed Feb 9, 2024

View reviewed changes

eirbjo added 2 commits February 9, 2024 11:28

Add comments to explain how we temporarily use the 'unknown' tag to p…

a5ad0c4

…revent ZipEntry.setExtra0 from processing the extra field.

Spell fix for "Contents"

34333c9

LanceAndersen approved these changes Feb 9, 2024

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Feb 9, 2024

eirbjo added 3 commits February 9, 2024 13:13

Add reference to APPNOTE.TXT for the 'unknown tag' (4.6.1 Third Party…

a6cc978

… Mappings)

Add zipdetails of the Zip64 extra field in updateCENHeaderToZip64

3264a4d

Use three dots consistently for representing an ellipsis

9f843fc

LanceAndersen approved these changes Feb 9, 2024

View reviewed changes

jaikiran approved these changes Feb 12, 2024

View reviewed changes

openjdk bot added the integrated Pull request has been integrated label Feb 13, 2024

openjdk bot closed this Feb 13, 2024

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Feb 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8303891: Speed up Zip64SizeTest using a small ZIP64 file #12948

8303891: Speed up Zip64SizeTest using a small ZIP64 file #12948

eirbjo commented Mar 9, 2023 •

edited by openjdk bot

Loading

bridgekeeper bot commented Mar 9, 2023

openjdk bot commented Mar 9, 2023

mlbridge bot commented Mar 9, 2023 •

edited

Loading

bridgekeeper bot commented Apr 7, 2023

bridgekeeper bot commented May 6, 2023

eirbjo commented Feb 8, 2024

openjdk bot commented Feb 8, 2024

eirbjo commented Feb 8, 2024

eirbjo commented Feb 8, 2024

openjdk bot commented Feb 8, 2024

jaikiran Feb 9, 2024

eirbjo Feb 9, 2024

jaikiran Feb 12, 2024

LanceAndersen left a comment

LanceAndersen Feb 9, 2024

eirbjo Feb 9, 2024

eirbjo Feb 9, 2024

LanceAndersen Feb 9, 2024

eirbjo Feb 9, 2024

openjdk bot commented Feb 9, 2024 •

edited

Loading

jaikiran left a comment

eirbjo commented Feb 13, 2024

openjdk bot commented Feb 13, 2024

openjdk bot commented Feb 13, 2024

8303891: Speed up Zip64SizeTest using a small ZIP64 file #12948

8303891: Speed up Zip64SizeTest using a small ZIP64 file #12948

Conversation

eirbjo commented Mar 9, 2023 • edited by openjdk bot Loading

Progress

Issues

Reviewers

Reviewing

Webrev

bridgekeeper bot commented Mar 9, 2023

openjdk bot commented Mar 9, 2023

mlbridge bot commented Mar 9, 2023 • edited Loading

Webrevs

bridgekeeper bot commented Apr 7, 2023

bridgekeeper bot commented May 6, 2023

eirbjo commented Feb 8, 2024

openjdk bot commented Feb 8, 2024

eirbjo commented Feb 8, 2024

eirbjo commented Feb 8, 2024

openjdk bot commented Feb 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LanceAndersen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openjdk bot commented Feb 9, 2024 • edited Loading

jaikiran left a comment

Choose a reason for hiding this comment

eirbjo commented Feb 13, 2024

openjdk bot commented Feb 13, 2024

openjdk bot commented Feb 13, 2024

eirbjo commented Mar 9, 2023 •

edited by openjdk bot

Loading

mlbridge bot commented Mar 9, 2023 •

edited

Loading

openjdk bot commented Feb 9, 2024 •

edited

Loading