Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8277069: [REDO] JDK-8276743 Make openjdk build Zip Archive generation "reproducible" #6673

Closed
wants to merge 3 commits into from

Conversation

magicus
Copy link
Member

@magicus magicus commented Dec 2, 2021

This is basically Andrew's old patch that was backed out, with a single change: I'm using ZipFile instead of ZipInputStream to read in the original zip. The latter is not capable of reading the extended attributes that contain the unix permissions. (Why this is so, is not fully clear to me. What's worse, it's by no means clear from the documentation. We should probably file a follow-up bug to improve the Javadoc for ZipInputStream, warning the users to stay away if they can.)

I have also added the possibility to opt-out of reproducible building by an argument to SetupZipArchive. This is used in the closed Oracle part of the build, where we create some temporary zip files that are later discarded, and not part of the deliverables. This saves us some unnecessary overhead.


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issue

  • JDK-8277069: [REDO] JDK-8276743 Make openjdk build Zip Archive generation "reproducible"

Reviewers

Contributors

  • Andrew Leonard <aleonard@openjdk.org>
  • Magnus Ihse Bursie <ihse@openjdk.org>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/6673/head:pull/6673
$ git checkout pull/6673

Update a local copy of the PR:
$ git checkout pull/6673
$ git pull https://git.openjdk.java.net/jdk pull/6673/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 6673

View PR using the GUI difftool:
$ git pr show -t 6673

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/6673.diff

@bridgekeeper
Copy link

bridgekeeper bot commented Dec 2, 2021

👋 Welcome back ihse! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk
Copy link

openjdk bot commented Dec 2, 2021

@magicus The following label will be automatically applied to this pull request:

  • build

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the build build-dev@openjdk.org label Dec 2, 2021
@magicus magicus marked this pull request as ready for review Dec 2, 2021
@openjdk openjdk bot added the rfr Pull request is ready for review label Dec 2, 2021
@mlbridge
Copy link

mlbridge bot commented Dec 2, 2021

Webrevs

@magicus
Copy link
Member Author

magicus commented Dec 2, 2021

I tried to find a way to make GitHub display the difference wrt the original, backed out, patch, but utterly failed. So here's a simple diff. This is the only way in which this patch differs from the original.

--- ./make/jdk/src/classes/build/tools/makezipreproducible/MakeZipReproducible.java	2021-12-02 19:57:38.000000000 +0100
+++ ../../jdk-ALT/open/make/jdk/src/classes/build/tools/makezipreproducible/MakeZipReproducible.java	2021-12-02 15:11:22.000000000 +0100
@@ -162,13 +162,8 @@

     // Process input zip file and add to sorted entries set
     boolean processInputEntries(File inFile) throws IOException {
-        try (FileInputStream fis = new FileInputStream(inFile);
-             ZipInputStream  zis = new ZipInputStream(fis)) {
-            ZipEntry entry;
-            while ((entry = zis.getNextEntry()) != null) {
-                entries.put(entry.getName(), entry);
-            }
-        }
+        ZipFile zipFile = new ZipFile(inFile);
+        zipFile.stream().forEach(entry -> entries.put(entry.getName(), entry));

         return true;
     }

@magicus
Copy link
Member Author

magicus commented Dec 2, 2021

/contributor add aleonard
/contributor add ihse

@openjdk
Copy link

openjdk bot commented Dec 2, 2021

@magicus
Contributor Andrew Leonard <aleonard@openjdk.org> successfully added.

@openjdk
Copy link

openjdk bot commented Dec 2, 2021

@magicus
Contributor Magnus Ihse Bursie <ihse@openjdk.org> successfully added.

erikj79
erikj79 approved these changes Dec 2, 2021
@openjdk
Copy link

openjdk bot commented Dec 2, 2021

@magicus This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8277069: [REDO] JDK-8276743 Make openjdk build Zip Archive generation "reproducible"

Co-authored-by: Andrew Leonard <aleonard@openjdk.org>
Co-authored-by: Magnus Ihse Bursie <ihse@openjdk.org>
Reviewed-by: erikj

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 12 new commits pushed to the master branch:

  • 4f15be2: 8277965: Enclosing instance optimization affects serialization
  • 8f196a2: 8278037: Clean up PPC32 related code in C1
  • 8b042d1: 8257856: Make ClassFileVersionsTest.java robust to JDK version updates
  • ad1ff27: 8277422: tools/jar/JarEntryTime.java fails with modified time mismatch
  • 7c4ef3a: 8277985: G1: Compare max_parallel_refinement_threads to UINT_MAX
  • dc2abc9: 8278080: Add --with-cacerts-src='user cacerts folder' to enable deterministic cacerts generation
  • 8d9cb2e: 8278078: Cannot reference super before supertype constructor has been called
  • 65960f7: 8277753: Long*VectorTests.java fail with "bad AD file" on x86_32 after JDK-8276162
  • 3889af3: 8277906: Incorrect type for IV phi of long counted loops after CCP
  • d2b16c8: 8276126: Dump time class transformation causes heap objects of non-boot classes to be archived
  • ... and 2 more: https://git.openjdk.java.net/jdk/compare/b79554bb5cef14590d427543a40efbcc60c66548...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready Pull request is ready to be integrated label Dec 2, 2021
@magicus
Copy link
Member Author

magicus commented Dec 2, 2021

FWIW, I've opened https://bugs.openjdk.java.net/browse/JDK-8278165 to at least get some documentation on the limitations of ZipInputStream.

@magicus
Copy link
Member Author

magicus commented Dec 2, 2021

/integrate

@openjdk
Copy link

openjdk bot commented Dec 2, 2021

Going to push as commit c93552c.
Since your change was applied there have been 20 commits pushed to the master branch:

  • b8ac0d2: 8276681: Additional malformed Javadoc inline tags in JDK source
  • b310f30: 8277396: [TESTBUG] In DefaultButtonModelCrashTest.java, frame is accessed from main thread
  • 15345e3: 8276661: (fs) UserDefinedFileAttributeView no longer works with long path (win)
  • 652b5f8: 8276674: Malformed Javadoc inline tags in JDK source
  • e0f1fc7: 8277358: Accelerate CRC32-C
  • 73a9654: 8276806: Use Objects.checkFromIndexSize where possible in java.base
  • 30087cc: 8275342: Change nested classes in java.prefs to static nested classes
  • 669c90e: 8278099: two sun/security/pkcs11/Signature tests failed with AssertionError
  • 4f15be2: 8277965: Enclosing instance optimization affects serialization
  • 8f196a2: 8278037: Clean up PPC32 related code in C1
  • ... and 10 more: https://git.openjdk.java.net/jdk/compare/b79554bb5cef14590d427543a40efbcc60c66548...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot closed this Dec 2, 2021
@openjdk openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Dec 2, 2021
@openjdk
Copy link

openjdk bot commented Dec 2, 2021

@magicus Pushed as commit c93552c.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

@magicus magicus deleted the redo-make-zip-reproducible branch Dec 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build build-dev@openjdk.org integrated Pull request has been integrated
2 participants