Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8277087: ZipException: zip END header not found at ZipFile#Source.findEND #6380

Closed
wants to merge 2 commits into from

Conversation

mrserb
Copy link
Member

@mrserb mrserb commented Nov 13, 2021

The ZipOutputStream class may create bogus zip data which cannot be opened by the ZipFile. The root cause is how the comment field is stored by the ZipOutputStream. According to the zip specification, the comment field should not be longer than 0xFFFF bytes, and we try to validate the length of the comment, but unfortunately, we do this after the comment was assigned already. So if the application saves the comment based on the user's input and then gets an exception from the ZipOutputStream.setComment() it may assume that the comment is too long and it will be ignored, but it will be saved as-is to the file.

Please take a look at this refactoring, and note:

  • The comment field is assigned before the length check
  • The null comment is ignored

The current fix will move the length validation before being assigned and will use the null comment as an empty text. Note that the behavior of the null parameter is not specified in the method/class/package so we are free here to implement it in any way, any thoughts/suggestions on which is better?


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issue

  • JDK-8277087: ZipException: zip END header not found at ZipFile#Source.findEND

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/6380/head:pull/6380
$ git checkout pull/6380

Update a local copy of the PR:
$ git checkout pull/6380
$ git pull https://git.openjdk.java.net/jdk pull/6380/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 6380

View PR using the GUI difftool:
$ git pr show -t 6380

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/6380.diff

@bridgekeeper
Copy link

@bridgekeeper bridgekeeper bot commented Nov 13, 2021

👋 Welcome back serb! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk
Copy link

@openjdk openjdk bot commented Nov 13, 2021

@mrserb The following label will be automatically applied to this pull request:

  • core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the core-libs label Nov 13, 2021
@mrserb mrserb marked this pull request as ready for review Nov 14, 2021
@openjdk openjdk bot added the rfr label Nov 14, 2021
@mlbridge
Copy link

@mlbridge mlbridge bot commented Nov 14, 2021

Webrevs

Copy link
Contributor

@LanceAndersen LanceAndersen left a comment

Thank you for the proposed fix. I am curious as to how you encountered this as comments are rarely used with Zip files.

}
this.comment = bytes;
Copy link
Contributor

@AlanBateman AlanBateman Nov 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation change looks okay. I assume this regression slipped through due to lack of tests.

The method description doesn't make it clear that the comment can be null (ZipEntry.setComment has the same issue) so we should fix this while we are in the area, as a separate JBS of course as it will need a CSR to track the spec clarification.

Copy link
Contributor

@LanceAndersen LanceAndersen Nov 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ZipEntry::setComment indicates that the comment will be truncated if needed and ZipOutputStream takes care of this.

Perhaps writeEND() should also be updated to something like:
writeBytes(comment, 0, Math.min(comment.length, 0xffff))

Which is similar to what happens in writeCEN

Yes it would be nice to clarify that a null is accepted by setComment. When null is specified, the comment length is written as 0

Copy link
Contributor

@AlanBateman AlanBateman Nov 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it would be nice to clarify that a null is accepted by setComment. When null is specified, the comment length is written as 0

@mrserb Are you taking the spec clarification or should we just created another issue in JBS to track it?

Copy link
Member Author

@mrserb mrserb Nov 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to update the spec for the null value for this method, and probably others in a separate CR, since this fix could be backported to the early releases. Will create such CR after agreement.

Copy link
Member Author

@mrserb mrserb Nov 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have filed a separate issue: https://bugs.openjdk.java.net/browse/JDK-8277495

Copy link
Contributor

@LanceAndersen LanceAndersen left a comment

Thank you again for this patch.

There appears to be a similar test, open/test/jdk/java/util/zip/ZipFile/Comment.java, I think we probably want to fold your changes into the existing test and possibly convert to use TestNG. If you prefer to keep this test separate, the test should have expanded coverage to validate that a comment that is set can be successfully read back and the test should be renamed as it does more than just validate an Empty/null comment.

I am also thinking that we should change line 314 of ZipOutputStream::writeEND:

        if (comment != null) {            // zip file comment
            writeShort(comment.length);
            writeBytes(comment, 0, comment.length);
        } else {
            writeShort(0);
        }

To be:
writeBytes(comment, 0, 0, Math.min(comment.length, 0xffff));

Which is done when writing an entry comment out in writeCEN.

@mrserb
Copy link
Member Author

@mrserb mrserb commented Nov 17, 2021

There appears to be a similar test, open/test/jdk/java/util/zip/ZipFile/Comment.java, I think we probably want to fold your changes into the existing test and possibly convert to use TestNG.

I know that test, and I explicitly created a new one, since the old one covers the positive cases of reading the different comments from the data by the ZipFile including an empty comment. This one is different, it checks the different use-cases all of which cause to save the empty comment into the data.

If you prefer to keep this test separate, the test should have expanded coverage to validate that a comment that is set can be successfully read back and the test should be renamed as it does more than just validate an Empty/null comment.

It is already checked by the ZipFile test cases.

To be: writeBytes(comment, 0, 0, Math.min(comment.length, 0xffff));
Which is done when writing an entry comment out in writeCEN.

It has a different implementation because of different specifications, the writeCEN codepath specified to cut long comments and save the first part, this method specified an exception to be thrown if a comment is too long-> an empty comment is saved.

@LanceAndersen
Copy link
Contributor

@LanceAndersen LanceAndersen commented Nov 17, 2021

There appears to be a similar test, open/test/jdk/java/util/zip/ZipFile/Comment.java, I think we probably want to fold your changes into the existing test and possibly convert to use TestNG.

I know that test, and I explicitly created a new one, since the old one covers the positive cases of reading the different comments from the data by the ZipFile including an empty comment. This one is different, it checks the different use-cases all of which cause to save the empty comment into the data.

If you prefer to keep this test separate, the test should have expanded coverage to validate that a comment that is set can be successfully read back and the test should be renamed as it does more than just validate an Empty/null comment.

It is already checked by the ZipFile test cases.

Sorry if my point was not clear. I would prefer to have 1 test to exercise a Zip file comment vs have tests in multiple areas. Expanding the existing test in this case keeps the primary coverage in one location and makes it easier for future maintainers.

There is no reason to not enhance existing tests to add additional coverage.

To be: writeBytes(comment, 0, 0, Math.min(comment.length, 0xffff));
Which is done when writing an entry comment out in writeCEN.

It has a different implementation because of different specifications, the writeCEN codepath specified to cut long comments and save the first part, this method specified an exception to be thrown if a comment is too long-> an empty comment is saved.

You are talking about the difference between the javadoc for ZipOutputStream::setComment and ZipEntry::setComment and yes I am aware of the difference.

I am not suggesting to not make your change, I am suggesting to include this change as well.

@mrserb
Copy link
Member Author

@mrserb mrserb commented Nov 17, 2021

Sorry if my point was not clear. I would prefer to have 1 test to exercise a Zip file comment vs have tests in multiple areas. Expanding the existing test in this case keeps the primary coverage in one location and makes it easier for future maintainers.

I have preferred to have separate tests for the separate use cases, other than one big testcases which cover all possible combinations of exceptions and parameters.

I am not suggesting to not make your change, I am suggesting to include this change as well.

I understood that you suggest adding this additional change as well, and I pointed out why it is not necessary, if that code will be executed with the long comment it will break the specification.

@LanceAndersen
Copy link
Contributor

@LanceAndersen LanceAndersen commented Nov 17, 2021

Sorry if my point was not clear. I would prefer to have 1 test to exercise a Zip file comment vs have tests in multiple areas. Expanding the existing test in this case keeps the primary coverage in one location and makes it easier for future maintainers.

I have preferred to have separate tests for the separate use cases, other than one big testcases which cover all possible combinations of exceptions and parameters.

This is really a style choice so I guess we can agree to disagree. Either way it would be preferable to have both tests in the same location.

I am not suggesting to not make your change, I am suggesting to include this change as well.

I understood that you suggest adding this additional change as well, and I pointed out why it is not necessary, if that code will be executed with the long comment it will break the specification.

The additional change simply helps provide a layer of extra protection from corruption creating the zip file.

Anyways, I will approve the changes as they are.

@openjdk
Copy link

@openjdk openjdk bot commented Nov 17, 2021

@mrserb This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8277087: ZipException: zip END header not found at ZipFile#Source.findEND

Reviewed-by: lancea

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 45 new commits pushed to the master branch:

  • 262d070: 8277246: Check for NonRepudiation as well when validating a TSA certificate
  • a907b2b: 8276177: nsk/jvmti/RedefineClasses/StressRedefineWithoutBytecodeCorruption failed with "assert(def_ik->is_being_redefined()) failed: should be being redefined to get here"
  • b687664: 8277159: Fix java/nio/file/FileStore/Basic.java test by ignoring /run/user/* mount points
  • 8f5a8f7: 8264293: Create implementation for NSAccessibilityMenu protocol peer
  • 9f2f46e: 8275037: Test vmTestbase/nsk/sysdict/vm/stress/btree/btree011/btree011.java crashes with memory exhaustion on Windows
  • 2af9e59: 8276139: TestJpsHostName.java not reliable, better to expand HostIdentifierCreate.java test
  • e9934e1: 8277221: G1: Remove methods without implementations in G1CollectedHeap
  • 9aa30de: 8275317: AArch64: Support some type conversion vectorization in SLP
  • 08f65a5: 8277313: Validate header failed for test/jdk/java/net/httpclient/HeadTest.java
  • 23e5117: 8276559: (httpclient) Consider adding an HttpRequest.Builder.HEAD method to build a HEAD request.
  • ... and 35 more: https://git.openjdk.java.net/jdk/compare/176d21d6c525f8fd9592db5b4975308ea0001856...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready label Nov 17, 2021
@mrserb
Copy link
Member Author

@mrserb mrserb commented Nov 23, 2021

/integrate

@openjdk
Copy link

@openjdk openjdk bot commented Nov 23, 2021

Going to push as commit e3243ee.
Since your change was applied there have been 121 commits pushed to the master branch:

  • 12f08ba: 8277507: Add jlink.debug system property while launching jpackage tests to help diagonize recent intermittent failures
  • bb11c55: 8277542: G1: Move G1CardSetFreePool and related classes to separate files
  • f62b81c: 8273095: vmTestbase/vm/mlvm/anonloader/stress/oome/heap/Test.java fails with "wrong OOME"
  • 05a9a51: 8277423: ciReplay: hidden class with comment expected error
  • 1049aba: 8277576: ProblemList runtime/ErrorHandling/CreateCoredumpOnCrash.java on macosx-X64
  • 851a362: 8264297: Create implementation for NSAccessibilityProgressIndicator protocol peer
  • 6cf4cd1: 8273341: Update Siphash to version 1.0
  • e3911a8: 8277429: Conflicting jpackage static library name
  • 33e2a51: 8265795: vmTestbase/nsk/jvmti/AttachOnDemand/attach022/TestDescription.java fails when running with JEP 416
  • 22f12ac: 8277522: Make formatting of null consistent in Elements
  • ... and 111 more: https://git.openjdk.java.net/jdk/compare/176d21d6c525f8fd9592db5b4975308ea0001856...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot closed this Nov 23, 2021
@openjdk openjdk bot added integrated and removed ready rfr labels Nov 23, 2021
@openjdk
Copy link

@openjdk openjdk bot commented Nov 23, 2021

@mrserb Pushed as commit e3243ee.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

@mrserb mrserb deleted the JDK-8277087 branch Nov 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core-libs integrated
5 participants