Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8080272 Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy #1853

Conversation

@turbanoff
Copy link
Member

@turbanoff turbanoff commented Dec 20, 2020


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issue

  • JDK-8080272: Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy

Reviewers

Download

$ git fetch https://git.openjdk.java.net/jdk pull/1853/head:pull/1853
$ git checkout pull/1853

@bridgekeeper
Copy link

@bridgekeeper bridgekeeper bot commented Dec 20, 2020

👋 Welcome back turbanoff! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk
Copy link

@openjdk openjdk bot commented Dec 20, 2020

@turbanoff The following labels will be automatically applied to this pull request:

  • 2d
  • awt
  • compiler
  • core-libs
  • hotspot-jfr
  • net
  • nio
  • security
  • serviceability
  • sound
  • swing

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

@mlbridge
Copy link

@mlbridge mlbridge bot commented Dec 20, 2020

Webrevs

@AlanBateman
Copy link
Contributor

@AlanBateman AlanBateman commented Dec 20, 2020

jrtfs is compiled twice, the second is to --release 8 so it can be packaged into jrt-fs.jar for use by IDEs/tools running on older JDK releases. So need to be careful with the changes here as it will likely causing build breakage.

We try to keep the changes to ASM to a minimum, might be better to leave this change out of the patch.

One or two of the sources changes should probably uses Files.copy, e.g. ZipPath, sjavac/CopyFile.java.

@turbanoff
Copy link
Member Author

@turbanoff turbanoff commented Dec 20, 2020

One or two of the sources changes should probably uses Files.copy, e.g. ZipPath, sjavac/CopyFile.java.

Good idea! Replaced in few places. But not in ZipPath: it's actually implementation of underlying call from Files.copy: jdk.nio.zipfs.ZipFileSystemProvider#copy. So, Files.copy call will be recursive.

@prrace
Copy link
Contributor

@prrace prrace commented Dec 20, 2020

So these changes are all over the place which means no one person knows how to test all of it.
Have you run the sound, swing tests, and the printing tests on unix and windows ?
For printing for sure you'll need to do some manual testing.

@mrserb
Copy link
Member

@mrserb mrserb commented Dec 20, 2020

I already suggested this, but anyway please always extract the changes to the java.desktop module to the separate PR.

@turbanoff turbanoff force-pushed the 8080272_use_transferTo_method_instead_of_manual_copy branch from ec602c1 to 6f8ec71 Dec 21, 2020
@turbanoff
Copy link
Member Author

@turbanoff turbanoff commented Dec 21, 2020

I've extracted changes in java.desktop into separate PR #1856
Reverted this changes from current PR.

@turbanoff turbanoff force-pushed the 8080272_use_transferTo_method_instead_of_manual_copy branch from 6f8ec71 to fceb20e Dec 21, 2020
@AlanBateman
Copy link
Contributor

@AlanBateman AlanBateman commented Dec 21, 2020

Probably best to drop the changes to src/java.xml.crypto/share/classes/com/sun/org/apache/xml/internal/security/** too as it gets refreshed periodically from the upstream Apache Santuario project.

Copy link
Contributor

@stsypanov stsypanov left a comment

I'm not a reviewer, but still think we could simplify sun.security.tools.keytool.Main

Copy link
Contributor

@mcimadamore mcimadamore left a comment

The changes to sjavac changes look good

@openjdk
Copy link

@openjdk openjdk bot commented Feb 15, 2021

@turbanoff This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8080272: Refactor I/O stream copying to use InputStream.transferTo/readAllBytes and Files.copy

Reviewed-by: mcimadamore, alanb

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 1076 new commits pushed to the master branch:

  • 0c718ab: 8262277: URLClassLoader.getResource throws undocumented IllegalArgumentException
  • 4f1cda4: 8263387: G1GarbageCollection JFR event gets gc phase, not gc type
  • 5ab5244: 8263514: Minor issue in JavacFileManager.SortFiles.REVERSE
  • 771b146: 8245025: MoveAndUpdateClosure::do_addr calls function with side-effects in an assert
  • 46d78f0: 6539707: (fc) MappedByteBuffer.force() method throws an IOException in a very simple test
  • 189289d: 8262326: MaxMetaspaceSize does not have to be aligned to metaspace commit alignment
  • d825198: 8263556: remove @modules java.base from tests
  • d6b5e18: 8263191: Consolidate ThreadInVMfromJavaNoAsyncException and ThreadBlockInVMWithDeadlockCheck with existing wrappers
  • 80cdf78: 8263544: Unused argument in ConstantPoolCacheEntry::set_field()
  • c0176c4: 8263552: Use String.valueOf() for char-to-String conversions
  • ... and 1066 more: https://git.openjdk.java.net/jdk/compare/fa50877c2e86d1a4e00724dd29d934f52d51f42c...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@wangweij, @mcimadamore, @AlanBateman) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

@openjdk openjdk bot added the ready label Feb 15, 2021
…ferTo

remove unnecessary file.exists() check
Copy link
Contributor

@AlanBateman AlanBateman left a comment

Thanks for perceiving with this one. I think you've addressed all issues in the latest revision.

@turbanoff
Copy link
Member Author

@turbanoff turbanoff commented Feb 16, 2021

/integrate

@openjdk openjdk bot added the sponsor label Feb 16, 2021
@openjdk
Copy link

@openjdk openjdk bot commented Feb 16, 2021

@turbanoff
Your change (at version 6e71e96) is now ready to be sponsored by a Committer.

@FrauBoes
Copy link
Member

@FrauBoes FrauBoes commented Feb 16, 2021

Hi @turbanoff, I'm happy to sponsor but I see two comments by @marschall - have they been addressed?
#1853 (comment)
#1853 (comment)

@openjdk openjdk bot removed the sponsor label Feb 16, 2021
@turbanoff
Copy link
Member Author

@turbanoff turbanoff commented Feb 18, 2021

@FrauBoes fixed in the last commit. Is there any way to de-integrate PR (to include last commit)?

@FrauBoes
Copy link
Member

@FrauBoes FrauBoes commented Feb 18, 2021

To re-integrate, just run the /integrate command again.

Before doing that, could someone from security libs acknowledge the changes in their area?

@@ -225,12 +224,7 @@ public X509CertPath(InputStream is, String encoding)
}

try {
if (is.markSupported() == false) {
Copy link
Member

@seanjmullan seanjmullan Feb 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you should remove lines 228-232. These methods are called by methods of CertificateFactory that take InputStream (which may contain a stream of security data) and they are designed such that they try to read one Certificate, CRL, or CertPath from the InputStream and leave the InputStream ready to parse the next structure instead of consuming all of the bytes. Thus they check if the InputStream supports mark in order to try to preserve that behavior. If mark is not supported, then it's ok to use InputStream.readAllBytes, otherwise, leave the stream as-is.

Copy link
Member Author

@turbanoff turbanoff Feb 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I can see only ByteArrayInputStream is actually passed in InputStream in current JDK code:

PKCS7 pkcs7 = new PKCS7(is.readAllBytes());
    private static List<X509Certificate> parsePKCS7(InputStream is)
        certs = parsePKCS7(is);
            public X509CertPath(InputStream is, String encoding)
                return new X509CertPath(new ByteArrayInputStream(data), encoding);

PKCS7 pkcs7 = new PKCS7(is.readAllBytes());
    private static List<X509Certificate> parsePKCS7(InputStream is)
        certs = parsePKCS7(is);
            public X509CertPath(InputStream is, String encoding)
                this(is, PKIPATH_ENCODING);
                    public X509CertPath(InputStream is) throws CertificateException {
                        return new X509CertPath(new ByteArrayInputStream(encoding));

изображение

Perhaps original marking approach was lost during refactoring?

Copy link
Member

@seanjmullan seanjmullan Feb 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, the code was refactored (way back in 2010) [1] to read one block at a time, so this check on mark can be removed. So, in this case, I think it is probably safe to just pass the InputStream as-is to PKCS7(InputStream), but maybe you can add a comment that says this should always be a ByteArrayInputStream. We can look at refactoring this code and clean it up a bit more later.

[1] https://hg.openjdk.java.net/jdk/jdk/rev/337ae296b6d6

Copy link
Member Author

@turbanoff turbanoff Mar 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find implementation of sun.security.pkcs.PKCS7#PKCS7(java.io.InputStream) a bit confusing (or even buggy). It uses only InputStream.available() to parse block.
So I would prefer to not use it.

@FrauBoes
Copy link
Member

@FrauBoes FrauBoes commented Mar 15, 2021

@turbanoff Given that this PR has been lingering for a while, you could drop the change in X509CertPath.java for now and integrate the remaining changes. I'm happy to sponsor in that case.

…dAllBytes and Files.copy

drop changes in X509CertPath
@FrauBoes
Copy link
Member

@FrauBoes FrauBoes commented Mar 15, 2021

@turbanoff Tier 1-3 still all clear. If you /integrate, I will sponsor this tomorrow.

@turbanoff
Copy link
Member Author

@turbanoff turbanoff commented Mar 15, 2021

/integrate

@openjdk openjdk bot added the sponsor label Mar 15, 2021
@openjdk
Copy link

@openjdk openjdk bot commented Mar 15, 2021

@turbanoff
Your change (at version 96920ee) is now ready to be sponsored by a Committer.

@FrauBoes
Copy link
Member

@FrauBoes FrauBoes commented Mar 16, 2021

/sponsor

@openjdk
Copy link

@openjdk openjdk bot commented Mar 16, 2021

@FrauBoes @turbanoff Since your change was applied there have been 1086 commits pushed to the master branch:

  • a31a23d: 8263595: Remove oop type punning in JavaCallArguments
  • a1f6591: 8263589: Introduce JavaValue::get_oop/set_oop
  • 20297a1: 8263577: C2: reachable nodes shouldn't have dead uses at the end of optimizations
  • c484d89: 8263557: Possible NULL dereference in Arena::destruct_contents()
  • ba35193: 8263559: Add missing initializers to VM_PopulateDumpSharedSpace
  • e03a594: 8262504: Some CLHSDB command cannot know they run on remote debugger
  • d896246: 8263420: Incorrect function name in NSAccessibilityStaticText native peer implementation
  • 8c1112a: 8261916: gtest/GTestWrapper.java vmErrorTest.unimplemented1_vm_assert failed
  • 1e57087: 8263392: Allow current thread to be specified in ExceptionMark
  • 4d1c08c: 8263616: 'Deprecatd' typo in src/hotspot/share/classfile/classFileParser.cpp
  • ... and 1076 more: https://git.openjdk.java.net/jdk/compare/fa50877c2e86d1a4e00724dd29d934f52d51f42c...master

Your commit was automatically rebased without conflicts.

Pushed as commit 68deb24.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

@turbanoff turbanoff deleted the 8080272_use_transferTo_method_instead_of_manual_copy branch Sep 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment