8272297: FileInputStream should override transferTo() for better performance #5097

bplb · 2021-08-12T01:16:04Z

Please consider this request to add an override java.io.FileInputStream.transferTo(OutputStream) with improved performance if the parameter is a FileOutputStream.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8272297: FileInputStream should override transferTo() for better performance

Reviewers

Alan Bateman (@AlanBateman - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/5097/head:pull/5097
$ git checkout pull/5097

Update a local copy of the PR:
$ git checkout pull/5097
$ git pull https://git.openjdk.java.net/jdk pull/5097/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 5097

View PR using the GUI difftool:
$ git pr show -t 5097

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/5097.diff

…ormance

bridgekeeper · 2021-08-12T01:16:36Z

👋 Welcome back bpb! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

bplb · 2021-08-12T01:16:48Z

Invoking transferTo() on a FileInputStream will use the superclass method which copies using looping over buffers of a fixed size. If the parameter is a FileOutputStream, the copy is greatly accelerated by using the java.nio.channels.FileChannel.transferTo() method with the FileChannels of the source and destination streams. Performance measurements were made for streams of length 100000, 1000000, 100000000, and 1000000000 bytes. Throughput improvement was observed for all cases on the three usual platforms, with the largest of nearly 5X being seen on Linux because in this case the sendfile() system call is used underneath. In no case did throughput decrease.

openjdk · 2021-08-12T01:19:15Z

@bplb The following label will be automatically applied to this pull request:

core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-08-12T01:22:40Z

Webrevs

src/java.base/share/classes/java/io/FileInputStream.java

bplb · 2021-08-12T23:33:44Z

Without the source change the test fails, but passes with the change in place.

src/java.base/share/classes/java/io/FileInputStream.java

rkennke · 2021-08-13T14:32:41Z

Is this what you have been asking @mkarg in #4263 to do? Optimize transferTo() only for FileInputStream? Would it interfere with #4263?

bplb · 2021-08-13T14:38:19Z

There is no overlap with #4263.

AlanBateman · 2021-08-13T14:41:54Z

Is this what you have been asking @mkarg in #4263 to do? Optimize transferTo() only for FileInputStream? Would it interfere with #4263?

#4263 is the input stream returned by Channels.newInputStream where the source may be a FileChannel and the output stream specified to InputStream::transferTo may be connected to a FileChannel.

src/java.base/share/classes/java/io/FileInputStream.java

AlanBateman · 2021-08-13T19:23:11Z

src/java.base/share/classes/java/io/FileInputStream.java

+            }
+        }
+        return transferred + super.transferTo(out);
+    }


Good, I think this version is right.

openjdk · 2021-08-15T16:20:55Z

@bplb This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8272297: FileInputStream should override transferTo() for better performance

Reviewed-by: alanb

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 84 new commits pushed to the master branch:

17b9350: 8266079: Lanai: AlphaComposite shows differences on Metal compared to OpenGL
87d2761: 8271883: Math CopySign optimization for x86
6b8b160: 8272396: mismatching debug output streams
0af645a: 8205138: Remove Applet references from Font2DTest
bd7f9b4: 8272459: ProblemList compiler/codecache/TestStressCodeBuffers.java on aarch64
717792c: 8263940: NPE when creating default file system when default file system provider is packaged as JAR file on class path
d06d0b9: 8272112: Arena code simplifications
0c4be76: 8058704: Nimbus does not honor JTextPane background color
020aec5: 8271366: [REDO] JDK-8266054 VectorAPI rotate operation optimization
4d4ba5c: 8272116: Update PerfDisableSharedMem with FLAG_SET_ERGO in PerfMemory::create_memory_region
... and 74 more: https://git.openjdk.java.net/jdk/compare/3ab95d19174100f440e2274d5b2c3119fb55e9e1...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

rkennke · 2021-08-15T16:51:59Z

Is this what you have been asking @mkarg in #4263 to do? Optimize transferTo() only for FileInputStream? Would it interfere with #4263?

#4263 is the input stream returned by Channels.newInputStream where the source may be a FileChannel and the output stream specified to InputStream::transferTo may be connected to a FileChannel.

Having followed both #4263 and this PR from the sidelines, may I ask why for #4263 much more rigorous testing is asked but not here? E.g. test for NPE, random-sized files/buffers, random position, return value, comprehensive JMH tests to show performance, etc? I'm all for high quality standards and good test coverage, but why are we seemingly measuring with double standards?

AlanBateman · 2021-08-16T08:00:48Z

Having followed both #4263 and this PR from the sidelines, may I ask why for #4263 much more rigorous testing is asked but not here? E.g. test for NPE, random-sized files/buffers, random position, return value, comprehensive JMH tests to show performance, etc? I'm all for high quality standards and good test coverage, but why are we seemingly measuring with double standards?

I don't think your comment is fair. The FIS PR is modest, covering one scenario, it's easy to review. The serious bugs that we pointed out have been addressed and test coverage has been added. The Channels PR is much more ambitious and is trying to cover a matrix of scenarios involving file channel and selectable channels, lots of corner cases and potential issues, and will require a lot of review cycles. None of the scenarios had any test coverage initially. This has been partially addressed but the proposal is still unwieldy, hence the discussion on reducing the scope to something manageable.

bplb · 2021-08-16T15:52:19Z

/integrate

openjdk · 2021-08-16T15:53:47Z

Going to push as commit 8268825.
Since your change was applied there have been 94 commits pushed to the master branch:

3677734: 8271471: [IR Framework] Rare occurrence of "" in PrintIdeal/PrintOptoAssembly can let tests fail
0a03481: 8272231: G1: Refactor G1CardSet::get_card_set to return G1CardSetHashTableValue*
83d0e12: 8267833: Improve G1CardSetInlinePtr::add()
69cc588: 8272235: G1: update outdated code root fixup
5db36ce: 8272158: SoftReference related bugs under memory pressure
7a5b37b: 8272439: G1: add documentation to G1CardSetInlinePtr
0209d9f: 8272461: G1: remove empty declaration of cleanup_after_scan_heap_roots
36e2dda: 8272348: Update CDS tests in anticipation of JDK-8270489
3f38a50: 8271203: C2: assert(iff->Opcode() == Op_If || iff->Opcode() == Op_CountedLoopEnd || iff->Opcode() == Op_RangeCheck) failed: Check this code when new subtype is added
6a5241c: 8272491: Problem list javax/swing/JFrame/NSTexturedJFrame/NSTexturedJFrame.java on macos
... and 84 more: https://git.openjdk.java.net/jdk/compare/3ab95d19174100f440e2274d5b2c3119fb55e9e1...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-08-16T15:54:01Z

@bplb Pushed as commit 8268825.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8272297: FileInputStream should override transferTo() for better perf…

54f1455

…ormance

openjdk bot added the rfr Pull request is ready for review label Aug 12, 2021

openjdk bot added the core-libs core-libs-dev@openjdk.org label Aug 12, 2021

AlanBateman suggested changes Aug 12, 2021

View reviewed changes

src/java.base/share/classes/java/io/FileInputStream.java Outdated Show resolved Hide resolved

8272297: Set source position after FC.transferTo(); add test

a199fc6

AlanBateman reviewed Aug 13, 2021

View reviewed changes

src/java.base/share/classes/java/io/FileInputStream.java Outdated Show resolved Hide resolved

AlanBateman reviewed Aug 13, 2021

View reviewed changes

src/java.base/share/classes/java/io/FileInputStream.java Outdated Show resolved Hide resolved

8272297: Improve growing source and partial transfer cases

796248e

AlanBateman reviewed Aug 13, 2021

View reviewed changes

src/java.base/share/classes/java/io/FileInputStream.java Outdated Show resolved Hide resolved

8272297: Remove unneeded check of position and couunt

3e575ba

AlanBateman reviewed Aug 13, 2021

View reviewed changes

AlanBateman approved these changes Aug 15, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Aug 15, 2021

openjdk bot closed this Aug 16, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Aug 16, 2021

bplb deleted the FileInputStream-transferTo-8272297 branch August 16, 2021 17:28

This was referenced Aug 19, 2021

8276779: (ch) InputStream returned by Channels.newInputStream should have fast path for SelectableChannels #5179

Closed

JDK-8273038: ChannelInputStream.transferTo() uses FileChannel.transferTo(FileChannel) #5209

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8272297: FileInputStream should override transferTo() for better performance #5097

8272297: FileInputStream should override transferTo() for better performance #5097

bplb commented Aug 12, 2021 •

edited by openjdk bot

bridgekeeper bot commented Aug 12, 2021

bplb commented Aug 12, 2021

openjdk bot commented Aug 12, 2021

mlbridge bot commented Aug 12, 2021 •

edited

bplb commented Aug 12, 2021

rkennke commented Aug 13, 2021

bplb commented Aug 13, 2021

AlanBateman commented Aug 13, 2021

AlanBateman Aug 13, 2021

openjdk bot commented Aug 15, 2021

rkennke commented Aug 15, 2021

AlanBateman commented Aug 16, 2021

bplb commented Aug 16, 2021

openjdk bot commented Aug 16, 2021

openjdk bot commented Aug 16, 2021

8272297: FileInputStream should override transferTo() for better performance #5097

8272297: FileInputStream should override transferTo() for better performance #5097

Conversation

bplb commented Aug 12, 2021 • edited by openjdk bot

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Aug 12, 2021

bplb commented Aug 12, 2021

openjdk bot commented Aug 12, 2021

mlbridge bot commented Aug 12, 2021 • edited

Webrevs

bplb commented Aug 12, 2021

rkennke commented Aug 13, 2021

bplb commented Aug 13, 2021

AlanBateman commented Aug 13, 2021

AlanBateman Aug 13, 2021

Choose a reason for hiding this comment

openjdk bot commented Aug 15, 2021

rkennke commented Aug 15, 2021

AlanBateman commented Aug 16, 2021

bplb commented Aug 16, 2021

openjdk bot commented Aug 16, 2021

openjdk bot commented Aug 16, 2021

bplb commented Aug 12, 2021 •

edited by openjdk bot

mlbridge bot commented Aug 12, 2021 •

edited