8266589: (fs) Improve performance of Files.copy() on macOS using copyfile(3) #3890

bplb · 2021-05-05T20:51:59Z

Please consider this request to use fcopyfile(3) to copy files via java.nio.file.Files(Path,Path,CopyOption...) on macOS. This change would improve the throughput of Files.copy() as shown in these results:

Copy via 8192 byte buffers

Benchmark           (size)   Mode  Cnt     Score     Error  Units
FilesCopy.copy       10240  thrpt    5  5432.671 ± 137.114  ops/s
FilesCopy.copy       51200  thrpt    5  3959.145 ± 157.467  ops/s
FilesCopy.copy      102400  thrpt    5  2931.325 ± 109.200  ops/s
FilesCopy.copy      512000  thrpt    5   655.550 ±  39.919  ops/s
FilesCopy.copy     1048568  thrpt    5   375.127 ±  10.111  ops/s
FilesCopy.copy    10485760  thrpt    5    64.048 ±   5.740  ops/s
FilesCopy.copy   104857600  thrpt    5     6.893 ±   0.415  ops/s
FilesCopy.copy  1073741824  thrpt    5     0.717 ±   0.026  ops/s

Copy via fcopyfile


Benchmark           (size)   Mode  Cnt     Score     Error  Units
FilesCopy.copy       10240  thrpt    5  5070.255 ± 817.419  ops/s
FilesCopy.copy       51200  thrpt    5  4469.218 ±  65.634  ops/s
FilesCopy.copy      102400  thrpt    5  3997.718 ± 301.440  ops/s
FilesCopy.copy      512000  thrpt    5  2064.223 ±  68.893  ops/s
FilesCopy.copy     1048568  thrpt    5   817.743 ±  83.076  ops/s
FilesCopy.copy    10485760  thrpt    5   145.799 ±   3.674  ops/s
FilesCopy.copy   104857600  thrpt    5    24.706 ±  14.178  ops/s
FilesCopy.copy  1073741824  thrpt    5     1.386 ±   0.073  ops/s

The smallest size tested shows a small degradation in throughput, but this could be rectified if desired by adding an empirically determined size threshold under which user-space buffers would be used instead of fcopyfile().

A small change is also made to the Linux sendfile() portion to use the largest permitted count value if the copy is uninterruptible, i.e., cancel == NULL.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8266589: (fs) Improve performance of Files.copy() on macOS using copyfile(3)

Reviewers

Alan Bateman (@AlanBateman - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3890/head:pull/3890
$ git checkout pull/3890

Update a local copy of the PR:
$ git checkout pull/3890
$ git pull https://git.openjdk.java.net/jdk pull/3890/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3890

View PR using the GUI difftool:
$ git pr show -t 3890

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3890.diff

…file(3)

bridgekeeper · 2021-05-05T20:52:42Z

👋 Welcome back bpb! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-05-05T20:55:10Z

@bplb The following label will be automatically applied to this pull request:

nio

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-05-05T20:59:11Z

Webrevs

01: Full - Incremental (ff2d2f9b)
00: Full (84708f5c)

AlanBateman · 2021-05-06T13:19:18Z

src/java.base/unix/native/libnio/fs/UnixCopyFile.c

+    }
+    return COPYFILE_CONTINUE;
+}
+#endif


I wasn't aware of this fcopyfile but it seems to be a good match. At some point I think we will have to split up UnixCopyFile to make it easier to maintain the different implementations. That might be the opportunity to do more than COPYFILE_DATA, meaning we could get it to copy the meta data too.

I think we should try to re-format the callback a bit to make it easier to distinguish the conditions in the if-statement from the body. I probably should re-format the declaration too to get it a more more consistent/readable.

I agree that it would be good to split it up but as part of a yet to be filed issue.

Would you please elaborate on reformatting the callback? Thanks.

I should have been clearer. L58 splits the conditions on two lines with the second line indented 4 spaces so it looks like the 3rd condition is in the block. I think you should re-format it to make it easier to read.

This could be made into one line, no split, or three lines. Otherwise an additional if would be needed.

AlanBateman

I see the re-examination of the callback means that COPYFILE_ERR is handled now. Good!

openjdk · 2021-05-07T05:17:21Z

@bplb This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8266589: (fs) Improve performance of Files.copy() on macOS using copyfile(3)

Reviewed-by: alanb

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 90 new commits pushed to the master branch:

ebb68d2: 8049700: Deprecate obsolete classes and methods in javax/swing/plaf/basic
3a474d9: 8265612: revise the help info for jmap histo command
c97f56c: 8266172: -Wstringop-overflow happens in vmError.cpp
43ad24f: 8265465: jcmd VM.cds should keep already dumped archive when exception happens
66191ff: 8266193: BasicJMapTest does not include testHistoParallel methods
36e5ad6: 8263236: runtime/os/TestTracePageSizes.java fails on old kernels
0ca86da: 8266002: vmTestbase/nsk/jvmti/ClassPrepare/classprep001 should skip events for unexpected classes
52f1db6: 8262002: java/lang/instrument/VerifyLocalVariableTableOnRetransformTest.sh failed with "TestCaseScaffoldException: DummyClassWithLVT did not match .class file"
04f7112: 8266293: Key protection using PBEWithMD5AndDES fails with "java.security.InvalidAlgorithmParameterException: Salt must be 8 bytes long"
a90b33a: 8266573: Make sure blackholes are tagged for all JVMCI paths
... and 80 more: https://git.openjdk.java.net/jdk/compare/0544a732a44309bf7cbb44846dd9320c6096de17...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

bplb · 2021-05-07T15:22:32Z

/integrate

openjdk · 2021-05-07T15:24:06Z

@bplb Since your change was applied there have been 103 commits pushed to the master branch:

947d69d: 8265042: javadoc HTML files not generated for types nested in records
946b0fe: 8266645: javac should not check for sealed supertypes in intersection types
74fecc0: 8266503: [UL] Make Decorations safely copy-able and reduce their size
86b8dc9: 8265426: Update java.security to use instanceof pattern variable
3fcdc50: 8266646: Add more diagnostic to java/lang/System/LoggerFinder/modules
9a19a0c: 8264760: JVM crashes when two threads encounter the same resolution error
14f0afe: 8214237: Join parallel phases post evacuation
2798b0d: 8266349: Pass down requested page size to reserve_memory_special
e0c8688: 8262992: Improve @see output
d2b5350: 8263507: Improve structure of package summary pages
... and 93 more: https://git.openjdk.java.net/jdk/compare/0544a732a44309bf7cbb44846dd9320c6096de17...master

Your commit was automatically rebased without conflicts.

Pushed as commit b5b3119.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8266589: (fs) Improve performance of Files.copy() on macOS using copy…

84708f5

…file(3)

openjdk bot added the rfr Pull request is ready for review label May 5, 2021

openjdk bot added the nio nio-dev@openjdk.org label May 5, 2021

AlanBateman reviewed May 6, 2021

View reviewed changes

8266589: Handle COPYFILE_ERR stage in fcopyfile_callback()

ff2d2f9

AlanBateman approved these changes May 7, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label May 7, 2021

openjdk bot closed this May 7, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels May 7, 2021

bplb deleted the macos-copyfile-8266589 branch May 7, 2021 19:12

8266589: (fs) Improve performance of Files.copy() on macOS using copyfile(3) #3890

8266589: (fs) Improve performance of Files.copy() on macOS using copyfile(3) #3890

bplb commented May 5, 2021 •

edited by openjdk bot

bridgekeeper bot commented May 5, 2021

openjdk bot commented May 5, 2021

mlbridge bot commented May 5, 2021 •

edited

AlanBateman May 6, 2021

bplb May 6, 2021

AlanBateman May 6, 2021

bplb May 6, 2021

AlanBateman left a comment

openjdk bot commented May 7, 2021 •

edited

bplb commented May 7, 2021

openjdk bot commented May 7, 2021

8266589: (fs) Improve performance of Files.copy() on macOS using copyfile(3) #3890

8266589: (fs) Improve performance of Files.copy() on macOS using copyfile(3) #3890

Conversation

bplb commented May 5, 2021 • edited by openjdk bot

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented May 5, 2021

openjdk bot commented May 5, 2021

mlbridge bot commented May 5, 2021 • edited

Webrevs

AlanBateman May 6, 2021

Choose a reason for hiding this comment

bplb May 6, 2021

Choose a reason for hiding this comment

AlanBateman May 6, 2021

Choose a reason for hiding this comment

bplb May 6, 2021

Choose a reason for hiding this comment

AlanBateman left a comment

Choose a reason for hiding this comment

openjdk bot commented May 7, 2021 • edited

bplb commented May 7, 2021

openjdk bot commented May 7, 2021

bplb commented May 5, 2021 •

edited by openjdk bot

mlbridge bot commented May 5, 2021 •

edited

openjdk bot commented May 7, 2021 •

edited