Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8265175: (fs) Files.copy(Path,Path,CopyOption...) should use sendfile on Linux #3476

Closed
wants to merge 5 commits into from

Conversation

bplb
Copy link
Member

@bplb bplb commented Apr 13, 2021

Please consider this request to change the underlying native implementation of java.nio.file.Files.copy(Path,Path,CopyOption...) on Linux only to perform zero-copy via sendfile(2). The sendfile() system call is already used in java.nio.channels.FIleChannel.transferTo​(long,long,WritableByteChannel). It is intentionally not proposed to use sendfile() in the native macOS implementation as on macOS the function requires that the destination file descriptor is for a socket.

This change showed some performance improvement as measured by JMH.

Before: user-space buffers (read() + write())

Benchmark           (size)   Mode  Cnt      Score     Error  Units
FilesCopy.copy       10240  thrpt    5  39167.400 ± 683.887  ops/s
FilesCopy.copy       51200  thrpt    5  20782.622 ± 558.031  ops/s
FilesCopy.copy      102400  thrpt    5  13260.709 ± 176.673  ops/s
FilesCopy.copy      512000  thrpt    5   3171.837 ± 175.803  ops/s
FilesCopy.copy     1048568  thrpt    5   1654.253 ±  39.419  ops/s
FilesCopy.copy    10485760  thrpt    5    145.328 ±   7.192  ops/s
FilesCopy.copy   104857600  thrpt    5     12.440 ±   2.275  ops/s
FilesCopy.copy  1073741824  thrpt    5      1.073 ±   0.081  ops/s

After: zero-copy (sendfile())

Benchmark           (size)   Mode  Cnt      Score     Error  Units
FilesCopy.copy       10240  thrpt    5  40571.516 ± 548.977  ops/s
FilesCopy.copy       51200  thrpt    5  23993.334 ± 506.817  ops/s
FilesCopy.copy      102400  thrpt    5  15927.485 ± 309.081  ops/s
FilesCopy.copy      512000  thrpt    5   4207.129 ±  95.454  ops/s
FilesCopy.copy     1048568  thrpt    5   2147.046 ±  33.446  ops/s
FilesCopy.copy    10485760  thrpt    5    148.798 ±   1.329  ops/s
FilesCopy.copy   104857600  thrpt    5     14.541 ±   0.675  ops/s
FilesCopy.copy  1073741824  thrpt    5      1.270 ±   0.029  ops/s

Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issue

  • JDK-8265175: (fs) Files.copy(Path,Path,CopyOption...) should use sendfile on Linux

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3476/head:pull/3476
$ git checkout pull/3476

Update a local copy of the PR:
$ git checkout pull/3476
$ git pull https://git.openjdk.java.net/jdk pull/3476/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3476

View PR using the GUI difftool:
$ git pr show -t 3476

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3476.diff

@bridgekeeper
Copy link

@bridgekeeper bridgekeeper bot commented Apr 13, 2021

👋 Welcome back bpb! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk openjdk bot added the rfr label Apr 13, 2021
@openjdk
Copy link

@openjdk openjdk bot commented Apr 13, 2021

@bplb The following label will be automatically applied to this pull request:

  • nio

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the nio label Apr 13, 2021
@mlbridge
Copy link

@mlbridge mlbridge bot commented Apr 13, 2021

Webrevs

Copy link
Contributor

@AlanBateman AlanBateman left a comment

I agree it's worth exploring using sendfile64 as file-to-file was not available when this code was written.

Have you measured the user vs. kernel time when testing? I'm also interested to see if you've tried using a buffer size that is a LCM of the source and destination block sizes.

For large files, can you can what byte_sent returns? I'm wondering if the cancellable mechanism is effective with this implementation.

The call already has the file size so the call to fstat is not needed. I think I'd prefer to continue to read to EOF as that it is a bit friendly to files that are changing when being copied. I assume the current patch is problematic on 32-bit (use sendfile64 and off64_t to be consistent with FileChannelImpl). It will also need to handle the EINTR case.

@bplb
Copy link
Member Author

@bplb bplb commented Apr 14, 2021

Have you measured the user vs. kernel time when testing?

No.

I'm also interested to see if you've tried using a buffer size that is a LCM of the source and destination block sizes.

No, the buffer size was 8192 and both block sizes 4096, which seems to be the largest that my Linux version can handle.

For large files, can you can what byte_sent returns? I'm wondering if the cancellable mechanism is effective with this implementation.

For all files that I observed, bytes_sent was equal to the file size. Therefore I am also skeptical of the possibility of cancellation. From the man page:

       Note that a successful call to sendfile()
       may write fewer bytes than requested; the caller should be
       prepared to retry the call if there were unsent bytes.

The call already has the file size so the call to fstat is not needed.

OK.

I think I'd prefer to continue to read to EOF as that it is a bit friendly to files that are changing when being copied. I assume the current patch is problematic on 32-bit (use sendfile64 and off64_t to be consistent with FileChannelImpl). It will also need to handle the EINTR case.

I will revise it to address these three points.

@bplb bplb changed the title 8265175: (fs) Files.copy(Path,Path,CopyOption...) should use zero-copy on Linux (fs) Files.copy(Path,Path,CopyOption...) should use sendfile on Linux Apr 14, 2021
@openjdk openjdk bot removed the rfr label Apr 14, 2021
@bplb bplb force-pushed the Linux-zero-copy-8265175 branch from ded30fc to b180a4c Compare Apr 14, 2021
@bplb bplb changed the title (fs) Files.copy(Path,Path,CopyOption...) should use sendfile on Linux 8265175: (fs) Files.copy(Path,Path,CopyOption...) should use sendfile on Linux Apr 14, 2021
@openjdk openjdk bot added the rfr label Apr 14, 2021
@bplb
Copy link
Member Author

@bplb bplb commented Apr 14, 2021

From the notes on the man page for sendfile(2):

sendfile() will transfer at most 0x7ffff000 (2,147,479,552)
bytes, returning the number of bytes actually transferred.  (This
is true on both 32-bit and 64-bit systems.)

I verified this behavior on Ubuntu. For files larger than 2147479552, it transfers chunks of size 2147479552 except possibly for the last chunk in which case the number of remaining bytes is returned. At EOF, zero is returned. Thus for such large files the cancel mechanism should work.

@openjdk
Copy link

@openjdk openjdk bot commented Apr 14, 2021

⚠️ @bplb This pull request contains merges that bring in commits not present in the target repository. Since this is not a "merge style" pull request, these changes will be squashed when this pull request in integrated. If this is your intention, then please ignore this message. If you want to preserve the commit structure, you must change the title of this pull request to Merge <project>:<branch> where <project> is the name of another project in the OpenJDK organization (for example Merge jdk:master).

@bplb
Copy link
Member Author

@bplb bplb commented Apr 14, 2021

The commit eddfa30 was verified on 32- and 64-bit JDKs in Ubuntu 20.04.

if (bytes_sent == 0) {
return;
}
#endif
Copy link
Contributor

@AlanBateman AlanBateman Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This version looks much better but it missing the error checking. If bytes_sent is -1 then we should throw UnixException with the errno like the existing code.

What would you think about setting count to 1MB or 4MB so that there is at least some chance for cancellation when the file is huge.

Given that the new implementation will call sendfile64 until EOF then there is no reason to fall through to the old implementation, the old implementation goes into the #else ...

Copy link
Member Author

@bplb bplb Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing count to 1MB and 4MB instead of 2147479552 bytes somewhat surprisingly did not have a huge effect on the throughput for the two largest transfer sizes tested.

return;
}
} while (bytes_sent > 0);
#else
Copy link
Contributor

@AlanBateman AlanBateman Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think this right. I'll double check it on Friday.

Copy link
Member Author

@bplb bplb Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All right, thanks.

@openjdk
Copy link

@openjdk openjdk bot commented Apr 16, 2021

@bplb This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8265175: (fs) Files.copy(Path,Path,CopyOption...) should use sendfile on Linux

Reviewed-by: alanb

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 56 new commits pushed to the master branch:

  • 1d66a15: 8265330: G1: Fix comment in G1FullGCPrepareTask::G1CalculatePointersClosure
  • b4ba74e: 8264987: G1: Fill BOTs for Survivor-turned-to-Old regions in full gc
  • fc89fe6: 8265119: G1: update_remset_before_rebuild mixes liveness in words with liveness in bytes
  • 50f3da8: 8264480: Unreachable code in nmethod.cpp inside #ifdef DEBUG
  • e0151a6: 8264104: Eliminate unnecessary vector mask conversion during VectorUnbox for floating point VectorMask
  • 64e2130: 8262108: SimpleDateFormat formatting broken for sq_MK Locale
  • 3423f3e: 8265180: JvmtiCompiledMethodLoadEvent should include the stub section of nmethods
  • f6e54f2: 8258794: Support for CLDR version 39
  • e89fd15: 8261301: StringWriter.flush() is NOOP but documentation does not indicate it
  • 0b1b5c8: 8264373: javac hangs when annotation is declared with sealed public modifier
  • ... and 46 more: https://git.openjdk.java.net/jdk/compare/e80012ede3564a868abc1705f332bcee942baf48...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready label Apr 16, 2021
@dmlloyd
Copy link
Contributor

@dmlloyd dmlloyd commented Apr 16, 2021

It might also be worth having a look at copy_file_range: https://man7.org/linux/man-pages/man2/copy_file_range.2.html

@bplb
Copy link
Member Author

@bplb bplb commented Apr 16, 2021

It might also be worth having a look at copy_file_range: https://man7.org/linux/man-pages/man2/copy_file_range.2.html

Thanks for the suggestion, but that requires a version of Linux too new for our production builds.

@bplb
Copy link
Member Author

@bplb bplb commented Apr 16, 2021

/integrate

@openjdk openjdk bot closed this Apr 16, 2021
@openjdk openjdk bot added the integrated label Apr 16, 2021
@openjdk openjdk bot removed ready rfr labels Apr 16, 2021
@openjdk
Copy link

@openjdk openjdk bot commented Apr 16, 2021

@bplb Since your change was applied there have been 64 commits pushed to the master branch:

  • cee4f1d: 8203925: tools/javac/importscope/T8193717.java ran out of java heap
  • 694e1cd: 8262060: compiler/whitebox/BlockingCompilation.java timed out
  • 6946d91: 8075915: The eight controls without black backgrounds with WinLAF & GTK LAF & Nimbus LAF
  • 714298a: 8265259: G1: Fix HeapRegion::block_is_obj for unloading class in full gc
  • ff5bb8c: 8265239: Shenandoah: Shenandoah heap region count could be off by 1
  • 17b6592: 8265335: Epsilon: Minor typo in EpsilonElasticTLABDecay description
  • 10ec38f: 8262462: IGV: cannot remove specific groups imported via network
  • 7137328: 8264958: C2 compilation fails with assert "n is later than its clone"
  • 1d66a15: 8265330: G1: Fix comment in G1FullGCPrepareTask::G1CalculatePointersClosure
  • b4ba74e: 8264987: G1: Fill BOTs for Survivor-turned-to-Old regions in full gc
  • ... and 54 more: https://git.openjdk.java.net/jdk/compare/e80012ede3564a868abc1705f332bcee942baf48...master

Your commit was automatically rebased without conflicts.

Pushed as commit 1c3fd46.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

@bplb bplb deleted the Linux-zero-copy-8265175 branch Apr 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integrated nio
3 participants