8274548: (fc) FileChannel gathering write fails with IOException "Invalid argument" on macOS 11.6 #5831

bplb · 2021-10-06T00:00:00Z

On macOS, handle the case where writev() is given an array of struct iovec the sum of whose iov_len fields overflows INT_MAX.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8274548: (fc) FileChannel gathering write fails with IOException "Invalid argument" on macOS 11.6

Reviewers

Alan Bateman (@AlanBateman - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/5831/head:pull/5831
$ git checkout pull/5831

Update a local copy of the PR:
$ git checkout pull/5831
$ git pull https://git.openjdk.java.net/jdk pull/5831/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 5831

View PR using the GUI difftool:
$ git pr show -t 5831

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/5831.diff

…alid argument" on macOS 11.6

bridgekeeper · 2021-10-06T00:00:21Z

👋 Welcome back bpb! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

bplb · 2021-10-06T00:01:36Z

The writev() function on Linux is specified to fail if:

     EINVAL The sum of the iov_len values in the iov array would
            overflow an ssize_t.

and similarly on macOS:

     [EINVAL] The sum of the iov_len values in the iov array
              overflows a 32-bit integer.

On Linux, writev() was observed not to fail for this case but to return 0x7ffff000, the maximum number of bytes that sendfile() can transfer. This of course might not be the case on all Linux variants.

The error could be handled by a code change in IOUtil but then it would affect all platforms. Also, on macOS the problem has only been observed on one version of the operating system so the added native code would not be always be called even though compiled. It would however probably be harmless to conditionally compile the code for Linux as well as macOS.

Testing has verified the fix on macOS 11.6 and earlier macOS versions with no effect (of course) on Linux.

openjdk · 2021-10-06T00:04:03Z

@bplb The following label will be automatically applied to this pull request:

nio

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-10-06T00:08:23Z

Webrevs

AlanBateman · 2021-10-06T06:52:38Z

We really need someone from Apple here to tell us if this is a man page issue or a bug in macOS 11.6. If I read it correctly, the issue doesn't occur with 10.5.x and the beta of macOS 12, is that correct? If we do need to put a workaround in for this then I think it will need an overhaul of IOUtil first. I'd prefer not have the iovec array be modified in two places.

bplb · 2021-10-06T19:26:20Z

V1 moves the logic up to the Java level and removes all native code changes.

mlbridge · 2021-10-06T19:49:40Z

Mailing list message from Brian Burkhalter on nio-dev:

On Oct 5, 2021, at 11:55 PM, Alan Bateman <alanb at openjdk.java.net<mailto:alanb at openjdk.java.net>> wrote:

If I read it correctly, the issue doesn't occur with 10.5.x and the best of macOS 12, is that correct?

That is correct: change in behavior was seen on 11.6 only; 10.5.x and 12-beta are identical and do not manifest it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/nio-dev/attachments/20211006/ce6f68a9/attachment.htm>

…olute put

AlanBateman · 2021-10-07T06:59:56Z

The latest version impacts all platforms. I think the first step here is to make contact with Apple and try to detect if this is an intended change in macOS 11.6 or not. The observation that they reverted it again in macOS 12 beta suggests it was unintentional but I think we need to find out more.

bplb · 2021-10-07T16:24:07Z

Not affecting all platforms is why it was in the C code in the first place.

Apple indicates it was an intentional change in macOS 11.x where x >= 0. I don't know yet about 12.

bplb · 2021-10-07T17:15:21Z

My original evaluation on macOS 12-beta was incorrect: I re-tested and the problem occurs there as well. Therefore we are seeing it on macOS x.y.z where x >= 11.

…array

AlanBateman · 2021-10-08T12:04:28Z

One approach is try to add a writevMax method along side of iovMax that returns the maximum value of the sum of the iov_len. If this is merged with your v2 then it might not be too bad.

bplb · 2021-10-08T18:47:22Z

v4 was verified to pass the test included in this PR on Linux-aarch64, Linux-x64, macOS 10.15.7 and 11.6, and Windows (Server 2016).

AlanBateman · 2021-10-09T07:17:55Z

src/java.base/unix/native/libnio/ch/IOUtil.c

+    // [EINVAL] The sum of the iov_len values in the iov array
+    //          overflows a 32-bit integer.
+    //
+    int darwin_version = get_darwin_version();


What you would think about dropping the OS version and just return Integer.MAX_VALUE on macOS? That would align with the EINVAL documented in the man page on all versions.

In passing, an inconsistency has kept into the native code. In some places we are using ifdef APPLE and in others ifdef MACOX. We should probably clean this up.

I'm find with dropping the OS version check on macOS and removing the Darwin version function.

I also noticed the inconsistency in the APPLE and MACOSX symbolic constants. There's also another one like ALL_BSD_SOURCE. I don't know whether we consider that redundant as well.

I'm find with dropping the OS version check on macOS and removing the Darwin version function.

I think it will align with the man page and remove any questions as to why this is macOS version specific.

I also noticed the inconsistency in the APPLE and MACOSX symbolic constants. There's also another one like ALL_BSD_SOURCE. I don't know whether we consider that redundant as well.

We can look at this in a separate issue.

... and dropping the OS version check leaves fewer moving parts.

On the conditional compilation symbolic constants I filed JDK-8275070.

…evMax()

openjdk · 2021-10-11T18:30:34Z

@bplb This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8274548: (fc) FileChannel gathering write fails with IOException "Invalid argument" on macOS 11.6

Reviewed-by: alanb

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 76 new commits pushed to the master branch:

75f5145: 8274925: Shenandoah: shenandoah/TestAllocHumongousFragment.java test failed on lock rank check
83c3771: 8273881: Metaspace: test repeated deallocations
3f01d03: 8275021: Test serviceability/sa/TestJmapCore.java fails with: java.io.IOException: Stack frame 0x4 not found
3f07337: 8273614: Shenandoah: intermittent timeout with ConcurrentGCBreakpoint tests
0d80f6c: 8274379: Allow process of unsafe access errors in check_special_condition_for_native_trans
b870468: 8274347: Passing a nested switch expression as a parameter causes an NPE during compile
110e38d: 8274753: ZGC: SEGV in MetaspaceShared::link_shared_classes
b7af890: 8274430: Remove some debug error printing code added in JDK-8017163
aaf2401: 8274927: Remove unnecessary G1ArchiveAllocator code
c55dd36: 8275008: gtest build failure due to stringop-overflow warning with gcc11
... and 66 more: https://git.openjdk.java.net/jdk/compare/bb0bab57a1ff447bfb41cfe10c91838a6812b93d...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

bplb · 2021-10-12T15:25:03Z

/integrate

openjdk · 2021-10-12T15:26:08Z

Going to push as commit 07b1f1c.
Since your change was applied there have been 91 commits pushed to the master branch:

f623460: 8274911: testlibrary_tests/ir_framework/tests/TestIRMatching.java fails with "java.lang.RuntimeException: Should have thrown exception"
e393c5e: 8275074: Cleanup unused code in JFR LeakProfiler
e16b93a: 8274770: [PPC64] resolve_jobject needs a generic implementation to support load barriers
1ab6414: 8275051: Shenandoah: Correct ordering of requested gc cause and gc request flag
b460d6d: 8275091: /src/jdk.management.jfr/share/classes/module-info.java has non-canonical order
d04d4ee: 8274894: Use Optional.empty() instead of ofNullable(null) in HttpResponse.BodySubscribers.discarding
33050f8: 8274986: max code printed in hs-err logs should be configurable
8de2636: 8274615: Support relaxed atomic add for linux-aarch64
7d2633f: 8275002: Remove unused AbstractStringBuilder.MAX_ARRAY_SIZE
cfe7471: 8177814: jdk/editpad is not in jdk TEST.groups
... and 81 more: https://git.openjdk.java.net/jdk/compare/bb0bab57a1ff447bfb41cfe10c91838a6812b93d...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-10-12T15:26:25Z

@bplb Pushed as commit 07b1f1c.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8274548: (fc) FileChannel gathering write fails with IOException "Inv…

27b5c92

…alid argument" on macOS 11.6

openjdk bot added the rfr label Oct 6, 2021

openjdk bot added the nio label Oct 6, 2021

8274548: Move logic from C to Java layer

6a9116b

8274548: Remove setting position which is not needed due to using abs…

1d41dfb

…olute put

8274548: Native macOS-only code change with no modification of iovec …

6da2c82

…array

8274548: Change logic to use WRITEV_MAX

f03fc29

8274548: Minor restructuring of get_darwin_version() and writevMax()

e44a468

AlanBateman reviewed Oct 9, 2021

View changes

8274548: On macOS, remove dependence on Darwin version in native writ…

079ce19

…evMax()

AlanBateman approved these changes Oct 11, 2021

View changes

openjdk bot added the ready label Oct 11, 2021

openjdk bot closed this Oct 12, 2021

openjdk bot added integrated and removed ready labels Oct 12, 2021

openjdk bot removed the rfr label Oct 12, 2021

bplb deleted the FileChannel-writev-8274548 branch Oct 12, 2021

8274548: (fc) FileChannel gathering write fails with IOException "Invalid argument" on macOS 11.6 #5831

8274548: (fc) FileChannel gathering write fails with IOException "Invalid argument" on macOS 11.6 #5831

bplb commented Oct 6, 2021 •

edited by openjdk[bot] bot

bridgekeeper bot commented Oct 6, 2021

bplb commented Oct 6, 2021

openjdk bot commented Oct 6, 2021

mlbridge bot commented Oct 6, 2021 •

edited

AlanBateman commented Oct 6, 2021 •

edited

bplb commented Oct 6, 2021

mlbridge bot commented Oct 6, 2021

AlanBateman commented Oct 7, 2021

bplb commented Oct 7, 2021

bplb commented Oct 7, 2021

AlanBateman commented Oct 8, 2021

bplb commented Oct 8, 2021

AlanBateman Oct 9, 2021

bplb Oct 11, 2021

AlanBateman Oct 11, 2021

bplb Oct 11, 2021 •

edited

openjdk bot commented Oct 11, 2021 •

edited

bplb commented Oct 12, 2021

openjdk bot commented Oct 12, 2021

openjdk bot commented Oct 12, 2021

8274548: (fc) FileChannel gathering write fails with IOException "Invalid argument" on macOS 11.6 #5831

8274548: (fc) FileChannel gathering write fails with IOException "Invalid argument" on macOS 11.6 #5831

Conversation

bplb commented Oct 6, 2021 • edited by openjdk[bot] bot

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Oct 6, 2021

bplb commented Oct 6, 2021

openjdk bot commented Oct 6, 2021

mlbridge bot commented Oct 6, 2021 • edited

Webrevs

AlanBateman commented Oct 6, 2021 • edited

bplb commented Oct 6, 2021

mlbridge bot commented Oct 6, 2021

AlanBateman commented Oct 7, 2021

bplb commented Oct 7, 2021

bplb commented Oct 7, 2021

AlanBateman commented Oct 8, 2021

bplb commented Oct 8, 2021

AlanBateman Oct 9, 2021

Choose a reason for hiding this comment

bplb Oct 11, 2021

Choose a reason for hiding this comment

AlanBateman Oct 11, 2021

Choose a reason for hiding this comment

bplb Oct 11, 2021 • edited

Choose a reason for hiding this comment

openjdk bot commented Oct 11, 2021 • edited

bplb commented Oct 12, 2021

openjdk bot commented Oct 12, 2021

openjdk bot commented Oct 12, 2021

bplb commented Oct 6, 2021 •

edited by openjdk[bot] bot

mlbridge bot commented Oct 6, 2021 •

edited

AlanBateman commented Oct 6, 2021 •

edited

bplb Oct 11, 2021 •

edited

openjdk bot commented Oct 11, 2021 •

edited