net: ReadMsgUnix should pass MSG_CMSG_CLOEXEC on Linux

[rittneje]

The implementation of ReadMsgUnix ultimately does not pass any flags to the recvmsg syscall. This means that any file descriptors sent in via a Unix rights SCM message will not be marked with the close-on-exec flag. For consistent behavior with how Go handles all other file descriptors on Linux, the MSG_CMSG_CLOEXEC flag should be passed. The need for this is the same as all other uses of the various CLOEXEC flags - to prevent a race condition where a child process is forked (and exec'd) after the file descriptor is created and before it can be marked close-on-exec, and is thus leaked.

Unfortunately, I don't think there's any equivalent of this flag on Mac, so there's no easy way to address the inherent race condition there.

[howjmay]

Let me take this issue.
I will use func Recvmsg to replace the current function call of func (*FD) ReadMsg

[gopherbot]

Change https://golang.org/cl/272226 mentions this issue: net: Pass MSG_CMSG_CLOEXEC flag in ReadMsgUnix

[ianlancetaylor]

I think that ideally we should aim for the same behavior on all Unix systems. That means that when MSG_CMSG_CLOEXEC is available we should use it, and when it is not available we should, if len(oob) > 0, acquire syscall.ForkLock and then make sure that we set the O_CLOEXEC flag on any descriptors.

[rittneje]

@ianlancetaylor To prevent all leakages, you would have to hold the lock during the recvmsg syscall. Does that have the potential to cause a problem?

Assuming it's okay, then yes, you could parse the SCMs, iterate through all of them, and then parse each one where Level == syscall.SOL_SOCKET && Type == syscall.SCM_RIGHTS. However, in practice, this will unfortunately lead to double parsing.

[ianlancetaylor]

Good point, we can't hold ForkLock while waiting for network I/O. I guess we'll have to allow for a window when the descriptor is not marked close-on-exec.

Yes, double parsing will be required, but only on systems that don't support MSG_CMSG_CLOEXEC. That set of systems will presumably diminish over time

[howjmay]

So for an OS that MSG_CMSG_CLOEXEC is available, we can use syscall.Recvmsg() directly with MSG_CMSG_CLOEXEC flag. And if MSG_CMSG_CLOEXEC is not available, then double parse the SCMs?

[rittneje]

Yes, "double parse" in as much as the net package will parse the SCMs to find all the fds, and then the actual client will presumably parse the SCMs again to use the fds. It also sounds like you need not bother with the fork lock at all since there's no way to fully guarantee the fds won't be leaked anyway.

[rittneje]

@howjmay I don't think we should just ditch FD.ReadMsg entirely, since that function does several things other than calling syscall.Recvmsg.

Another interesting dimension of the problem: The FD.ReadMsg method in internal/poll is generic and doesn't know whether it is working with a Unix socket specifically. But it shouldn't attempt to parse the OOB data as SCMs unless it is a Unix socket. Otherwise, there is the distinct possibility that we parse some other kind of OOB data that by chance happens to look like SCMs, and then we try to manipulate some junk fds. I'm also not sure if passing MSG_CMSG_CLOEXEC on a non-Unix socket is okay in general.

In short, I think we should:

1. Add flags int as a parameter to FD.ReadMsg in internal/poll.
2. Add flags int as a parameter to netFD.readMsg in net and pass it through to FD.ReadMsg.
3. Update all existing calls to netFD.readMsg in net to pass 0 for flags.
4. Update UnixConn.readMsg to:
   a. Linux: pass syscall.MSG_CMSG_CLOEXEC to netFD.readMsg for flags
   b. non-Linux: if oobn > 0 parse the SCMs in oob[:oobn] and call syscall.CloseOnExec on each fd

4a and 4b will require separate files to avoid compilation issues (i.e., one file named unixsock_linux.go, and another file with the same build tags as unixsock_posix.go sans linux).

Note: FD.ReadMsg is independently defined for Windows. To properly support passing flags, you would have to assign to o.msg.Flags. https://golang.org/src/internal/poll/fd_windows.go?s=25399:25483#L979