Rework ExclusiveAddressUse and ReuseAddress on non-Windows platforms

[ericeil]

The ExclusiveAddressUse socket option is a Windows-specific option that, when set to "true," tells Windows not to allow another socket to use the same local address as this socket. This is only needed on Windows because it otherwise (at least in some versions/configurations) allows any socket with the ReuseAddress option set to "steal" the address from a socket that did not have any options set.

On Unix, we previously treated this as an "unsupported" option. However, it is recommended to set this option to "true" on Windows, to avoid malicious theft of a service's address, so we need to support the option, in some fashion, on Unix, so that it's possible to write portable code that works reliably everywhere. Since the only behavior on Linux/OSX is equivalent to "ExclusiveAddressUse=true" on Windows, we implement this option as a no-op if it's set to "true," and as an unsupported option if set to "false." (It's possible that we could come up with a better failure for the "false" case, but I'm treating it as "unsupported" for compatiblity with the 1.0 release).

Another related option is ReuseAddress. On Windows, this option allows a socket's address and port to be reused. It's equivalent to two native options on Unix: SO_REUSEADDR and SO_REUSEPORT. Again, for portability, we need an option that will work roughly the same way on all platforms. We could introduce a new option (ReuseAddressAndPort?) but existing code is already using the current ReuseAddress option. So this change makes ReuseAddress set both SO_REUSEADDR and SO_REUSEPORT on Unix. If we need to support these options individually, on Unix only, in the future, we'll need to introduce two new options (maybe ReuseAddressOnly and ReusePort) which will likely need to be treated as "unsupported" options on Windows. For now, no need for managed support for more fine-grained options has been demonstrated.

For more information on the underlying native options on Windows, Linux, OSX, etc., see this great writeup on stackoverflow. Also, the Windows docs discuss the SO_EXCLUSIVEADDR option in depth.

Closes #11292.

@stephentoub @CIPop

[ericeil]

cc: @christianhuening

[CIPop]

As pointed out in the design discussion, this could be confusing: what if on *NIX I only want to set SO_REUSEPORT but not SO_REUSEADDR?
Is that a a valid *NIX scenario? If yes, is there a way to achieve that somehow?

[ericeil]

Is that a a valid *NIX scenario? If yes, is there a way to achieve that somehow?

I'm not sure if that's a valid scenario. It's not one anyone's asking for right now. 😃

The premise of this change is that we have an already-established semantic for ReuseAddress in managed code, and there's more value in this semantic being portable than having the managed socket option names map 1:1 with native socket options. I agree that breaking that 1:1 mapping may be confusing for folks who are specifically targeting Unix socket options, but keeping the 1:1 mapping is also confusing, to those who are trying to get consistent behavior across platforms.

[christianhuening]

Well I've done a bit of research and found that SO_REUSEPORT seems to be a bit exotic yet useful in extreme load scenarios (i.e. at Google: https://lwn.net/Articles/542629/).
My guess is that no one would ask for this in the near future, since it hasn't been available in .NET Framework so far. However can it hurt to add an option to allow to possibly tweak high performance socket code in special conditions, when you know you're running on *NIX?

EDIT: Here is the original motivation for this feature (source: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c617f398edd4db2b8567a28e899a88f8f574798d):

The motivating case for so_resuseport in TCP would be something like
a web server binding to port 80 running with multiple threads, where
each thread might have it's own listener socket. This could be done
as an alternative to other models: 1) have one listener thread which
dispatches completed connections to workers. 2) accept on a single
listener socket from multiple threads. In case #1 the listener thread
can easily become the bottleneck with high connection turn-over rate.
In case #2, the proportion of connections accepted per thread tends
to be uneven under high connection load (assuming simple event loop:
while (1) { accept(); process() }, wakeup does not promote fairness
among the sockets. We have seen the disproportion to be as high
as 3:1 ratio between thread accepting most connections and the one
accepting the fewest. With so_reusport the distribution is
uniform.

Nginx uses it as well: https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/
And Apache does so too: https://httpd.apache.org/docs/trunk/en/mod/mpm_common.html#listencoresbucketsratio

[CIPop]

Thanks @christianhuening for the investigation!

Maybe properly supporting this and documenting/adding a new property/method would be a better long-term plan.

[stephentoub]

However can it hurt to add an option to allow to possibly tweak high performance socket code in special conditions, when you know you're running on *NIX?

Once Socket has its public Handle property added back, this could be done via a P/Invoke that's only invoked when on Unix. That could serve as a stop-gap until such time as it was determined to be important to add .NET APIs for.

[CIPop]

LGTM (left one commend on the design)

[christianhuening]

Thank you very much! Greatly appreciated!

[stephentoub]

What's the expected behavior of these cases on Unix? Should we verify those failures via a test that's Unix-only?

[stephentoub]

OSX seems to be having an issue with the change:

System.Net.Sockets.Tests.SocketOptionNameTest.ReuseAddress(exclusiveAddressUse: null, firstSocketReuseAddress: True, secondSocketReuseAddress: True, expectFailure: False) [FAIL]
        System.Net.Sockets.SocketException : Address already in use
        Stack Trace:
              at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
              at System.Net.Sockets.Socket.Bind(EndPoint localEP)
              at System.Net.Sockets.Tests.SocketOptionNameTest.ReuseAddress(Nullable`1 exclusiveAddressUse, Nullable`1 firstSocketReuseAddress, Nullable`1 secondSocketReuseAddress, Boolean expectFailure)

[stephentoub]

Some comments, and a failing test, but otherwise LGTM. Thanks for the fix.

[christianhuening]

Any updates on this? Would it help if I try this version as well? Or has the OSX error already been investigated?

[ericeil]

The test is failing on OSX, even on my machine. I could swear it passed before I submitted this PR, but clearly something went wrong. I haven't had time to investigate.

@christianhuening, if you want to try these changes, it would be useful to know if your scenario works.

[christianhuening]

@ericeil Alright, I'll give it a try in a docker container then to see whether the logic works.
UPDATE:
@ericeil I actually have trouble building your repo.

On Ubuntu 14.04 I receive the following error when running ./build-native.sh

root@e4d746115f74:/corefx# ./build.sh
Installing dotnet cli...
Initializing BuildTools...
/corefx/init-tools.sh: line 121: /corefx/packages/Microsoft.DotNet.BuildTools/1.0.26-prerelease-00809-01/lib/init-tools.sh: No such file or directory
ERROR: An error occured when trying to initialize the tools. Please check '/corefx/init-tools.log' for more details.1

On OSX I have trouble with OpenSSL:
When trying to link with brew as described, I get this:

christian$ brew link --force openssl
Warning: Refusing to link: openssl
Linking keg-only openssl means you may end up linking against the insecure,
deprecated system OpenSSL while using the headers from Homebrew's openssl.
Instead, pass the full include/library paths to your compiler e.g.:
  -I/usr/local/opt/openssl/include -L/usr/local/opt/openssl/lib

then running build-native.sh gives me:

christian$ ./build-native.sh 
Tools are already initialized
Running: /Users/christian/Documents/Softwareprojekte/corefx/src/Native/build-native.sh   x64  Debug  OSX  --numproc 12
Setting up directories for build
Checking pre-requisites...
Commencing build of corefx native components for OSX.x64.Debug
Invoking cmake with arguments: "/Users/christian/Documents/Softwareprojekte/corefx/src/Native/Unix" DEBUG 
No CMake extra Args specified
-- Configuring incomplete, errors occurred!
See also "/Users/christian/Documents/Softwareprojekte/corefx/bin/obj/OSX.x64.Debug/Native/CMakeFiles/CMakeOutput.log".
See also "/Users/christian/Documents/Softwareprojekte/corefx/bin/obj/OSX.x64.Debug/Native/CMakeFiles/CMakeError.log".
Failed to generate native component build project!
CMake Error at /usr/local/Cellar/cmake/3.6.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
  Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
  system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.6.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.6.1/share/cmake/Modules/FindOpenSSL.cmake:380 (find_package_handle_standard_args)
  System.Security.Cryptography.Native/CMakeLists.txt:18 (find_package)

Any idea what's wrong here? To which OpenSSL should I point the variable to?

[karelz]

@ericeil can you help?

[ericeil]

@christianhuening, I'm very sorry I didn't see your last response. I'll see if I can get this working.

[christianhuening]

@ericeil That would be great! This is the only thing which is left to have my project(s) ported successfully. Everything else is working now :-)

[ericeil]

Turns out I just had some bad error checking. Added a commit with a fix.

[ericeil]

CentOS Debug "build timed out.". No obvious cause from the log. This change is unlikely to have caused this, and all other tests passed, so I'm going ahead with the merge.