Enable SO_REUSEPORT in ipv6-only environments as well

[sreecha]

SO_REUSEPORT was previously getting disabled in ipv6-only environment due to a bug in our tcp_server_posix.cc code

This was causing significant perf regression in ipv6-only environments..

Relevant background and details

What is SO_REUSEPORT ?
SO_REUSEPORT lets multiple listening sockets bind to same port. If we only had one listening socket and multiple threads were doing listen() call on that socket, the OS provided no guarantees on fairly distributing the incoming connections to different threads. SO_REUSEPORT option feature was added so that multiple sockets can all listen on the same port (and different threads can call listen on different sockets) (more details here)

Why and how is it used in gRPC code
In gRPC server, if SO_REUSEPORT option is available, we create multiple sockets - one per each server completion queue - and add them to the corresponding completion queue's pollset. This way, the threads calling grpc_completion_queue_next() on those server completion queues listen on multiple sockets (but all of them are bound to the same port)

Without this option, we only create one listening socket and added the same socket to all pollsets (corresponding to server completion queues). This means all threads end up listening on the same socket and hence are vulnerable to the load distribution problem described above.

Clearly, using SO_REUSEPORT gives significant performance improvements (and this is why, it is generally enabled in our code - unless we are running on some older kernel version that didn't have this feature)

Bug in our code
In our code, we check whether SO_REUSEPORT option is enabled or not by first creating a socket and then setting that option and checking if the option is correctly set or not.

The API we were using to create socket only created IPV4 sockets. Therefore, this check was failing in Ipv6 environments and we were effectively disabling SO_REUSEPORT option..and causing performance regression.

[grpc-testing]

****************************************************************

libgrpc.so

     VM SIZE                                              FILE SIZE
 ++++++++++++++ GROWING                                ++++++++++++++
  +0.3%     +16 src/core/lib/iomgr/tcp_server_posix.cc     +16  +0.3%
       +28%     +23 init                                       +23   +28%

 -+-+-+-+-+-+-+ MIXED                                  +-+-+-+-+-+-+-
  -0.0%     -16 [None]                                     +56  +0.0%

  [ = ]       0 TOTAL                                      +72  +0.0%


****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[grpc-testing]

[trickle] No significant performance differences

[grpc-testing]

[microbenchmarks] No significant performance differences

[grpc-testing]

****************************************************************

libgrpc.so

     VM SIZE                                              FILE SIZE
 ++++++++++++++ GROWING                                ++++++++++++++
  +0.3%     +16 src/core/lib/iomgr/tcp_server_posix.cc     +16  +0.3%
       +28%     +23 init                                       +23   +28%

 -+-+-+-+-+-+-+ MIXED                                  +-+-+-+-+-+-+-
  -0.0%     -16 [None]                                     +56  +0.0%

  [ = ]       0 TOTAL                                      +72  +0.0%


****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[grpc-testing]

[trickle] No significant performance differences

[grpc-testing]

[microbenchmarks] No significant performance differences

[sreecha]

Talked to @adelez and the failures at ""Bazel Debug build for C/C++" are known failures.
All other tests passed. Merging.