[9101] tcp: adding support for SO_REUSEPORT #759

wdauchy · 2017-04-01T06:13:33Z

this makes it possible to listen to the same tcp port using different
processes under linux >= v3.9

codecov · 2017-04-01T07:29:20Z

Codecov Report

Merging #759 into trunk will decrease coverage by 1.1%.
The diff coverage is 64.7%.

@@            Coverage Diff             @@
##            trunk     #759      +/-   ##
==========================================
- Coverage   89.62%   88.52%   -1.11%     
==========================================
  Files         844      844              
  Lines      147851   147868      +17     
  Branches    13065    13067       +2     
==========================================
- Hits       132518   130897    -1621     
- Misses      12919    14505    +1586     
- Partials     2414     2466      +52

wdauchy · 2017-04-02T10:12:23Z

I couldn't validate codecov tests since it requires linux >= 3.9 and not available on windows. However it does work on my local machine. In order to go through codecov, I would have to remove the test.

hawkowl · 2017-04-10T05:56:37Z

@wdauchy Hi! Do you have a ticket on https://twistedmatrix.com/trac that we can reference it against? When we have a ticket number, we can sync it to our buildbot builders (of which some have a kernel version greater than 3.9) so we can run it against the full test suite.

Please file a ticket, or add it to the description, and add the 'review' tag to the trac ticket with a link to this PR, then we can move this along :)

hawkowl · 2017-04-16T06:16:18Z

@wdauchy Oh, I found the ticket. http://twistedmatrix.com/trac/ticket/9101 :)

hawkowl · 2017-04-16T06:19:46Z

Running tests on builders...

wdauchy · 2017-04-19T16:51:47Z

@hawkowl thanks for the update; is there anything else I can do to help? From what I have seen, failing tests are either on windows or ubuntu 12.04 which has linux 3.2

wdauchy · 2017-04-27T10:57:29Z

@hawkowl ping?

hawkowl · 2017-04-27T10:58:54Z

@wdauchy Sorry, there was a problem on our build infra that prevented the tests from going green. Re-running...

create the socket on our side to use our own options this will avoid the need to bind each carbon process to a different port and balance the traffic with iptables rules this requires linux >= 3.9 patches are on their way to support reuseport directly in twisted see twisted/twisted#759 for the tcp part, udp will fillow Signed-off-by: William Dauchy <w.dauchy@criteo.com>

wdauchy · 2017-04-28T11:59:19Z

@hawkowl how does it sound to you? is there anything I can improve?

hawkowl · 2017-04-29T06:24:44Z

@wdauchy I think it might be better implemented how we do the other TCP options: https://github.com/wdauchy/twisted/blob/66929c6d2843e23a6bc16deff0ae8d55e979ac2e/src/twisted/internet/tcp.py#L302 , is that possible? It would reduce the API changes a lot (which I think is a bit excessive). You needn't implement it in iocpreactor, as Windows doesn't support it, so you can just do a twisted.python.runtime.platform.isLinux() check for the tests.

wdauchy · 2017-04-29T10:27:55Z

@hawkowl thanks for your review. I will remove the iocpreactor part as soon as possible; I indeed did not catch it was windows specific; that is understood now.
I don't think it is possible to implement it as a regular tcp option since it has to be set before binding; so it requires us to pass it as argument; do you see any other way to do it in order to avoid API changes?
for the test, I used "hasattr(socket, "SO_REUSEPORT"), which is supposed to avoid issues on both windows or mac. I can however replace it with platform.isLinux() if you prefer; I might also be wrong if those are defined on windows and mac.

hawkowl · 2017-04-29T10:30:23Z

@wdauchy you're right about the hasattr way, because an isLinux check isn't sufficient on older Linuxes. Right now, SO_REUSEPORT is just a Linux specific addition, I don't think it'll ever get adopted by macOS or Windows.

Let me think about the API ramifications. @glyph , you have any ideas, maybe?

glyph · 2017-05-01T00:11:59Z

SO_REUSEADDR on Windows actually behaves a lot like SO_REUSEPORT on Linux; however, it describes this behavior as "non-deterministic" and mostly seems to talk about it as a security hole. See here: See here: https://msdn.microsoft.com/en-us/library/windows/desktop/ms740621(v=vs.85).aspx

Someone with more time than me should do the research into how to get consistent behavior between platforms, and add an API that does that, rather than trying to make the exact flags (which mean different things anyway) apply on both platforms.

this makes it possible to listen to the same tcp port using different processes available on linux >= 3.9 avoid windows test since it is not supported Signed-off-by: William Dauchy <w.dauchy@criteo.com>

this makes it possible to listen to the same tcp port (over ssl) using different processes Signed-off-by: William Dauchy <w.dauchy@criteo.com>

wdauchy · 2017-05-02T13:15:29Z

@hawkowl @glyph so on windows we may set SO_REUSEADDR in case of multiListen set and SO_EXCLUSIVEADDRUSE otherwise
I don't see how to get a real consistent behavior here, since the exact reuseport functionnality is simply not supported elsewhere.

For now, I put back the IOCP interfaces, since it was breaking the endpoints interface; I did not saw a clean way to avoid it.

create the socket on our side to use our own options this will avoid the need to bind each carbon process to a different port and balance the traffic with iptables rules this requires linux >= 3.9 patches are on their way to support reuseport directly in twisted see twisted/twisted#759 for the tcp part, udp will fillow Signed-off-by: William Dauchy <w.dauchy@criteo.com>

moshez

Thanks for working on this! I've recently had the occasion to have to shim in reuseport support into Twisted for an example, and that was pretty painful -- so it would be wonderful to have it in the core.

I've listed a few things that would be better done in a different way.

moshez · 2017-10-04T18:43:10Z

src/twisted/internet/tcp.py

@@ -961,6 +964,8 @@ def createInternetSocket(self):
        s = base.BasePort.createInternetSocket(self)
        if platformType == "posix" and sys.platform != "cygwin":
            s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+            if self.listenMultiple and hasattr(socket, "SO_REUSEPORT"):


This should probably fail if listenMultiple has been requested and we cannot do it, since it literally changes semantics (e.g., we can't run the program twice).

moshez · 2017-10-04T18:47:46Z

src/twisted/internet/endpoints.py

@@ -463,11 +463,15 @@ def __init__(self, reactor, port, backlog, interface):

        @param interface: The hostname to bind to
        @type interface: str
+
+        @param listenMultiple: allow multiple processes to listen to the same port
+        @type listenMultiple: bool


This feature is called:

listenMultiple (in some places)

multiListen (in others)

SO_REUSEPORT (by the kernel level documentation that someone will read and wonder "does Twisted support it)

I suggest collapsing all of these into a single name "reuseport" (lose the SO), thus keeping close to the kernel terminology and not inventing our own.

moshez · 2017-10-04T19:06:02Z

src/twisted/internet/interfaces.py

@@ -728,7 +728,7 @@ def lookupZone(name, timeout=None):

 class IReactorTCP(Interface):

-    def listenTCP(port, factory, backlog=50, interface=''):
+    def listenTCP(port, factory, backlog=50, interface='', listenMultiple=False):


This is changing an interface to add a new optional parameter.

This is a backwards incompatible change -- since someone external to Twisted who has written their own reactor is now no longer complying with this interface.

If we want to do a backwards incompatible change, we should go by the appropriate process -- probably following the "exceptions" section in https://twistedmatrix.com/documents/current/core/development/policy/compatibility-policy.html#procedure-for-exceptions-to-this-policy

An alternative is to add reuse port in a separate interface (e.g., :code:IReactorTCPListenMulti), which we would only add to the reactor if we detected SO_REUSEPORT in socket)

Also -- presumably we would like to be able to REUSEPORT on IPv6 as well?
(Either way, whether we do a backwards incompatible change or if we add a new interface,
it would be worthwhile to chunk those into one change.)

OTOH -- I'm not that sure we need the support in listenSSL -- since that functionality can easily be accomplished with the txsni plugin (and we probably want to get away from SSL having to lock-step with TCP)

moshez · 2018-04-10T03:31:31Z

FYI -- there has been no response on this for about 6 months. I am going to continue working on this in #994 . Once it is green on buildbot, I will close this one with a link.

wdauchy · 2018-04-14T13:45:26Z

@moshez sorry for the lack of reply.

wiml · 2018-10-29T01:29:21Z

Someone with more time than me should do the research into how to get consistent behavior between platforms, and add an API that does that, rather than trying to make the exact flags (which mean different things anyway) apply on both platforms.

FWIW, this Stack Overflow reply does an excellent job of covering different OS's various REUSE options, and how there are at least two distinct behaviors being controlled by SO_REUSEfoo sockopts.

wbarnha · 2023-04-28T17:25:53Z

Has this been abandoned? I would really like to get support for SO_REUSEPORT.

adiroiban · 2023-04-28T19:07:07Z

The last update on this PR was in 2017/2018.
I would say this is not active.

glyph · 2023-04-28T21:09:47Z

Has this been abandoned? I would really like to get support for SO_REUSEPORT.

@wbarnha The original author definitely seems uninterested in addressing the feedback and conflicts, so I'm going to go ahead and close this PR. If you'd like to see this feature, open a new PR to address #9101 (maybe have a look at #994 too, to see if there's anything useful to be gleaned there?), and feel free to base it on this code.

wdauchy · 2023-04-28T21:50:02Z

@wbarnha The original author definitely seems uninterested in addressing the feedback and conflicts, so I'm going to go ahead and close this PR. If you'd like to see this feature, open a new PR to address #9101 (maybe have a look at #994 too, to see if there's anything useful to be gleaned there?), and feel free to base it on this code.

Feel free to follow-up indeed. I thought someone had done the work since then.

wdauchy force-pushed the 9101-wdauchy-tcp_reuseport branch from a092dfe to 3854807 Compare April 1, 2017 07:28

wdauchy force-pushed the 9101-wdauchy-tcp_reuseport branch 11 times, most recently from 9e29d84 to 63b13fe Compare April 1, 2017 20:37

wdauchy force-pushed the 9101-wdauchy-tcp_reuseport branch from 66929c6 to 0dfe419 Compare April 29, 2017 10:20

wdauchy added 2 commits May 2, 2017 15:06

tcp: adding support for SO_REUSEPORT

7022845

this makes it possible to listen to the same tcp port using different processes available on linux >= 3.9 avoid windows test since it is not supported Signed-off-by: William Dauchy <w.dauchy@criteo.com>

tcp, ssl: adding support for SO_REUSEPORT

d84cdf9

this makes it possible to listen to the same tcp port (over ssl) using different processes Signed-off-by: William Dauchy <w.dauchy@criteo.com>

wdauchy force-pushed the 9101-wdauchy-tcp_reuseport branch from 0dfe419 to d84cdf9 Compare May 2, 2017 13:10

moshez requested changes Oct 4, 2017

View reviewed changes

twm changed the title ~~tcp: adding support for SO_REUSEPORT~~ [9101] tcp: adding support for SO_REUSEPORT Jul 19, 2020

glyph closed this Apr 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[9101] tcp: adding support for SO_REUSEPORT #759

[9101] tcp: adding support for SO_REUSEPORT #759

wdauchy commented Apr 1, 2017 •

edited

Loading

codecov bot commented Apr 1, 2017 •

edited

Loading

wdauchy commented Apr 2, 2017

hawkowl commented Apr 10, 2017

hawkowl commented Apr 16, 2017

hawkowl commented Apr 16, 2017

wdauchy commented Apr 19, 2017

wdauchy commented Apr 27, 2017

hawkowl commented Apr 27, 2017

wdauchy commented Apr 28, 2017

hawkowl commented Apr 29, 2017

wdauchy commented Apr 29, 2017

hawkowl commented Apr 29, 2017

glyph commented May 1, 2017

wdauchy commented May 2, 2017

moshez left a comment

moshez Oct 4, 2017

moshez Oct 4, 2017

moshez Oct 4, 2017

moshez commented Apr 10, 2018

wdauchy commented Apr 14, 2018

wiml commented Oct 29, 2018

wbarnha commented Apr 28, 2023

adiroiban commented Apr 28, 2023

glyph commented Apr 28, 2023

wdauchy commented Apr 28, 2023

[9101] tcp: adding support for SO_REUSEPORT #759

[9101] tcp: adding support for SO_REUSEPORT #759

Conversation

wdauchy commented Apr 1, 2017 • edited Loading

codecov bot commented Apr 1, 2017 • edited Loading

Codecov Report

wdauchy commented Apr 2, 2017

hawkowl commented Apr 10, 2017

hawkowl commented Apr 16, 2017

hawkowl commented Apr 16, 2017

wdauchy commented Apr 19, 2017

wdauchy commented Apr 27, 2017

hawkowl commented Apr 27, 2017

wdauchy commented Apr 28, 2017

hawkowl commented Apr 29, 2017

wdauchy commented Apr 29, 2017

hawkowl commented Apr 29, 2017

glyph commented May 1, 2017

wdauchy commented May 2, 2017

moshez left a comment

Choose a reason for hiding this comment

moshez Oct 4, 2017

Choose a reason for hiding this comment

moshez Oct 4, 2017

Choose a reason for hiding this comment

moshez Oct 4, 2017

Choose a reason for hiding this comment

moshez commented Apr 10, 2018

wdauchy commented Apr 14, 2018

wiml commented Oct 29, 2018

wbarnha commented Apr 28, 2023

adiroiban commented Apr 28, 2023

glyph commented Apr 28, 2023

wdauchy commented Apr 28, 2023

wdauchy commented Apr 1, 2017 •

edited

Loading

codecov bot commented Apr 1, 2017 •

edited

Loading