Skip to content

Add socket.bind_socket() convenience function #61761

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
giampaolo opened this issue Mar 27, 2013 · 29 comments
Closed

Add socket.bind_socket() convenience function #61761

giampaolo opened this issue Mar 27, 2013 · 29 comments
Assignees
Labels
3.8 (EOL) end of life stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@giampaolo
Copy link
Contributor

BPO 17561
Nosy @loewis, @gpshead, @jaraco, @nirs, @pitrou, @vstinner, @giampaolo, @bitdancer, @asvetlov, @berkerpeksag, @vadmium, @jleedev, @csabella
PRs
  • bpo-17561: add socket.create_server_sock() and socket.has_dual_stack() #11215
  • bpo-35934: Add socket.create_server() utility function #11784
  • BPO-17561: set create_server backlog default to None #12735
  • Files
  • socket.patch: adds create_server_sock()
  • socket2.patch: disable IPV6_V6ONLY, adds has_dual_stack()
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/giampaolo'
    closed_at = <Date 2019-04-09.10:34:52.387>
    created_at = <Date 2013-03-27.16:48:29.927>
    labels = ['3.8', 'type-feature', 'library']
    title = 'Add socket.bind_socket() convenience function'
    updated_at = <Date 2019-04-09.17:53:09.958>
    user = 'https://github.com/giampaolo'

    bugs.python.org fields:

    activity = <Date 2019-04-09.17:53:09.958>
    actor = 'vstinner'
    assignee = 'giampaolo.rodola'
    closed = True
    closed_date = <Date 2019-04-09.10:34:52.387>
    closer = 'giampaolo.rodola'
    components = ['Library (Lib)']
    creation = <Date 2013-03-27.16:48:29.927>
    creator = 'giampaolo.rodola'
    dependencies = []
    files = ['29596', '29600']
    hgrepos = []
    issue_num = 17561
    keywords = ['patch', 'needs review']
    message_count = 29.0
    messages = ['185348', '185362', '185369', '185370', '185372', '185374', '185443', '185447', '185456', '185483', '185486', '185487', '185489', '185497', '185499', '185765', '185849', '185919', '327719', '331943', '331944', '335021', '335036', '335285', '339678', '339687', '339694', '339723', '339794']
    nosy_count = 20.0
    nosy_names = ['loewis', 'gregory.p.smith', 'jaraco', 'nirs', 'pitrou', 'vstinner', 'giampaolo.rodola', 'josiah.carlson', 'r.david.murray', 'asvetlov', 'neologix', 'berker.peksag', 'martin.panter', 'jpokorny', 'jleedev', 'dazhaoyu', 'andreasr', 'Carlos.Ralli', 'Paul Marks', 'cheryl.sabella']
    pr_nums = ['11215', '11784', '12735']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue17561'
    versions = ['Python 3.8']

    @giampaolo
    Copy link
    Contributor Author

    Here's a function similar to socket.create_connection() which addresses all the repetitive tasks needed to order to create an IPv4/IPv6 agnostic server socket.

    @giampaolo giampaolo added easy stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Mar 27, 2013
    @neologix
    Copy link
    Mannequin

    neologix mannequin commented Mar 27, 2013

    I think that's a good idea.

    However, there's a problem with the implementation: if one passes "" or None as address on a dual-stack node, the resulting socket will be either IPv4 bound to INADDR_ANY, or IPv6 bound to IN6ADDR_ANY, whereas one would expect to be bound both in IPv4 and IPv6. In the later case, some platforms (like Linux) use IPv4-mapped addresses by default, some others (e.g. some versions of FreeBSD) don't. So, depending on IPV6_V6ONLY setting, binding to IPV6 any won't accept IPv4 connections. Also, some platforms don't support mapped addresses at all, so the only portable solution is to bind both to 0.0.0.0 and ::, whith two different sockets, and use select() to accept both.

    So it would maybe make sense to expose a ServerSocket class, with an accept method which would do the right thing (as an added bonus, it could expose set_reuse_addr(bool), since SO_REUSEADDR have subtle semantic differences between Unix and Windows).

    @giampaolo
    Copy link
    Contributor Author

    What you say is right but whether the kernel supports an hybrid IPv4/6 stack or not there's not much we can do about it anyway.
    Exactly what are you suggesting with the ServerSocket class you mentioned? What do you expect it to do?
    Note that platforms supporting the dual-stack are already supported. This:

    >> socket.create_server_socket(("::", 0))

    ...on Linux will create a socket which is reachable both as "::1" and "127.0.0.1".
    So if I'm not mistaken the alias for specifying "listen on all interfaces for both IPv4 and IPv6 addresses" would be "::" instead of "" / None.
    We can document that.

    @giampaolo
    Copy link
    Contributor Author

    Side note: this is how in pyftpdlib I determine whether a platform supports the dual stack:

    def support_hybrid_ip_v4_v6():
        # Note: IPPROTO_IPV6 constant is broken on Windows, see:
        # http://bugs.python.org/issue6926
        sock = None
        try:
            if not socket.has_ipv6:
                return False
            sock = socket.socket(socket.AF_INET6)
            return not sock.getsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY)
        except (socket.error, AttributeError):
            return False
        finally:
            if sock is not None:
                sock.close()

    @gvanrossum
    Copy link
    Member

    Tulip has something similar. Someone should compare the two and make sure they are equivalent or similar.

    @neologix
    Copy link
    Mannequin

    neologix mannequin commented Mar 27, 2013

    What you say is right but whether the kernel supports an hybrid IPv4/6 stack
    or not there's not much we can do about it anyway.
    Exactly what are you suggesting with the ServerSocket class you mentioned?
    What do you expect it to do?

    There's a confusion.
    Dual-stack merely means that the host supports both IPv4 and IPv6 natively.
    The IPV6_V6ONLY tunable is just a way to enable or not IPv4-mapped
    addresses (i.e. ::ffff::<IPv4 address>) so that an IPv4 client can
    connect to this socket. It can be set system-wide, or on a socket
    basis.

    Note that platforms supporting the dual-stack are already supported. This:

    >>> socket.create_server_socket(("::", 0))

    ...on Linux will create a socket which is reachable both as "::1" and
    "127.0.0.1".

    Try the same thing after:
    # echo 1 > /proc/sys/net/ipv6/bindv6only

    It won't accept IPv4 connections anymore.

    And that's the default on many (most) platforms, e.g. FreeBSD and
    OpenBSD (I think it's also true for Windows).

    So the bottom line is that with the current code, on some - most -
    platforms, we'll only accept IPv6 connections (and since you iterate
    over getaddrinfo() in an unspecified order you may very well bind to
    IPv4 only first, in which case you'll only accept IPv4 clients).

    The proper way to procedd, on platforms which don't support unsetting
    IPV6_V6ONLY, is to use two sockets, one in IPv4, and one IPv6, and use
    select() to accept connections.

    This would propably belong to an overriden accept() method in a
    ServerSocket class, since it's far from trivial.

    @giampaolo
    Copy link
    Contributor Author

    Thanks for clarifying, I have a better understanding of the problem now.
    Providing a custom class instantiating two sockets looks like a dead end to me though. To say one, what is getsockname() supposed to return? Same for detach(), fileno(), 'family' and probably others I can't think of right now.

    @neologix
    Copy link
    Mannequin

    neologix mannequin commented Mar 28, 2013

    Providing a custom class instantiating two sockets looks like a dead end to me though. To say one, what is getsockname() supposed to return? Same for detach(), fileno(), 'family' and probably others I can't think of right now.

    Indeed.

    So we might opt for a best-effort approach: try disabling IPV6_V6ONLY
    on the socket if it's set: that way it should work on most platforms.
    And add a note to the documentation. IIRC that's what Java does.

    @giampaolo
    Copy link
    Contributor Author

    Agreed. Then it probably makes sense to expose also a socket.has_dual_stack() function so that the user can pre-emptively decide whether using a custom class.
    Updated draft patch is in attachment.

    @giampaolo
    Copy link
    Contributor Author

    I managed to write a container class which listens on multiples addresses and uses select/poll on accept() as suggested by Charles.
    FWICT it seems to work pretty well and supports non-blocking sockets and timeouts (tested on Linux, Windows and OSX).

    It is available as a recipe here:
    http://code.activestate.com/recipes/578504

    IMO has_dual_stack() and create_server_sock() functions can already be included as-is (will adapt the recipe in order to get rid of Python 2.X support and submit another patch).

    As for including also MultipleSocketsListener that's debatable.
    It can either be added or linked/mentioned it in the doc.

    @gvanrossum
    Copy link
    Member

    Perhaps you can contribute something like this to Tulip? We've got code to run a server that can handle IPv4 and IPv6, but we currently don't have something that just creates a separate socket for each address family. Our UDP and TCP paths are also quite different.

    @giampaolo
    Copy link
    Contributor Author

    Yep, no prob. It would also be a good chance to test it in a real-world app. Where should I look?

    @gvanrossum
    Copy link
    Member

    Tulip is at code.google.com/p/tulip

    On Thu, Mar 28, 2013 at 7:51 PM, Giampaolo Rodola'
    <report@bugs.python.org>wrote:

    Giampaolo Rodola' added the comment:

    Yep, no prob. It would also be a good chance to test it in a real-world
    app. Where should I look?

    ----------


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue17561\>


    @giampaolo
    Copy link
    Contributor Author

    > Where should I look?
    Tulip is at code.google.com/p/tulip

    I meant in the code (and what needs to be done/refactored exactly).

    @gvanrossum
    Copy link
    Member

    start_serving() in base_events.py.

    On Fri, Mar 29, 2013 at 4:58 AM, Giampaolo Rodola'
    <report@bugs.python.org>wrote:

    Giampaolo Rodola' added the comment:

    >> Where should I look?
    > Tulip is at code.google.com/p/tulip

    I meant in the code (and what needs to be done/refactored exactly).

    ----------


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue17561\>


    @giampaolo
    Copy link
    Contributor Author

    Being Tulip asynchronous I think that what it needs is an utility function which returns *multiple* sockets as are the addresses returned by getaddrinfo() and also possibly even disable the IPv4/6 dual stack in order to be consistent across all platforms.

    After the sockets are returned they can be "registered" against the event loop as two separate entities such as, say, ("0.0.0.0", 8000) *and* ("::", 8000).
    If you think this makes sense I can contribute something like this into Tulip, or I can bring it up on Tulip's ml and ask for other people's opinions.

    My current recipe is different in that it provides a function which bind()s on one socket only and tries to enable the dual stack whenever possible in order to support IPv4 and IPv6 with a single socket.
    In this it is similar to socket.create_connection() and clearly favors blocking socket usages (although it can also be used in non-blocking apps) which kind of represents the default for the stdlib.

    @gvanrossum
    Copy link
    Member

    Nikolay, you may want to check out this Python tracker issue.

    Giampaolo: I didn't even know it was possible for a single socket to be
    dual-stack. (It would seem problematic for UDP recvfrom() for example.) In
    Tulip it does indeed make more sense to create separate sockets and just
    listen on all of them (using the same protocol factory). This can be
    completely transparent to the code calling start_serving() and to the
    protocol implementation. So if you're brave you can just send a code review
    using codereview.appspot.com to the python-tulip list!

    On Mon, Apr 1, 2013 at 1:13 PM, Giampaolo Rodola' <report@bugs.python.org>wrote:

    Giampaolo Rodola' added the comment:

    Being Tulip asynchronous I think that what it needs is an utility function
    which returns *multiple* sockets as are the addresses returned by
    getaddrinfo() and also possibly even disable the IPv4/6 dual stack in order
    to be consistent across all platforms.

    After the sockets are returned they can be "registered" against the event
    loop as two separate entities such as, say, ("0.0.0.0", 8000) *and* ("::",
    8000).
    If you think this makes sense I can contribute something like this into
    Tulip, or I can bring it up on Tulip's ml and ask for other people's
    opinions.

    My current recipe is different in that it provides a function which
    bind()s on one socket only and tries to enable the dual stack whenever
    possible in order to support IPv4 and IPv6 with a single socket.
    In this it is similar to socket.create_connection() and clearly favors
    blocking socket usages (although it can also be used in non-blocking apps)
    which kind of represents the default for the stdlib.

    ----------


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue17561\>


    @giampaolo
    Copy link
    Contributor Author

    Here's a patch for Tulip:
    https://codereview.appspot.com/8307045
    Will ping Tulip's ml as well.

    @csabella
    Copy link
    Contributor

    Since Tulip/asyncio has gone through a lot of development since this issue was added, I wasn't sure if this has been included already or if there was still interest in it. In either case, I think it might be able to be closed, but I wanted to make sure first. Thanks!

    @jaraco
    Copy link
    Member

    jaraco commented Dec 17, 2018

    I do believe this issue is still important and relevant. See bpo-25667 for a duplicate ticket (and references to implementations) and bpo-24209 for another issue where this could have been applied.

    @jaraco jaraco added the 3.8 (EOL) end of life label Dec 17, 2018
    @giampaolo
    Copy link
    Contributor Author

    Interesting. Yes, I agree this proposal is still desirable. We can reuse socket.create_server_sock() in smtpd, ftplib, socketserver (bpo-20215) and http.server (bpo-24209) modules which will inherit dual-stack IPv4/6 capabilities for free. I should be able to provide a PR sometime during this month.

    @giampaolo giampaolo self-assigned this Dec 17, 2018
    @jaraco
    Copy link
    Member

    jaraco commented Feb 7, 2019

    In bpo-24209, I ended up settling on this implementation (

    cpython/Lib/http/server.py

    Lines 1227 to 1234 in f289084

    def _get_best_family(*address):
    infos = socket.getaddrinfo(
    *address,
    type=socket.SOCK_STREAM,
    flags=socket.AI_PASSIVE,
    )
    family, type, proto, canonname, sockaddr = next(iter(infos))
    return family, sockaddr
    ), which seems to work well.

    @giampaolo
    Copy link
    Contributor Author

    After careful thinking I realize I'm not completely sure about how to expose the IPv4/6 functionality yet. I prefer to defer it for later and start landing a bind_socket() utility function which serves as a base for this functionality. See: bpo-17561.

    @giampaolo
    Copy link
    Contributor Author

    After iterating over this over the last few days I realized it makes more sense to implement and discuss the whole thing in here and as a single PR, so sorry about previously splitting this in a separate ticket/PR. Relevant PR is now this one and is ready for review:
    #11784
    Relevant changes which probably require attention/discussion are:

    1. since it was sort of ambiguous I renamed "has_dual_stack()" to "supports_hybrid_ipv46()". There could be space for some bikeshed as we already have "socket.has_ipv6" so this may be "has_hybrid_ipv46()" called instead

    2. if family is unspecified and determined from *host* (e.g. "localhost" or "") the function sorts getaddrinfo() results preferring AF_INET over AF_INET6

    3. doc includes link to my http://code.activestate.com/recipes/578504 recipe for platforms not supporting hybrid_ipv46 natively

    4. it may be worthwhile (or maybe not?) to have another complementary bind_sockets() (plural) function returning all items from getaddrinfo(). That would be useful for non-blocking apps/frameworks and could be reused by asyncio.

    Also, I'm CC-ing people from bpo-20215 as it contains relevant comments.

    @giampaolo giampaolo removed the easy label Feb 12, 2019
    @giampaolo giampaolo changed the title Add socket.create_server_sock() convenience function Add socket.bind_socket() convenience function Feb 12, 2019
    @giampaolo
    Copy link
    Contributor Author

    Patch committed as of:
    eb7e29f
    For posterity, since the API evolved since the original proposal (as per PR review/suggestions), this is the final incarnation:

        # IPv4 only
        >>> socket.create_server(addr)  
        # IPv6 only
        >>> socket.create_server(addr, family=socket.AF_INET6)
        # IPv4 + IPv6
        >>> socket.create_server(addr, family=socket.AF_INET6, dualstack_ipv6=True)
        # IPv4/6 if possible and don't care about IPv4 mapped addresses, else IPv4
        >>> if socket.has_dualstack_ipv6():
        ...    s = socket.create_server(addr, family=socket.AF_INET6, dualstack_ipv6=True)
        ... else:
        ...    s = socket.create_server(addr)

    @vstinner
    Copy link
    Member

    vstinner commented Apr 9, 2019

    The change broke multiple buildbots. Example:

    https://buildbot.python.org/all/#builders/16/builds/2625

    Traceback (most recent call last):
      File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/_test_multiprocessing.py", line 4377, in test_wait_socket
        self.assertEqual(b''.join(v), expected)
    AssertionError: b'1\n2\n3\n4\n5\n6\n7\n8\n9\n' != b'0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n'

    More logs:

    #11784 (comment)

    @vstinner vstinner reopened this Apr 9, 2019
    @giampaolo
    Copy link
    Contributor Author

    New changeset 8702b67 by Giampaolo Rodola in branch 'master':
    BPO-17561: set create_server backlog default to None (GH-12735)
    8702b67

    @giampaolo
    Copy link
    Contributor Author

    Fixed. There's a remaining failing BB:
    https://buildbot.python.org/all/#/builders/176/builds/185/steps/4/logs/stdio
    ...but the failure appears unrelated and it has been red for a while.

    @vstinner
    Copy link
    Member

    vstinner commented Apr 9, 2019

    """
    Fixed. There's a remaining failing BB:
    https://buildbot.python.org/all/#/builders/176/builds/185/steps/4/logs/stdio
    ...but the failure appears unrelated and it has been red for a while.
    """

    That's known and unrelated issue: https://bugs.python.org/issue31453

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @gpshead gpshead changed the title Add socket.bind_socket() convenience function Support UDP in socket.create_server() Feb 6, 2025
    @gpshead gpshead changed the title Support UDP in socket.create_server() Add socket.bind_socket() convenience function Feb 6, 2025
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 (EOL) end of life stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants