Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setting SO_REUSEPORT fails at API 21 #71045

Closed
xdegaye mannequin opened this issue Apr 26, 2016 · 28 comments
Closed

setting SO_REUSEPORT fails at API 21 #71045

xdegaye mannequin opened this issue Apr 26, 2016 · 28 comments
Labels
3.7 expert-asyncio type-bug An unexpected behavior, bug, or error

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented Apr 26, 2016

BPO 26858
Nosy @gvanrossum, @vstinner, @giampaolo, @pjenvey, @xdegaye, @Fak3, @1st1, @moreati, @yan12125
Dependencies
  • bpo-27027: add the 'is_android' attribute to test.support
  • Files
  • test.asyncio.patch
  • test.asyncio_2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2016-04-26.13:37:22.784>
    labels = ['type-bug', '3.7', 'expert-asyncio']
    title = 'setting SO_REUSEPORT fails at API 21'
    updated_at = <Date 2017-12-10.14:24:43.827>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2017-12-10.14:24:43.827>
    actor = 'xdegaye'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['asyncio']
    creation = <Date 2016-04-26.13:37:22.784>
    creator = 'xdegaye'
    dependencies = ['27027']
    files = ['42610', '42940']
    hgrepos = []
    issue_num = 26858
    keywords = ['patch']
    message_count = 22.0
    messages = ['264281', '264602', '264605', '264635', '264653', '264662', '264666', '264690', '265928', '266061', '276615', '285392', '285394', '285402', '285406', '302351', '302357', '304704', '304708', '307947', '307950', '307960']
    nosy_count = 9.0
    nosy_names = ['gvanrossum', 'vstinner', 'giampaolo.rodola', 'pjenvey', 'xdegaye', 'Roman.Evstifeev', 'yselivanov', 'Alex.Willmer', 'yan12125']
    pr_nums = []
    priority = 'normal'
    resolution = None
    stage = 'needs patch'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue26858'
    versions = ['Python 3.6', 'Python 3.7']

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Apr 26, 2016

    Android defines SO_REUSEPORT on android API 21 but setting this option in the asyncio tests raises OSError: [Errno 92] Protocol not available.

    The attached patch assumes there is a platform.android_ver() function to detect that this is the android platform. The patch does not take into account the fact that this may be fixed in future versions of android.

    @xdegaye xdegaye mannequin added build The build process and cross-build type-bug An unexpected behavior, bug, or error labels Apr 26, 2016
    @gvanrossum
    Copy link
    Member

    gvanrossum commented May 1, 2016

    Where is the patch that adds android_ver()? At the very least that should be listed as a dependency here.

    I don't very much like to have changes to asyncio that can only work in Python 3.6; there's still an upstream version (https://github.com/python/asyncio) that is used to do PyPI releases of asyncio for Python 3.3. I try to keep that in sync with the stdlib on all platforms, otherwise figuring out whether any changes need to be packported is a nightmare.

    You might also update the comment somewhere that currently states that reuse_port is not supported on Windows.

    PS. It amuses me that each platform apparently has a different way to check (there are three different types of platform checks in the lines you change).

    @yan12125
    Copy link
    Mannequin

    yan12125 mannequin commented May 1, 2016

    Where is the patch that adds android_ver()?

    http://bugs.python.org/issue26855

    Xavier de Gaye's implementation is buggy while works for most cases.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 2, 2016

    The patch was not intended to be proposed as a way to fix this problem, but as an indication of what the problem is. This issue is a dependency of issue bpo-26865: Meta-issue: support of the android platform.

    I should have posted instead the result of running 'python -m test -v test_asyncio' on an android emulator running an x86 system image at API level 21, which is:

    ======================================================================
    ERROR: test_create_datagram_endpoint_sockopts (test.test_asyncio.test_base_events.BaseEventLoopWithS
    electorTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_asyncio/test_base_events.py", line 1506,
     in test_create_datagram_endpoint_sockopts
        socket.SOL_SOCKET, socket.SO_REUSEPORT))
    OSError: [Errno 92] Protocol not available

    ======================================================================
    ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.EPollEventLoopTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_asyncio/test_events.py", line 830, in te
    st_create_server_reuse_port
        socket.SOL_SOCKET, socket.SO_REUSEPORT))
    OSError: [Errno 92] Protocol not available

    ======================================================================
    ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.PollEventLoopTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_asyncio/test_events.py", line 830, in te
    st_create_server_reuse_port
        socket.SOL_SOCKET, socket.SO_REUSEPORT))
    OSError: [Errno 92] Protocol not available

    ======================================================================
    ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.SelectEventLoopTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_asyncio/test_events.py", line 830, in te
    st_create_server_reuse_port
        socket.SOL_SOCKET, socket.SO_REUSEPORT))
    OSError: [Errno 92] Protocol not available

    Ran 991 tests in 16.528s

    FAILED (errors=4, skipped=2)
    test test_asyncio failed
    1 test failed:
    test_asyncio
    Total duration: 0:00:18

    @gvanrossum
    Copy link
    Member

    gvanrossum commented May 2, 2016

    OK, I'll wait until Android support is closer. Do you have a core dev who's mentoring/reviewing here?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 2, 2016

    Stefan has already commited and closed some of the issues listed at issue bpo-26865. Martin has closed issue bpo-22359, a blocker for the cross-compilation of python.

    I will enter an issue for all the failures listed at https://bitbucket.org/xdegaye/pyona/wiki/testsuite and add them to the list in the android meta-issue.

    @gvanrossum
    Copy link
    Member

    gvanrossum commented May 2, 2016

    If there are a lot of Android issue maybe you could get a tracker manager
    to add a new keyword or category or whatever?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 3, 2016

    Running the "Search" button on the tracker with "Components" set as "Cross-Build" lists 42 issues (20 in the last year).

    I guess using this existing category is fine if the titles' issues are prefixed with "android:".

    @xdegaye xdegaye mannequin changed the title setting SO_REUSEPORT fails on android android: setting SO_REUSEPORT fails May 3, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 20, 2016

    Should use boolean 'is_android' being added to module test.support by issue bpo-27027, instead of platform.android_ver()[0].

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 22, 2016

    New patch test.asyncio_2.patch uses test.support.is_android and adds a dependency to issue bpo-27027.

    @1st1
    Copy link
    Member

    1st1 commented Sep 15, 2016

    This issue was fixed in bpo-28174.

    @1st1 1st1 closed this as completed Sep 15, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jan 13, 2017

    The changes made in issue bpo-28174 fix the problem for the Android x86 platform and for the armv7 platform at Android API level 24, but the problem is still there on the armv7 platform at Android API level 21.

    ======================================================================
    ERROR: test_create_datagram_endpoint_sockopts (test.test_asyncio.test_base_events.BaseEventLoopWithS
    electorTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_asyncio/test_base_events.py", line 1593,
     in test_create_datagram_endpoint_sockopts
        socket.SOL_SOCKET, socket.SO_REUSEPORT))
    OSError: [Errno 92] Protocol not available

    ======================================================================
    ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.EPollEventLoopTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_asyncio/test_events.py", line 913, in te
    st_create_server_reuse_port
        socket.SOL_SOCKET, socket.SO_REUSEPORT))
    OSError: [Errno 92] Protocol not available

    ======================================================================
    ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.PollEventLoopTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_asyncio/test_events.py", line 913, in te
    st_create_server_reuse_port
        socket.SOL_SOCKET, socket.SO_REUSEPORT))
    OSError: [Errno 92] Protocol not available

    ======================================================================
    ERROR: test_create_server_reuse_port (test.test_asyncio.test_events.SelectEventLoopTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_asyncio/test_events.py", line 913, in te
    st_create_server_reuse_port
        socket.SOL_SOCKET, socket.SO_REUSEPORT))
    OSError: [Errno 92] Protocol not available

    Ran 1476 tests in 382.973s

    FAILED (errors=4, skipped=2)
    test test_asyncio failed
    test_asyncio failed in 7 min 25 sec

    1 test failed:
    test_asyncio

    Total duration: 7 min 27 sec
    Tests result: FAILURE

    @xdegaye xdegaye mannequin added expert-asyncio 3.7 and removed build The build process and cross-build labels Jan 13, 2017
    @xdegaye xdegaye mannequin reopened this Jan 13, 2017
    @vstinner
    Copy link
    Member

    vstinner commented Jan 13, 2017

    IMHO it's ok to skip a few asyncio tests on Android. It's safe if it doesn't touch the code of asyncio itself.

    It don't really care if the Android version is not checked yet. Moreover, if we really want to check the version, we should check the *runtime* verison, not the *build* version of Android. We need the discussed platform.android_ver() function which doesn't exist yet. So it's fine to postpone the version check in asyncio tests.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jan 13, 2017

    Skipping those tests on Android has already been proposed with the existing patches in this issue.
    The reason why the changes made by bpo-28174 fix only partly the problem should be understood.

    @vstinner
    Copy link
    Member

    vstinner commented Jan 13, 2017

    The reason why the changes made by bpo-28174 fix only partly the problem should be understood.

    The change c1c247cf3488 catchs OSError on sock.setsockopt() in asyncio/base_events.py, whereas the test fails while calling sock.getsockopt() in test_asyncio/tets_events.py. Extract of the test:

        def test_create_server_reuse_port(self):
            proto = MyProto(self.loop)
            f = self.loop.create_server(
                lambda: proto, '0.0.0.0', 0)
            server = self.loop.run_until_complete(f)
            self.assertEqual(len(server.sockets), 1)
            sock = server.sockets[0]
            self.assertFalse(
                sock.getsockopt(
                    socket.SOL_SOCKET, socket.SO_REUSEPORT))
            server.close()

    The change c1c247cf3488 doesn't touch the code of this unit test.

    test_base_events.py also calls directly sock.getsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT).

    @pjenvey
    Copy link
    Member

    pjenvey commented Sep 16, 2017

    Note that this isn't an android specific issue: See bpo-19901, and 9791c5d55f52 for an appropriate solution patch.

    (Renaming title)

    @pjenvey pjenvey changed the title android: setting SO_REUSEPORT fails setting SO_REUSEPORT fails Sep 16, 2017
    @gvanrossum
    Copy link
    Member

    gvanrossum commented Sep 17, 2017

    Can't you just submit a PR? It's so easy now!

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Oct 21, 2017

    Philip, 9791c5d55f52 was commited in november 2013 and bpo-19901 was closed in december 2013, so how can they be an appropriate solution for a problem reported on 2017-01-13 in msg285392 ?

    @gvanrossum
    Copy link
    Member

    gvanrossum commented Oct 21, 2017

    I assume he means that a similar piece of code should be inserted elsewhere. I am still waiting for Philip's PR though.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 10, 2017

    This problem does not occur at API 24.

    @xdegaye xdegaye mannequin changed the title setting SO_REUSEPORT fails setting SO_REUSEPORT fails at API 21 Dec 10, 2017
    @vstinner
    Copy link
    Member

    vstinner commented Dec 10, 2017

    Do we have a function in Python to get the running Android version?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 10, 2017

    This is bpo-26855.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @kumaraditya303
    Copy link
    Contributor

    kumaraditya303 commented Jul 9, 2022

    Closing as android is not supported, if you want to work on this, then create a new issue targeting main branch and the patch would have to be PRs.

    @kumaraditya303 kumaraditya303 closed this as not planned Won't fix, can't repro, duplicate, stale Jul 9, 2022
    @gvanrossum
    Copy link
    Member

    gvanrossum commented Jul 9, 2022

    So are we now going to close all issues listed in #71052 as “Android not supported”? That seems backwards, since the (laudable) goal is to support Android.

    @kumaraditya303
    Copy link
    Contributor

    kumaraditya303 commented Jul 9, 2022

    So are we now going to close all issues listed in #71052 as “Android not supported”? That seems backwards, since the (laudable) goal is to support Android.

    No atleast I won't, I only close issue which have old patches and no comments in last 2~3 years and patch does not apply cleanly and platform is unsupported. I try patch before closing.

    error: patch failed: Lib/test/test_asyncio/test_base_events.py:1505
    error: Lib/test/test_asyncio/test_base_events.py: patch does not apply
    error: patch failed: Doc/library/asyncio-eventloop.rst:345
    error: Doc/library/asyncio-eventloop.rst: patch does not apply

    @kumaraditya303
    Copy link
    Contributor

    kumaraditya303 commented Jul 9, 2022

    If anyone wants to port CPython to other platforms then IMO they should target main branch and land patches in form of PRs otherwise the old patches would conflicts and get out of date.

    @gvanrossum
    Copy link
    Member

    gvanrossum commented Jul 9, 2022

    Okay, just making clear what you're doing. That sounds like a reasonable process.

    @pmp-p
    Copy link
    Contributor

    pmp-p commented Aug 4, 2022

    Closing dangling issues is really not a problem if at least a specifig tag ( aosp? ) is added to those, so people that may have time to work on can find them easy.
    ( i think i could but i'm not triager )

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 expert-asyncio type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants