Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-35934: Add socket.create_server() utility function #11784

Merged
merged 46 commits into from
Apr 8, 2019
Merged

Conversation

giampaolo
Copy link
Contributor

@giampaolo giampaolo commented Feb 7, 2019

Copy link
Contributor

@asvetlov asvetlov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the proposal but have a few comments regarding implementation.

Doc/library/socket.rst Outdated Show resolved Hide resolved
Lib/socket.py Outdated Show resolved Hide resolved
Lib/socket.py Outdated Show resolved Hide resolved
Lib/socket.py Outdated Show resolved Hide resolved
Lib/socket.py Outdated Show resolved Hide resolved
Lib/socket.py Outdated Show resolved Hide resolved
Copy link
Contributor Author

@giampaolo giampaolo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added reuse_addr parameter

Lib/socket.py Outdated Show resolved Hide resolved
Lib/socket.py Outdated Show resolved Hide resolved
Lib/socket.py Outdated Show resolved Hide resolved
Doc/library/socket.rst Outdated Show resolved Hide resolved
Lib/socket.py Outdated Show resolved Hide resolved
Doc/library/socket.rst Outdated Show resolved Hide resolved
@vstinner
Copy link
Member

Oh nice! I assumed it was not supported and didn't investigate further because on Windows socket.IPPROTO_IPV6 constant is missing.

Hum, the constant is used in the following example:
https://stackoverflow.com/questions/37729475/create-dual-stack-socket-on-all-loopback-interfaces-on-windows

Do we miss some header files?

@giampaolo
Copy link
Contributor Author

Yes it appears we miss some constants on Windows. That is being tracked here:
https://bugs.python.org/issue29515

@giampaolo giampaolo requested review from asvetlov, 1st1, vadmium, njsmith, jaraco and vstinner and removed request for njsmith and jaraco March 5, 2019 23:29
@@ -595,6 +595,53 @@ The following functions all create :ref:`socket objects <socket-objects>`.
.. versionchanged:: 3.2
*source_address* was added.

.. function:: create_server(address, *, family=AF_INET, backlog=0, reuse_port=False, dualstack_ipv6=False)

Convenience function which creates a :data:`SOCK_STREAM` type socket
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason why you don't just say it creates a TCP socket? I think this is guaranteed by Posix for AF_INET and AF_INET6 with SOCK_STREAM and proto set to zero.

Doc/library/socket.rst Outdated Show resolved Hide resolved
.. function:: has_dualstack_ipv6()

Return ``True`` if the platform supports creating a :data:`SOCK_STREAM`
socket which can handle both :data:`AF_INET` or :data:`AF_INET6`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think AF_INET and AF_INET6 are mutually exclusive for a given socket. Perhaps it is more accurate to say the socket can handle both IP v4 and v6.

self.listening_sock.bind(address)
self.listening_sock.listen(1)
self.listening_sock = socket.create_server(
address, family=family, type=type, backlog=1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is type a valid keyword? If not, maybe this needs a test case :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ouch! I missed that, thanks. I just removed this part and left idlelib alone in f786884. I will probably get back at this in a separate PR because it's more controversial.

Lib/socket.py Outdated
raise error(err.errno, msg) from None
sock.listen(backlog)
return sock
except Exception:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use plain except to catch all exceptions (e.g. KeyboardInterrupt could be raised during a slow DNS lookup)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I think it's saner to just catch socket.error, the same way it's done in create_connection(). Changed it in f786884.

with sock:
conn, _ = sock.accept()
with conn:
event.wait(self.timeout)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this event necessary? It is set just after the thread is started.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it is because you want to wait for accept() to succeed before doing connect(). Without this sync primitive in place the test will fail.

@vstinner
Copy link
Member

vstinner commented Mar 6, 2019

@giampaolo: Sorry, I don't have the bandwidth right now to review this PR.

@giampaolo
Copy link
Contributor Author

@vstinner it's ok, we still depend on https://bugs.python.org/issue29515 which should be fixed first.

@giampaolo
Copy link
Contributor Author

OK, I just committed a fix for https://bugs.python.org/issue29515 which adds IPPROTO_IPV6 constant on Windows. Since dual-stack is supported on Windows I removed the reference to the external recipe from the doc (at this point I am not aware of any platform NOT supporting dual-stack). I think this is good to go but I will wait one more week before merging just in case somebody else wants to comment. Thanks everybody for the great review and suggestions without which this couldn't have happened.

@giampaolo giampaolo merged commit eb7e29f into master Apr 8, 2019
@giampaolo giampaolo deleted the bind-socket branch April 8, 2019 22:34
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x SLES 3.x has failed when building commit eb7e29f.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/16/builds/2625) and take a look at the build logs.
  4. Check if the failure is related to this commit (eb7e29f) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/16/builds/2625

Click to see traceback logs
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/_test_multiprocessing.py", line 4377, in test_wait_socket
    self.assertEqual(b''.join(v), expected)
AssertionError: b'1\n2\n3\n4\n5\n6\n7\n8\n9\n' != b'0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n'

----------------------------------------------------------------------

Ran 344 tests in 65.088s

FAILED (failures=1, skipped=33)


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/_test_multiprocessing.py", line 4377, in test_wait_socket
    self.assertEqual(b''.join(v), expected)
AssertionError: b'1\n2\n3\n4\n5\n6\n7\n8\n9\n' != b'0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n'

----------------------------------------------------------------------

Ran 344 tests in 109.525s

FAILED (failures=1, skipped=32)


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/_test_multiprocessing.py", line 4377, in test_wait_socket
    self.assertEqual(b''.join(v), expected)
AssertionError: b'1\n2\n3\n4\n5\n6\n7\n8\n9\n' != b'0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n'

----------------------------------------------------------------------

Ran 344 tests in 67.728s

FAILED (failures=1, skipped=33)
Re-running test 'test_multiprocessing_spawn' in verbose mode

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x SLES 3.x has failed when building commit eb7e29f.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/16/builds/2626) and take a look at the build logs.
  4. Check if the failure is related to this commit (eb7e29f) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/16/builds/2626

Click to see traceback logs
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/_test_multiprocessing.py", line 4377, in test_wait_socket
    self.assertEqual(b''.join(v), expected)
AssertionError: b'1\n2\n3\n4\n5\n6\n7\n8\n9\n' != b'0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n'

----------------------------------------------------------------------

Ran 344 tests in 76.263s

FAILED (failures=1, skipped=29)


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/_test_multiprocessing.py", line 4377, in test_wait_socket
    self.assertEqual(b''.join(v), expected)
AssertionError: b'1\n2\n3\n4\n5\n6\n7\n8\n9\n' != b'0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n'

----------------------------------------------------------------------

Ran 344 tests in 75.618s

FAILED (failures=1, skipped=29)

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD CURRENT Shared 3.x has failed when building commit eb7e29f.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/168/builds/885) and take a look at the build logs.
  4. Check if the failure is related to this commit (eb7e29f) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/168/builds/885

Click to see traceback logs
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/_test_multiprocessing.py", line 4337, in _child_test_wait_socket
    s.connect(address)
ConnectionResetError: [Errno 54] Connection reset by peer
/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py:318: ResourceWarning: unclosed <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 64392)>
  traceback.print_exc()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Timeout (0:25:00)!
Thread 0x0000000800acc000 (most recent call first):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/socket.py", line 212 in accept
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/_test_multiprocessing.py", line 4361 in test_wait_socket
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/_test_multiprocessing.py", line 4383 in test_wait_socket_slow
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/case.py", line 680 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/case.py", line 740 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/runner.py", line 176 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/support/__init__.py", line 1968 in _run_suite
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/support/__init__.py", line 2064 in run_unittest
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest.py", line 178 in test_runner
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest.py", line 182 in runtest_inner
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest.py", line 127 in runtest
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 68 in run_tests_worker
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 600 in _main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 586 in main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 640 in main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/regrtest.py", line 46 in _main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/regrtest.py", line 50 in <module>
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/runpy.py", line 85 in _run_code
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/runpy.py", line 192 in _run_module_as_main
running: test_multiprocessing_fork (13 min 3 sec)
running: test_multiprocessing_fork (13 min 33 sec)
running: test_multiprocessing_fork (14 min 3 sec)
running: test_multiprocessing_fork (14 min 33 sec)
running: test_multiprocessing_fork (15 min 3 sec)
running: test_multiprocessing_fork (15 min 34 sec)
running: test_multiprocessing_fork (16 min 4 sec)
running: test_multiprocessing_fork (16 min 34 sec)
running: test_multiprocessing_fork (17 min 4 sec)
running: test_multiprocessing_fork (17 min 34 sec)
running: test_multiprocessing_fork (18 min 4 sec)
running: test_multiprocessing_fork (18 min 34 sec)
running: test_multiprocessing_fork (19 min 4 sec)
running: test_multiprocessing_fork (19 min 34 sec)
running: test_multiprocessing_fork (20 min 4 sec)
running: test_multiprocessing_fork (20 min 34 sec)
running: test_multiprocessing_fork (21 min 4 sec)
running: test_multiprocessing_fork (21 min 34 sec)
running: test_multiprocessing_fork (22 min 5 sec)
running: test_multiprocessing_fork (22 min 35 sec)
running: test_multiprocessing_fork (23 min 5 sec)
running: test_multiprocessing_fork (23 min 35 sec)
running: test_multiprocessing_fork (24 min 5 sec)
running: test_multiprocessing_fork (24 min 35 sec)


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/_test_multiprocessing.py", line 4337, in _child_test_wait_socket
    s.connect(address)
ConnectionResetError: [Errno 54] Connection reset by peer
/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py:318: ResourceWarning: unclosed <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 2397)>
  traceback.print_exc()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Process Process-60:
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/_test_multiprocessing.py", line 4337, in _child_test_wait_socket
    s.connect(address)
ConnectionResetError: [Errno 54] Connection reset by peer
/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py:318: ResourceWarning: unclosed <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 2398)>
  traceback.print_exc()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Process Process-59:
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/_test_multiprocessing.py", line 4337, in _child_test_wait_socket
    s.connect(address)
ConnectionResetError: [Errno 54] Connection reset by peer
/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/multiprocessing/process.py:318: ResourceWarning: unclosed <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 2399)>
  traceback.print_exc()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Timeout (0:25:00)!
Thread 0x0000000800acc000 (most recent call first):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/socket.py", line 212 in accept
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/_test_multiprocessing.py", line 4361 in test_wait_socket
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/case.py", line 680 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/case.py", line 740 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/unittest/runner.py", line 176 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/support/__init__.py", line 1968 in _run_suite
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/support/__init__.py", line 2064 in run_unittest
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest.py", line 178 in test_runner
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest.py", line 182 in runtest_inner
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest.py", line 137 in runtest
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 305 in rerun_failed_tests
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 623 in _main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 586 in main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 640 in main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/__main__.py", line 2 in <module>
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/runpy.py", line 85 in _run_code
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/runpy.py", line 192 in _run_module_as_main

giampaolo added a commit that referenced this pull request Apr 9, 2019
It turns out doing socket.listen(0) does not equal to "choose a
reasonable default". It actually means "set backlog to 0".
As such set backlog=None as the default for socket.create_server.
Fixes the following BB failures:
#11784 (comment)
Ref. BPO-1756, GH-11784.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-feature A feature request or enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.