test_asyncio: test_start_tls_server_1() fails on Python on x86 Windows7 3.7 and 3.x

[vstinner]

BPO	33694
Nosy	@vstinner, @ned-deily, @miss-islington
PRs	WIP: bpo-33694, asyncio: fix proactor transport pause/resume #7479 bpo-33694: asyncio: Fix race in Proactor's Transport.set_protocol() #7486 bpo-33694: Fix race condition on proactor recv() #7498 [3.7] bpo-33694: Fix race condition in asyncio proactor (GH-7498) #7499 bpo-33694: Fix typo in asyncio helper function name #7522 [3.7] bpo-33694: Fix typo in helper function name (GH-7522) #7523 bpo-33694: Remove test_asyncio ProactorDatagramTransportTests #16288 [3.8] bpo-33694: Remove test_asyncio ProactorDatagramTransportTests (GH-16288) #16305
Files	Screen Shot 2018-05-30 at 11.38.38 AM.png race.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-06-09.16:44:39.217>
created_at = <Date 2018-05-30.11:19:38.205>
labels = ['3.8', '3.7', 'tests', 'expert-asyncio']
title = 'test_asyncio: test_start_tls_server_1() fails on Python on x86 Windows7 3.7 and 3.x'
updated_at = <Date 2019-09-20.21:27:31.745>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-09-20.21:27:31.745>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2018-06-09.16:44:39.217>
closer = 'vstinner'
components = ['Tests', 'asyncio']
creation = <Date 2018-05-30.11:19:38.205>
creator = 'vstinner'
dependencies = []
files = ['47626', '47632']
hgrepos = []
issue_num = 33694
keywords = ['patch']
message_count = 32.0
messages = ['318164', '318165', '318193', '318194', '318195', '318209', '318424', '318569', '318705', '318707', '318809', '318869', '318872', '318873', '318874', '318875', '318876', '318915', '318921', '318977', '318979', '318993', '319034', '319035', '319155', '319175', '319209', '319220', '319299', '319450', '352891', '352894']
nosy_count = 3.0
nosy_names = ['vstinner', 'ned.deily', 'miss-islington']
pr_nums = ['7479', '7486', '7498', '7499', '7522', '7523', '16288', '16305']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue33694'
versions = ['Python 3.7', 'Python 3.8']

[vstinner]

test_asyncio.test_start_tls_server_1() got many fixes recently: see bpo-32458 and bpo-33674... but it still fails on Python on x86 Windows7 3.x at revision bb9474f which contains all these fixes.

The test fails even when test_asyncio is re-run alone (not when other tests run in parallel).

http://buildbot.python.org/all/#/builders/58/builds/930

======================================================================
ERROR: test_start_tls_server_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_sslproto.py", line 467, in test_start_tls_server_1
    self.loop.run_until_complete(run_main())
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 566, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.

======================================================================
FAIL: test_start_tls_server_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\functional.py", line 42, in tearDown
    self.fail('unexpected calls to loop.call_exception_handler()')
AssertionError: unexpected calls to loop.call_exception_handler()

The test fails also on x86 Windows7 3.7:

======================================================================
ERROR: test_start_tls_server_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.7.bolen-windows7\build\lib\test\test_asyncio\test_sslproto.py", line 467, in test_start_tls_server_1
    self.loop.run_until_complete(run_main())
  File "D:\cygwin\home\db3l\buildarea\3.7.bolen-windows7\build\lib\asyncio\base_events.py", line 566, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.

======================================================================
FAIL: test_start_tls_server_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.7.bolen-windows7\build\lib\test\test_asyncio\functional.py", line 42, in tearDown
    self.fail('unexpected calls to loop.call_exception_handler()')
AssertionError: unexpected calls to loop.call_exception_handler()

Moreover, 3.7 got an additional failure:

======================================================================
ERROR: test_pipe_handle (test.test_asyncio.test_windows_utils.PipeTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.7.bolen-windows7\build\lib\test\test_asyncio\test_windows_utils.py", line 73, in test_pipe_handle
    raise RuntimeError('expected ERROR_INVALID_HANDLE')
RuntimeError: expected ERROR_INVALID_HANDLE

The Windows7 buildbot is known to be slow. For example, test_value() of test_multiprocessing_spawn failed with a timeout (15 min) on x86 Windows7 3.x:

0:48:12 [276/416/1] test_multiprocessing_spawn crashed (Exit code 1)
Timeout (0:15:00)!
Thread 0x000008e4 (most recent call first):
File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\popen_spawn_win32.py", line 80 in wait
File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\process.py", line 140 in join
File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\_test_multiprocessing.py", line 1925 in test_value

Maybe something is wrong with my karma, I don't know. Or test_asyncio just hate me.

[vstinner]

The test fails also on x86 Windows7 3.7:
http://buildbot.python.org/all/#/builders/111/builds/297

[1st1]

I can't reproduce test_start_tls_server_1 fails when I do (screenshot attached)

1. run test_asyncio
2. run test_asyncio.test_sslproto
3. run test_asyncio.test_sslproto -m test_start_tls_server_1

I run them in Windows 7 VM on Mac OS.

All other tests pass fine for me except test_sendfile_close_peer_in_the_middle_of_receiving. Andrew, please look at this, this is your test, and I'm not sure I follow why we use SNDBUF at all. IMO it should be possible to write a test without using that (or if not, maybe time to use mocks).

With regards to start_tls tests -- I give up, I don't have time to look into this before 3.7.0. I suggest to mask the test on the specific win buildbot it crashes on and fix it before 3.7.1 (or maybe when Windows users come up with a reliable way to reproduce the bug).

[asvetlov]

I used SNDBUF to enforce send buffer overloading.
It is not required by sendfile tests but I thought that better to have non-mocked way to test such situations.
We can remove the socket buffers size manipulation at all without any problem.

[1st1]

We can remove the socket buffers size manipulation at all without any problem.

When I tried to do that I think I was having more failures with that test. But really up to you.

[vstinner]

I used SNDBUF to enforce send buffer overloading.

What is that? Would you mind to elaborate? I'm curious :-)

[vstinner]

The test also failed on AMD64 Windows8.1 Refleaks 3.7. This buildbot is also very slow because it runs reference leak hunting. It confirms that the remaining issue is a race condition which is only seen when the system becomes slow.

http://buildbot.python.org/all/#/builders/132/builds/154

======================================================================
ERROR: test_start_tls_server_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.7.ware-win81-release.refleak\build\lib\test\test_asyncio\test_sslproto.py", line 467, in test_start_tls_server_1
    self.loop.run_until_complete(run_main())
  File "D:\buildarea\3.7.ware-win81-release.refleak\build\lib\asyncio\base_events.py", line 566, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.

======================================================================
FAIL: test_start_tls_server_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.7.ware-win81-release.refleak\build\lib\test\test_asyncio\functional.py", line 42, in tearDown
    self.fail('unexpected calls to loop.call_exception_handler()')
AssertionError: unexpected calls to loop.call_exception_handler()

[pablogsal]

The test fails also on x86 Windows7 3.x:
http://buildbot.python.org/all/#/builders/58/builds/947

[vstinner]

A variant of the bug?

AMD64 Windows8.1 Non-Debug 3.x:
http://buildbot.python.org/all/#/builders/12/builds/948

======================================================================
ERROR: test_create_connection_ssl_failed_certificate (test.test_asyncio.test_sslproto.ProactorStartTLSTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release\build\lib\test\test_asyncio\test_sslproto.py", line 623, in test_create_connection_ssl_failed_certificate
    self.loop.run_until_complete(client(srv.addr))
  File "D:\buildarea\3.x.ware-win81-release\build\lib\asyncio\base_events.py", line 566, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.

[vstinner]

AMD64 Windows8.1 Non-Debug 3.x
ERROR: test_create_connection_ssl_failed_certificate (test.test_asyncio.test_sslproto.ProactorStartTLSTests)

Not surprising: same error on AMD64 Windows8.1 Non-Debug 3.7:
http://buildbot.python.org/all/#builders/133/builds/304

[vstinner]

Another variant seen on AppVeyor (PR 7439, master branch):

https://ci.appveyor.com/project/python/cpython/build/3.8build16980

======================================================================
ERROR: test_create_connection_ssl_failed_certificate (test.test_asyncio.test_sslproto.ProactorStartTLSTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\projects\cpython\lib\test\test_asyncio\test_sslproto.py", line 623, in test_create_connection_ssl_failed_certificate
    self.loop.run_until_complete(client(srv.addr))
  File "C:\projects\cpython\lib\asyncio\base_events.py", line 566, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.

---