Tests fail on 1-core machine

[bmwiedemann]

While working on reproducible builds for openSUSE, I found that our python-nbclient package fails tests on a 1-core VM

Is this a bug or expected behaviour?
jmcarpenter2/swifter#102 solved a similar issue by skipping such tests when not enough cores were available.

to reproduce on openSUSE or Debian:

osc co openSUSE:Factory/python-nbclient && cd $_
osc build --vm-type=kvm -j1 --noservice standard

The issue can also be triggered via taskset 1 pytest-3.8

example failure log:

 nbclient/tests/test_client.py:270: 
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 
 expected = {'cells': [{'cell_type': 'code', 'execution_count': 1, 'metadata': {'tags': ['raises-exception']}, 'outputs': [{'ename...utput_type': 'stream', 'text': 'ok\n'}], 'source': "print('ok')"}], 'metadata': {}, 'nbformat': 4, 'nbformat_minor': 1}
 actual = {'cells': [{'cell_type': 'code', 'metadata': {'tags': ['raises-exception'], 'execution': {'iopub.status.busy': '2020-0...ents_lexer': 'ipython3', 'nbconvert_exporter': 'python', 'file_extension': '.py'}}, 'nbformat': 4, 'nbformat_minor': 1}
 
     def assert_notebooks_equal(expected, actual):
         expected_cells = expected['cells']
         actual_cells = actual['cells']
         assert len(expected_cells) == len(actual_cells)
     
         for expected_cell, actual_cell in zip(expected_cells, actual_cells):
             # Uncomment these to help debug test failures better
             # from pprint import pprint
             # pprint(expected_cell)
             # pprint(actual_cell)
             expected_outputs = expected_cell.get('outputs', [])
             actual_outputs = actual_cell.get('outputs', [])
             normalized_expected_outputs = list(map(normalize_output, expected_outputs))
             normalized_actual_outputs = list(map(normalize_output, actual_outputs))
 >           assert normalized_expected_outputs == normalized_actual_outputs
 E           AssertionError: assert [{'name': 'st...ext': 'ok\n'}] == []
 E             Left contains one more item: {'name': 'stdout', 'output_type': 'stream', 'text': 'ok\n'}
 E             Full diff:
 E             - []
 E             + [{'name': 'stdout', 'output_type': 'stream', 'text': 'ok\n'}]
 
 nbclient/tests/test_client.py:228: AssertionError
 ____________________ TestExecute.test_cleanup_kernel_client ____________________
 
 self = <nbclient.tests.test_client.TestExecute testMethod=test_cleanup_kernel_client>
 
     def test_cleanup_kernel_client(self):
         filename = os.path.join(current_dir, 'files', 'HelloWorld.ipynb')
     
         with io.open(filename) as f:
             input_nb = nbformat.read(f, 4)
     
         executor = NotebookClient(
             input_nb,
             resources=self.build_resources(),
         )
     
         executor.execute()
         # we asked to cleanup the kernel client (default is True)
         assert executor.kc is None
     
 >       executor.execute(cleanup_kc=False)

[MSeal]

That's definitely not an expected behavior. I take it your build chain with 2 cores is passing (to eliminate possibility of it being dependency related rather than hardware)? Are you running the test suite sequentially or in parallel with other tests? If you can modify the source running uncommenting those debug lines above the stack trace might be helpful to see if it is one missed message or something more systemic.

[bmwiedemann]

just re-tested and found that there are also issues with 2 and 4 cores :-(

FAILED nbclient/tests/test_client.py::TestExecute::test_cleanup_kernel_client
FAILED nbclient/tests/test_client.py::TestExecute::test_reset_kernel_client

we are on 0.4.0 - I can also try to go to 0.4.3 first.

[bmwiedemann]

0.4.3 looks somewhat better. there, 2-core and 4-core runs passed and 1-core tests had just this 1 failure:

 =================================== FAILURES ===================================
 ________________________ test_async_parallel_notebooks _________________________
 
 capfd = <_pytest.capture.CaptureFixture object at 0x7f16c8b959d0>
 tmpdir = local('/tmp/pytest-of-abuild/pytest-0/test_async_parallel_notebooks0')
 
     def test_async_parallel_notebooks(capfd, tmpdir):
         """Two notebooks should be able to be run simultaneously without problems.
     
         The two notebooks spawned here use the filesystem to check that the other notebook
         wrote to the filesystem."""
     
         opts = dict(kernel_name="python")
         input_name = "Parallel Execute {label}.ipynb"
         input_file = os.path.join(current_dir, "files", input_name)
         res = notebook_resources()
     
         with modified_env({"NBEXECUTE_TEST_PARALLEL_TMPDIR": str(tmpdir)}):
             tasks = [
                 async_run_notebook(input_file.format(label=label), opts, res)
                 for label in ("A", "B")
             ]
             loop = asyncio.get_event_loop()
 >           loop.run_until_complete(asyncio.gather(*tasks))
 
 nbclient/tests/test_client.py:340: 
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 /usr/lib64/python3.8/asyncio/base_events.py:616: in run_until_complete
     return future.result()
 nbclient/tests/test_client.py:112: in async_run_notebook
     output_nb = await executor.async_execute()
 nbclient/client.py:535: in async_execute
     await self.async_execute_cell(
 nbclient/client.py:827: in async_execute_cell
     self._check_raise_for_error(cell, exec_reply)
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 
 self = <nbclient.client.NotebookClient object at 0x7f16c4306040>
 cell = {'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2035-10-28T18:12:31.520994Z', 'iopub.execute_in...nelse:\n    assert False, "Timed out –\xa0didn\'t get a message from {}".format(other_notebook)', 'execution_count': 3}
 exec_reply = {'buffers': [], 'content': {'ename': 'AssertionError', 'engine_info': {'engine_id': -1, 'engine_uuid': '9e75de9b-82a2-...e, 'engine': '9e75de9b-82a2-465e-b2b7-07eab5d42c43', 'started': '2035-10-28T18:12:31.509561Z', 'status': 'error'}, ...}
 
     def _check_raise_for_error(
             self,
             cell: NotebookNode,
             exec_reply: t.Optional[t.Dict]) -> None:
     
         cell_allows_errors = self.allow_errors or "raises-exception" in cell.metadata.get(
             "tags", []
         )
     
         if self.force_raise_errors or not cell_allows_errors:
             if (exec_reply is not None) and exec_reply['content']['status'] == 'error':
 >               raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
 E               nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
 E               ------------------
 E               start = time.time()
 E               timeout = 5
 E               end = start + timeout
 E               target_file = os.path.join(directory, 'test_file_{}.txt'.format(other_notebook))
 E               while time.time() < end:
 E                   time.sleep(0.1)
 E                   if os.path.exists(target_file):
 E                       with open(target_file, 'r') as f:
 E                           text = f.read()
 E                       if text == 'Hello from {}'.format(other_notebook):
 E                           break
 E               else:
 E                   assert False, "Timed out – didn't get a message from {}".format(other_notebook)
 E               ------------------
 E               
 E               ---------------------------------------------------------------
 E               AssertionError                            Traceback (most recent call last)
 E               <ipython-input-3-b099cd1c75aa> in <module>
 E                    11             break
 E                    12 else:
 E               ---> 13     assert False, "Timed out – didn't get a message from {}".format(other_notebook)
 E               
 E               AssertionError: Timed out – didn't get a message from B
 E               AssertionError: Timed out – didn't get a message from B
 
 nbclient/client.py:735: CellExecutionError
 =============================== warnings summary ===============================
 nbclient/tests/test_client.py::test_startnewkernel_with_kernelmanager
 nbclient/tests/test_client.py::test_start_new_kernel_history_file_setting
   /usr/lib/python3.8/site-packages/jupyter_client/manager.py:362: FutureWarning: Method cleanup(connection_file=True) is deprecated, use cleanup_resources(restart=False).
     warnings.warn("Method cleanup(connection_file=True) is deprecated, use cleanup_resources(restart=False).",
 
 nbclient/tests/test_client.py::TestRunCell::test_deadline_iopub
   /usr/lib64/python3.8/unittest/mock.py:740: RuntimeWarning: coroutine 'make_async.<locals>._' was never awaited
     return object.__setattr__(self, name, value)
 
 nbclient/tests/test_client.py::TestRunCell::test_display_data_message
 nbclient/tests/test_client.py::TestRunCell::test_raises_exception_tag
   /usr/lib64/python3.8/unittest/mock.py:2076: RuntimeWarning: coroutine 'make_async.<locals>._' was never awaited
     self.name = name
 
 nbclient/tests/test_client.py::TestRunCell::test_eventual_deadline_iopub
   /home/abuild/rpmbuild/BUILD/nbclient-0.4.3/nbclient/tests/test_client.py:843: RuntimeWarning: coroutine 'make_async.<locals>._' was never awaited
     message_mock.side_effect = message_seq(list(message_mock.side_effect)[:-1])
 
 nbclient/tests/test_client.py::TestRunCell::test_eventual_deadline_iopub
   /usr/lib64/python3.8/unittest/mock.py:773: RuntimeWarning: coroutine 'make_async.<locals>._' was never awaited
     return object.__setattr__(self, name, value)
 
 nbclient/tests/test_client.py::TestRunCell::test_execute_result_with_display_message
   /usr/lib64/python3.8/enum.py:309: RuntimeWarning: coroutine 'make_async.<locals>._' was never awaited
     return cls.__new__(cls, value)
 
 -- Docs: https://docs.pytest.org/en/stable/warnings.html
 =========================== short test summary info ============================
 FAILED nbclient/tests/test_client.py::test_async_parallel_notebooks - nbclien...
 ====== 1 failed, 71 passed, 1 deselected, 8 warnings in 129.20s (0:02:09) ======

[bmwiedemann]

Also tested 0.5.0 and in 1 of 2 tries, I got the test_async_parallel_notebooks Timeout.

[MSeal]

test_async_parallel_notebooks has a known issue rooted in jupyter_client that is unresolved: jupyter/jupyter_client#487. Basically there's a race condition that test are sometimes loosing for port aquisition.