Test failure with IPython 7

[toddrme2178]

I am trying to package IPython 7 for openSUSE and we are getting the following build error when building nbconvert against it:

=================================== FAILURES ===================================
________________________ TestExecute.test_run_notebooks ________________________

self = <nbconvert.preprocessors.tests.test_execute.TestExecute testMethod=test_run_notebooks>

    def test_run_notebooks(self):
        """Runs a series of test notebooks and compares them to their actual output"""
        input_files = glob.glob(os.path.join(current_dir, 'files', '*.ipynb'))
        shared_opts = dict(kernel_name="python")
        for filename in input_files:
            if os.path.basename(filename) == "Disable Stdin.ipynb":
                continue
            elif os.path.basename(filename) == "Interrupt.ipynb":
                opts = dict(timeout=1, interrupt_on_timeout=True, allow_errors=True)
            elif os.path.basename(filename) == "Skip Exceptions.ipynb":
                opts = dict(allow_errors=True)
            else:
                opts = dict()
            res = self.build_resources()
            res['metadata']['path'] = os.path.dirname(filename)
            opts.update(shared_opts)
            input_nb, output_nb = self.run_notebook(filename, opts, res)
>           self.assert_notebooks_equal(input_nb, output_nb)

../../../BUILDROOT/python-jupyter_nbconvert-5.4.0-15.18.x86_64/usr/lib/python3.6/site-packages/nbconvert/preprocessors/tests/test_execute.py:139: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../BUILDROOT/python-jupyter_nbconvert-5.4.0-15.18.x86_64/usr/lib/python3.6/site-packages/nbconvert/preprocessors/tests/test_execute.py:80: in assert_notebooks_equal
    self.assertEqual(normalized_expected_outputs, normalized_actual_outputs)
E   AssertionError: Lists differ: [{'ename': 'KeyboardInterrupt', 'evalue': ''[278 chars] ']}] != [{'output_type': 'error', 'ename': 'Keyboard[276 chars] ']}]
E   
E   First differing element 0:
E   {'ename': 'KeyboardInterrupt', 'evalue': ''[277 chars]: ']}
E   {'output_type': 'error', 'ename': 'Keyboard[275 chars]: ']}
E   
E     [{'ename': 'KeyboardInterrupt',
E       'evalue': '',
E       'output_type': 'error',
E       'traceback': ['---------------------------------------------------------------------------',
E                     'KeyboardInterrupt                         Traceback (most '
E                     'recent call last)',
E   -                 '<IPY-INPUT> in <module>()\n----> 1 while True: continue\n',
E   ?                                         --
E   
E   +                 '<IPY-INPUT> in <module>\n----> 1 while True: continue\n',
E                     'KeyboardInterrupt: ']}]
------------------------------ Captured log call -------------------------------
execute.py                 426 ERROR    Timeout waiting for execute reply (1s).
execute.py                 428 ERROR    Interrupting kernel

[takluyver]

I just came across this and fixed it as an incidental part of #809 .

[costrouc]

Is this related? I am packaging for ipython 7 for nixpkgs and I ran into the exact issue with a slight different test failure but similar enough.

________________________ TestExecute.test_run_notebooks ________________________

self = <nbconvert.preprocessors.tests.test_execute.TestExecute testMethod=test_run_notebooks>

    def test_run_notebooks(self):
        """Runs a series of test notebooks and compares them to their actual output"""
        input_files = glob.glob(os.path.join(current_dir, 'files', '*.ipynb'))
        shared_opts = dict(kernel_name="python")
        for filename in input_files:
            if os.path.basename(filename) == "Disable Stdin.ipynb":
                continue
            elif os.path.basename(filename) == "Interrupt.ipynb":
                opts = dict(timeout=1, interrupt_on_timeout=True, allow_errors=True)
            elif os.path.basename(filename) == "Skip Exceptions.ipynb":
                opts = dict(allow_errors=True)
            else:
                opts = dict()
            res = self.build_resources()
            res['metadata']['path'] = os.path.dirname(filename)
            opts.update(shared_opts)
            input_nb, output_nb = self.run_notebook(filename, opts, res)
>           self.assert_notebooks_equal(input_nb, output_nb)

nbconvert/preprocessors/tests/test_execute.py:139:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
nbconvert/preprocessors/tests/test_execute.py:80: in assert_notebooks_equal
    self.assertEqual(normalized_expected_outputs, normalized_actual_outputs)
E   AssertionError: Lists differ: [{'ename': 'Exception', 'evalue': 'message',[301 chars]e']}] != [{'output_type': 'error', 'ename': 'Exceptio[299 chars]e']}]
E
E   First differing element 0:
E   {'ename': 'Exception', 'evalue': 'message',[300 chars]ge']}
E   {'output_type': 'error', 'ename': 'Exceptio[298 chars]ge']}
E
E     [{'ename': 'Exception',
E       'evalue': 'message',
E       'output_type': 'error',
E       'traceback': ['---------------------------------------------------------------------------',
E                     'Exception                                 Traceback (most '
E                     'recent call last)',
E   -                 '<IPY-INPUT> in <module>()\n'
E   ?                                         --
E
E   +                 '<IPY-INPUT> in <module>\n'
E                     '      1 # üñîçø∂é\n'
E                     '----> 2 raise Exception("message")\n',
E                     'Exception: message']}]

[takluyver]

Yup, the traceback format changed slightly in IPython 7. Unfortunately nbconvert has a few tests which are checking against saved notebooks created with earlier versions.

[toddrme2178]

So what is the best approach for downstream packagers? I would prefer to not disable tests entirely if I can avoid it.

[takluyver]

If you will only be building against IPython >= 7, you could apply this patch:

93e8d64

[toddrme2178]

@takluyver We need to support both python2 and python3 for the time being so we need something that will work for both IPython 7 and IPython 5.

[takluyver]

I suspect skipping the relevant tests is the only straightforward option. This is why I dislike tests based on comparing output against a reference.

…

On Fri, 16 Nov 2018, 7:47 p.m. Todd ***@***.*** wrote: @takluyver <https://github.com/takluyver> We need to support both python2 and python3 for the time being so we need something that will work for both IPython 7 and IPython 5. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#898 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUA9bzasIRrakDA55T9u9b8yxr28bl0ks5uvxZrgaJpZM4Xddkg> .

[MSeal]

Closing as this is unrelated to nbconvert directly and had a work-around posted.