Skip to content
This repository

Py3 tests2 #913

Merged
merged 5 commits into from over 2 years ago

3 participants

Thomas Kluyver Min RK Fernando Perez
Thomas Kluyver
Collaborator

Fixes for a couple of tests under Python 3.

@ellisonbg, I'm having some trouble with a test in nbformat. With these changes, I get:

======================================================================
FAIL: test_roundtrip (IPython.nbformat.v2.tests.test_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thomas/Code/virtualenvs/ipy-2to3/lib/python3.2/site-packages/ipython-0.12.dev-py3.2.egg/IPython/nbformat/v2/tests/test_json.py", line 18, in test_roundtrip
    self.assertEquals(reads(s),nb0)

followed by a long dump of the before and after dicts - I can't even spot the != in there, but I think the difference is between:

{'latex': '$a$', 'prompt_number': 3, 'svg': '<svg>', 'javascript': 'var i=0;', 'jpeg': 'ZGF0YQ==\n', 'json': 'json data', 'html': 'The HTML rep', 'text': '<array a>', 'output_type': 'pyout', 'png': 'ZGF0YQ==\n'}

and (N.B. b'data'):

{'latex': '$a$', 'prompt_number': 3, 'svg': '<svg>', 'javascript': 'var i=0;', 'jpeg': b'data', 'json': 'json data', 'html': 'The HTML rep', 'text': '<array a>', 'output_type': 'pyout', 'png': b'data'}

Any immediate thoughts?

Min RK
Owner

There are definitely some bugs in the string/unicode/bytes logic in nbformat. In the JSON roundtrip, the problem is that the inverse of the b64 encode is never called. This doesn't come up, because the bytes are already b64-encoded in actual practice, and the bytes-logic in the JSONEncoder is actually never called in Python2 (as far as I can tell).

I think the real bug is that the base64_encode/decode functions in rwbase encode cell.png / cell.jpeg, but as far as I can tell, those attributes never exist. png/jpegs are attached to outputs of a cell, not the cells themselves.

This doesn't come up in Python2, because that output appears to always already be b64-encoded bytes, which pass right through to/from JSON without issue.

Thomas Kluyver
Collaborator

Is there anything simple that can be done to fix it? I'm not at all familiar with this part of the code.

Min RK
Owner

Further detail: every bytes object in the notebook is already a b64-encoded bytes object. So, instead of performing b64 encoding/decoding around JSON, it should be an ascii encoding/decoding, leaving the contents unchanged.

Min RK
Owner

Yes, I've got it working locally - the answer is to stop using b64 to/from JSON, as it has already been encoded. I'll PR to your branch shortly with fixes after I clean up all my code.

Min RK fix base64 code in nbformat.v2
base64 encoding functions were called, but had no effect, because
the notebook already has everything as b64-encoded bytestrings, which
are valid ascii literals on Python 2.

However, the encode/decode logic is actually triggered on Python 3, revealing its errors.

This fixes the base64 functions that had no effect to have their intended effect,
but does not use them.  Rather, it is assumed that
bytes objects are already b64-encoded (and thus ascii-safe), which
assumption was already made in Python 2.
155b20c
Min RK
Owner

PR issued

Thomas Kluyver
Collaborator

Thanks Min - I've added that to this PR, and the nbformat tests are passing now.

IPython/core/tests/test_magic.py
@@ -324,7 +324,10 @@ def check_cpaste(code, should_fail=False):
324 324
     _ip.user_ns['code_ran'] = False
325 325
 
326 326
     src = StringIO()
327  
-    src.encoding = None   # IPython expects stdin to have an encoding attribute
  327
+    try:
  328
+        src.encoding = None  # IPython expects stdin to have an encoding attribute
  329
+    except Exception:
  330
+        pass                 # ...but it's a read-only attribute in Python 3
2
Min RK Owner
minrk added a note October 23, 2011

Shouldn't this be 'if not hasattr', rather than try/except? We don't need to do it if it already exists.

Thomas Kluyver Collaborator

Good point. Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Thomas Kluyver
Collaborator

I've also made a couple of decorators to mark known failures, so it's easier to spot regressions. The parallel package is still failing - I'll try to look into it if I get time, but I think it's fairly low priority.

Fernando Perez
Owner

This is looking pretty solid, so I'll merge it now to avoid bitrot. When you get a chance to tackle the parallel ones, a new PR can come in for those. Thanks for this work! Looking better and better on py3 :)

Fernando Perez fperez merged commit 9463121 into from October 27, 2011
Fernando Perez fperez closed this October 27, 2011
Fernando Perez fperez referenced this pull request from a commit January 10, 2012
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 5 unique commits by 2 authors.

Oct 20, 2011
Thomas Kluyver Fix a couple of core tests under Python 3. 267e503
Thomas Kluyver Minor tweak to nbformat, still not passing test on Python 3. 38efc4e
Oct 22, 2011
Min RK fix base64 code in nbformat.v2
base64 encoding functions were called, but had no effect, because
the notebook already has everything as b64-encoded bytestrings, which
are valid ascii literals on Python 2.

However, the encode/decode logic is actually triggered on Python 3, revealing its errors.

This fixes the base64 functions that had no effect to have their intended effect,
but does not use them.  Rather, it is assumed that
bytes objects are already b64-encoded (and thus ascii-safe), which
assumption was already made in Python 2.
155b20c
Oct 26, 2011
Thomas Kluyver Tweak test for magic_cpaste. 15192a1
Thomas Kluyver Add decorators to mark known failures on Python 3. 21bd935
This page is out of date. Refresh to see the latest.
3  IPython/core/oinspect.py
@@ -31,6 +31,7 @@
31 31
 
32 32
 # IPython's own
33 33
 from IPython.core import page
  34
+from IPython.testing.skipdoctest import skip_doctest_py3
34 35
 from IPython.utils import PyColorize
35 36
 from IPython.utils import io
36 37
 from IPython.utils import py3compat
@@ -297,6 +298,8 @@ def pdef(self,obj,oname=''):
297 298
         else:
298 299
             print >>io.stdout, header,self.format(output),
299 300
 
  301
+    # In Python 3, all classes are new-style, so they all have __init__.
  302
+    @skip_doctest_py3
300 303
     def pdoc(self,obj,oname='',formatter = None):
301 304
         """Print the docstring for any object.
302 305
 
2  IPython/core/tests/test_interactiveshell.py
@@ -219,4 +219,4 @@ def tearDown(self):
219 219
     def test_1(self):
220 220
         """Test safe_execfile with non-ascii path
221 221
         """
222  
-        _ip.shell.safe_execfile(self.fname, raise_exceptions=True)
  222
+        _ip.shell.safe_execfile(self.fname, {}, raise_exceptions=True)
4  IPython/core/tests/test_magic.py
@@ -324,7 +324,9 @@ def check_cpaste(code, should_fail=False):
324 324
     _ip.user_ns['code_ran'] = False
325 325
 
326 326
     src = StringIO()
327  
-    src.encoding = None   # IPython expects stdin to have an encoding attribute
  327
+    if not hasattr(src, 'encoding'):
  328
+        # IPython expects stdin to have an encoding attribute
  329
+        src.encoding = None
328 330
     src.write('\n')
329 331
     src.write(code)
330 332
     src.write('\n--\n')
5  IPython/extensions/tests/test_autoreload.py
@@ -11,6 +11,7 @@
11 11
 
12 12
 from IPython.extensions.autoreload import AutoreloadInterface
13 13
 from IPython.core.hooks import TryNext
  14
+from IPython.testing.decorators import knownfailureif
14 15
 
15 16
 #-----------------------------------------------------------------------------
16 17
 # Test fixture
@@ -293,8 +294,12 @@ def check_module_contents():
293 294
         self.shell.run_code("pass") # trigger reload
294 295
         nt.assert_equal(mod.x, -99)
295 296
 
  297
+    # The autoreload extension needs to be updated for Python 3.2, as .pyc files
  298
+    # are stored in a different location. See gh-846.
  299
+    @knownfailureif(sys.version_info >= (3,2))
296 300
     def test_smoketest_aimport(self):
297 301
         self._check_smoketest(use_aimport=True)
298 302
 
  303
+    @knownfailureif(sys.version_info >= (3,2))
299 304
     def test_smoketest_autoreload(self):
300 305
         self._check_smoketest(use_aimport=False)
4  IPython/lib/tests/test_irunner.py
@@ -13,6 +13,7 @@
13 13
 
14 14
 # IPython imports
15 15
 from IPython.lib import irunner
  16
+from IPython.testing.decorators import known_failure_py3
16 17
 
17 18
 # Testing code begins
18 19
 class RunnerTestCase(unittest.TestCase):
@@ -56,6 +57,8 @@ def _test_runner(self,runner,source,output):
56 57
         self.assert_(mismatch==0,'Number of mismatched lines: %s' %
57 58
                      mismatch)
58 59
 
  60
+    # irunner isn't working on Python 3 (due to pexpect)
  61
+    @known_failure_py3
59 62
     def testIPython(self):
60 63
         """Test the IPython runner."""
61 64
         source = """
@@ -129,6 +132,7 @@ def testIPython(self):
129 132
         runner = irunner.IPythonRunner(out=self.out)
130 133
         self._test_runner(runner,source,output)
131 134
 
  135
+    @known_failure_py3
132 136
     def testPython(self):
133 137
         """Test the Python runner."""
134 138
         runner = irunner.PythonRunner(out=self.out)
2  IPython/lib/tests/test_irunner_pylab_magic.py
@@ -22,6 +22,7 @@ def setUp(self):
22 22
         self.out = StringIO.StringIO()
23 23
         #self.out = sys.stdout
24 24
 
  25
+    @decorators.known_failure_py3
25 26
     def _test_runner(self,runner,source,output):
26 27
         """Test that a given runner's input/output match."""
27 28
 
@@ -82,6 +83,7 @@ def test_pylab_import_all_enabled(self):
82 83
         runner = irunner.IPythonRunner(out=self.out)
83 84
         self._test_runner(runner,source,output)
84 85
 
  86
+    @decorators.known_failure_py3
85 87
     @decorators.skipif_not_matplotlib
86 88
     def test_pylab_import_all_disabled(self):
87 89
         "Verify that plot is not available when pylab_import_all = False"
8  IPython/nbformat/v2/nbjson.py
@@ -16,9 +16,8 @@
16 16
 # Imports
17 17
 #-----------------------------------------------------------------------------
18 18
 
19  
-from base64 import encodestring
20 19
 from .nbbase import from_dict
21  
-from .rwbase import NotebookReader, NotebookWriter, base64_decode
  20
+from .rwbase import NotebookReader, NotebookWriter, restore_bytes
22 21
 import json
23 22
 
24 23
 #-----------------------------------------------------------------------------
@@ -26,9 +25,10 @@
26 25
 #-----------------------------------------------------------------------------
27 26
 
28 27
 class BytesEncoder(json.JSONEncoder):
  28
+    """A JSON encoder that accepts b64 (and other *ascii*) bytestrings."""
29 29
     def default(self, obj):
30 30
         if isinstance(obj, bytes):
31  
-            return unicode(encodestring(bytes))
  31
+            return obj.decode('ascii')
32 32
         return json.JSONEncoder.default(self, obj)
33 33
 
34 34
 
@@ -40,7 +40,7 @@ def reads(self, s, **kwargs):
40 40
         return nb
41 41
 
42 42
     def to_notebook(self, d, **kwargs):
43  
-        return base64_decode(from_dict(d))
  43
+        return restore_bytes(from_dict(d))
44 44
 
45 45
 
46 46
 class JSONWriter(NotebookWriter):
56  IPython/nbformat/v2/rwbase.py
@@ -19,31 +19,67 @@
19 19
 from base64 import encodestring, decodestring
20 20
 import pprint
21 21
 
  22
+from IPython.utils.py3compat import str_to_bytes
  23
+
22 24
 #-----------------------------------------------------------------------------
23 25
 # Code
24 26
 #-----------------------------------------------------------------------------
25 27
 
  28
+def restore_bytes(nb):
  29
+    """Restore bytes of image data from unicode-only formats.
  30
+    
  31
+    Base64 encoding is handled elsewhere.  Bytes objects in the notebook are
  32
+    always b64-encoded. We DO NOT encode/decode around file formats.
  33
+    """
  34
+    for ws in nb.worksheets:
  35
+        for cell in ws.cells:
  36
+            if cell.cell_type == 'code':
  37
+                for output in cell.outputs:
  38
+                    if 'png' in output:
  39
+                        output.png = str_to_bytes(output.png, 'ascii')
  40
+                    if 'jpeg' in output:
  41
+                        output.jpeg = str_to_bytes(output.jpeg, 'ascii')
  42
+    return nb
  43
+
  44
+
  45
+# b64 encode/decode are never actually used, because all bytes objects in
  46
+# the notebook are already b64-encoded, and we don't need/want to double-encode
  47
+
26 48
 def base64_decode(nb):
27  
-    """Base64 encode all bytes objects in the notebook."""
  49
+    """Restore all bytes objects in the notebook from base64-encoded strings.
  50
+    
  51
+    Note: This is never used
  52
+    """
28 53
     for ws in nb.worksheets:
29 54
         for cell in ws.cells:
30 55
             if cell.cell_type == 'code':
31  
-                if 'png' in cell:
32  
-                    cell.png = bytes(decodestring(cell.png))
33  
-                if 'jpeg' in cell:
34  
-                    cell.jpeg = bytes(decodestring(cell.jpeg))
  56
+                for output in cell.outputs:
  57
+                    if 'png' in output:
  58
+                        if isinstance(output.png, unicode):
  59
+                            output.png = output.png.encode('ascii')
  60
+                        output.png = decodestring(output.png)
  61
+                    if 'jpeg' in output:
  62
+                        if isinstance(output.jpeg, unicode):
  63
+                            output.jpeg = output.jpeg.encode('ascii')
  64
+                        output.jpeg = decodestring(output.jpeg)
35 65
     return nb
36 66
 
37 67
 
38 68
 def base64_encode(nb):
39  
-    """Base64 decode all binary objects in the notebook."""
  69
+    """Base64 encode all bytes objects in the notebook.
  70
+    
  71
+    These will be b64-encoded unicode strings
  72
+    
  73
+    Note: This is never used
  74
+    """
40 75
     for ws in nb.worksheets:
41 76
         for cell in ws.cells:
42 77
             if cell.cell_type == 'code':
43  
-                if 'png' in cell:
44  
-                    cell.png = unicode(encodestring(cell.png))
45  
-                if 'jpeg' in cell:
46  
-                    cell.jpeg = unicode(encodestring(cell.jpeg))
  78
+                for output in cell.outputs:
  79
+                    if 'png' in output:
  80
+                        output.png = encodestring(output.png).decode('ascii')
  81
+                    if 'jpeg' in output:
  82
+                        output.jpeg = encodestring(output.jpeg).decode('ascii')
47 83
     return nb
48 84
 
49 85
 
15  IPython/nbformat/v2/tests/nbexamples.py
... ...
@@ -1,10 +1,15 @@
  1
+import os
  2
+from base64 import encodestring
  3
+
1 4
 from ..nbbase import (
2 5
     NotebookNode,
3 6
     new_code_cell, new_text_cell, new_worksheet, new_notebook, new_output,
4 7
     new_metadata, new_author
5 8
 )
6 9
 
7  
-
  10
+# some random base64-encoded *bytes*
  11
+png = encodestring(os.urandom(5))
  12
+jpeg = encodestring(os.urandom(6))
8 13
 
9 14
 ws = new_worksheet(name='worksheet1')
10 15
 
@@ -42,8 +47,8 @@
42 47
         output_text=u'<array a>',
43 48
         output_html=u'The HTML rep',
44 49
         output_latex=u'$a$',
45  
-        output_png=b'data',
46  
-        output_jpeg=b'data',
  50
+        output_png=png,
  51
+        output_jpeg=jpeg,
47 52
         output_svg=u'<svg>',
48 53
         output_json=u'json data',
49 54
         output_javascript=u'var i=0;',
@@ -53,8 +58,8 @@
53 58
         output_text=u'<array a>',
54 59
         output_html=u'The HTML rep',
55 60
         output_latex=u'$a$',
56  
-        output_png=b'data',
57  
-        output_jpeg=b'data',
  61
+        output_png=png,
  62
+        output_jpeg=jpeg,
58 63
         output_svg=u'<svg>',
59 64
         output_json=u'json data',
60 65
         output_javascript=u'var i=0;'
3  IPython/testing/decorators.py
@@ -323,6 +323,9 @@ def module_not_available(module):
323 323
 
324 324
 skip_known_failure = knownfailureif(True,'This test is known to fail')
325 325
 
  326
+known_failure_py3 = knownfailureif(sys.version_info[0] >= 3, 
  327
+                                    'This test is known to fail on Python 3.')
  328
+
326 329
 # A null 'decorator', useful to make more readable code that needs to pick
327 330
 # between different decorators based on OS or other conditions
328 331
 null_deco = lambda f: f
6  IPython/testing/skipdoctest.py
@@ -4,6 +4,7 @@
4 4
 numpy and sympy if they're present. Since this decorator is used in core parts
5 5
 of IPython, it's in a separate module so that running IPython doesn't trigger
6 6
 those imports."""
  7
+import sys
7 8
 
8 9
 def skip_doctest(f):
9 10
     """Decorator - mark a function or method for skipping its doctest.
@@ -13,3 +14,8 @@ def skip_doctest(f):
13 14
     etc."""
14 15
     f.skip_doctest = True
15 16
     return f
  17
+
  18
+def skip_doctest_py3(f):
  19
+    """Decorator - skip the doctest under Python 3."""
  20
+    f.skip_doctest = (sys.version_info[0] >= 3)
  21
+    return f
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.