Add Unicode support to the doctesting framework #14153

jdemeyer · 2013-02-20T22:03:46Z

Support

Doctest (i.e. sage:) lines containing non-ASCII characters.
Doctest results containing non-ASCII characters.
Print statements in doctests outputting unicode.
Source files with a PEP 0263 encoding declaration.

see also #18370

CC: @nexttime @embray @jdemeyer @hivert

Component: doctest framework

Keywords: unicode

Author: Frédéric Chapoton

Branch/Commit: ed1e3b5

Reviewer: Erik Bray, Volker Braun, David Roe

Issue created by migration from https://trac.sagemath.org/ticket/14153

The text was updated successfully, but these errors were encountered:

nexttime · 2013-03-19T11:17:11Z

comment:2

Attachment: doctest_unicode.patch.gz

kiwifb · 2013-03-19T23:15:02Z

comment:3

I just tried this patch on the suspicion it could help with the test weirdness I reported in 5.9.beta0 on sage-release

sage -t --long devel/sage-main/sage/misc/interpreter.py
**********************************************************************
File "devel/sage-main/sage/misc/interpreter.py", line 150, in 
sage.misc.interpreter.sage_prompt
Failed example:
    shell.run_cell('sage_prompt()')
Expected:
    u'sage'
Got:
    u'sage'
**********************************************************************
File "devel/sage-main/sage/misc/interpreter.py", line 187, in 
sage.misc.interpreter.SageInteractiveShell.system_raw
Failed example:
    shell.system_raw('R --version')
Expected:
    R version ...
Got:
    WARNING: ignoring environment value of R_HOME
    R version 2.15.2 (2012-10-26) -- "Trick or Treat"
    Copyright (C) 2012 The R Foundation for Statistical Computing
    ISBN 3-900051-07-0
    Platform: x86_64-unknown-linux-gnu (64-bit)
    <BLANKLINE>
    R is free software and comes with ABSOLUTELY NO WARRANTY.
    You are welcome to redistribute it under the terms of the
    GNU General Public License versions 2 or 3.
    For more information about these matters see
    http://www.gnu.org/licenses/.
    <BLANKLINE>
**********************************************************************
File "devel/sage-main/sage/misc/interpreter.py", line 566, in 
sage.misc.interpreter.interface_shell_embed
Failed example:
    shell.run_cell('List( [1..10], IsPrime )')
Expected:
    [ false, true, true, false, true, false, true, false, false, false ]
Got:
    [ false, true, true, false, true, false, true, false, false, false ]
**********************************************************************
3 items had failures:
   1 of  15 in sage.misc.interpreter.SageInteractiveShell.system_raw
   1 of   4 in sage.misc.interpreter.interface_shell_embed
   1 of   4 in sage.misc.interpreter.sage_prompt
    [107 tests, 3 failures, 2.4 s]

The R error was fixed by unsetting R_HOME before running the test. Then I applied the patch and here is the result

sage -t --long devel/sage-main/sage/misc/interpreter.py
**********************************************************************
File "devel/sage-main/sage/misc/interpreter.py", line 150, in sage.misc.interpreter.sage_prompt
Failed example:
    shell.run_cell('sage_prompt()')
Expected:
    u'sage'
Got:
    u'sage'
    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    <ipython-input-1-e17c2c2153ac> in <module>()
    ----> 1 sage_prompt()
    
    /home/work/fbissey/sandbox/sage-5.9.beta0/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in __call__(self, result)
        240             self.update_user_ns(result)
        241             self.log_output(format_dict)
    --> 242             self.finish_displayhook()
        243 
        244     def flush(self):
    <BLANKLINE>
    /home/work/fbissey/sandbox/sage-5.9.beta0/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in finish_displayhook(self)
        224         """Finish up all displayhook activities."""
        225         io.stdout.write(self.shell.separate_out2)
    --> 226         io.stdout.flush()
        227 
        228     def __call__(self, result=None):
    <BLANKLINE>
    AttributeError: IOStream instance has no attribute 'flush'
**********************************************************************
File "devel/sage-main/sage/misc/interpreter.py", line 566, in sage.misc.interpreter.interface_shell_embed
Failed example:
    shell.run_cell('List( [1..10], IsPrime )')
Expected:
    [ false, true, true, false, true, false, true, false, false, false ]
Got:
    [ false, true, true, false, true, false, true, false, false, false ]
    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    <ipython-input-1-a772973ec1ce> in <module>()
    ----> 1 sage.misc.all.logstr("""[ false, true, true, false, true, false, true, false, false, false ]""")
    
    /home/work/fbissey/sandbox/sage-5.9.beta0/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in __call__(self, result)
        240             self.update_user_ns(result)
        241             self.log_output(format_dict)
    --> 242             self.finish_displayhook()
        243 
        244     def flush(self):
    <BLANKLINE>
    /home/work/fbissey/sandbox/sage-5.9.beta0/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in finish_displayhook(self)
        224         """Finish up all displayhook activities."""
        225         io.stdout.write(self.shell.separate_out2)
    --> 226         io.stdout.flush()
        227 
        228     def __call__(self, result=None):
    <BLANKLINE>
    AttributeError: IOStream instance has no attribute 'flush'
**********************************************************************
2 items had failures:
   1 of   4 in sage.misc.interpreter.interface_shell_embed
   1 of   4 in sage.misc.interpreter.sage_prompt
    [107 tests, 2 failures, 3.0 s]

fchapoton · 2016-11-10T20:13:58Z

Branch: public/14153

fchapoton · 2016-11-10T20:13:58Z

comment:11

just made a branch from the old patch, not tried to test it

New commits:

`cffdb90`	`trac 14153 unicode in documentation`

fchapoton · 2016-11-10T20:13:58Z

Commit: cffdb90

sagetrac-git · 2017-01-18T10:59:25Z

Changed commit from cffdb90 to 1712c87

sagetrac-git · 2017-01-18T10:59:25Z

Branch pushed to git repo; I updated commit sha1. New commits:

`abd83e5`	`Merge branch 'public/14153' in 7.5.1`
`1712c87`	`trac 14153 fixing doc default call`

fchapoton · 2017-01-18T16:15:02Z

comment:13

I have repaired a detail, but this is not working, in several distinct ways.

This seems to be broken on lines having unicode characters in comments such as:

sage: 0 + 0    # c'est trop bête

Some doctests come from the notebook, I made a pull request.

A failing doctest in src/sage/misc/rest_index_of_methods.py
is related to the presence of "Lov\xc3\xa1sz theta"

fchapoton · 2017-01-18T16:53:52Z

comment:14

seems that something like that is hurting us (in forker):

sage: md=hashlib.md5()
sage: md.update(u'été')
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-9-6a9959b36b47> in <module>()
----> 1 md.update(u'été')

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 0: ordinal not in range(128)

solution: encode ?

sage: md.update(u'été'.encode('utf8'))

sagetrac-git · 2017-07-21T09:09:48Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1c0dd43`	`trac 14153 trying to fix the gap doc doctests`

fchapoton · 2017-07-21T09:10:12Z

comment:62

not sure at all that this is a good solution..

embray · 2017-07-21T09:31:45Z

comment:63

Interesting. It's not clear to me where TermEncoding is used in the process of displaying some help docs. But if you were able to reproduce by setting that then it must be involved somewhere, somehow...

fchapoton · 2017-07-21T09:33:50Z

comment:64

see the section 5.3-2 GAPDoc2Text in the link https://www.gap-system.org/Manuals/pkg/GAPDoc-1.5.1/doc/chap5.html

I think this explain what GAP does when its doc is required as a text.

embray · 2017-07-21T09:50:06Z

comment:65

Great, that should explain it then.

Hmm--maybe rather than forcing TermEncoding to a specific value we should read TermEncoding and use that to determine how to decode text from GAP.

sagetrac-git · 2017-07-21T11:18:59Z

Branch pushed to git repo; I updated commit sha1. New commits:

`0faa943`	`trac 14153 detect gap encoding`

sagetrac-git · 2017-07-21T11:18:59Z

Changed commit from 1c0dd43 to 0faa943

fchapoton · 2017-07-21T11:25:55Z

comment:67

indeed, here is a try..

fchapoton · 2017-07-25T06:37:15Z

comment:69

bot seems to be stable-green, please review

fchapoton · 2017-07-31T20:29:06Z

comment:70

ping ? Jeroen ? Erik ?

vbraun · 2017-08-05T15:20:59Z

comment:71

Can you replace the unconditional except: with something more sensible? Probably just name = str(name) instead of the entire try/except + assertion. Rest lgtm

sagetrac-git · 2017-08-05T15:36:03Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b6077c0`	`Merge branch 'u/chapoton/14153v2' in 8.1.b1`
`ed1e3b5`	`trac 14153 more precise except clause`

sagetrac-git · 2017-08-05T15:36:03Z

Changed commit from 0faa943 to ed1e3b5

fchapoton · 2017-08-05T15:37:59Z

comment:73

Thanks Volker for having a look.

I have made a more precise except. I guess one could also have removed the try-except.

fchapoton · 2017-08-06T19:52:58Z

comment:74

bot is morally green

roed314 · 2017-08-06T20:00:52Z

Changed reviewer from Erik Bray to Erik Bray, Volker Bruan, David Roe

roed314 · 2017-08-06T20:00:52Z

comment:75

Looks good. Sorry that this review took so long!

vbraun · 2017-08-09T19:23:05Z

Changed reviewer from Erik Bray, Volker Bruan, David Roe to Erik Bray, Volker Braun, David Roe

roed314 · 2017-08-11T02:16:36Z

comment:77

Sorry for the typo Volker!

vbraun · 2017-08-11T18:18:15Z

Changed branch from u/chapoton/14153v2 to ed1e3b5

jdemeyer added this to the sage-5.11 milestone Feb 20, 2013

jdemeyer added t: tests labels Feb 20, 2013

jdemeyer assigned sagetrac-mvngu Feb 20, 2013

This comment has been minimized.

Sign in to view

roed314 assigned roed314 and unassigned sagetrac-mvngu Mar 28, 2013

roed314 added c: doctest framework and removed t: tests labels Mar 28, 2013

jdemeyer modified the milestones: sage-5.11, sage-5.12 Aug 13, 2013

sagetrac-vbraun-spam mannequin modified the milestones: sage-6.1, sage-6.2 Jan 30, 2014

sagetrac-vbraun-spam mannequin modified the milestones: sage-6.2, sage-6.3 May 6, 2014

sagetrac-vbraun-spam mannequin modified the milestones: sage-6.3, sage-6.4 Aug 10, 2014

jdemeyer added the c: python3 label Aug 12, 2016

nexttime mannequin modified the milestones: sage-6.4, sage-7.4 Aug 12, 2016

fchapoton added s: needs review and removed s: needs work labels Jul 21, 2017

roed314 added s: positive review and removed s: needs review labels Aug 6, 2017

vbraun removed the s: positive review label Aug 11, 2017

vbraun closed this as completed in 44fba6b Aug 11, 2017

jdemeyer mentioned this issue Sep 10, 2021

Metaticket: improve doctesting framework #11337

Closed

wluebbe mannequin mentioned this issue Aug 15, 2020

Meta-ticket for Python 3.6+ preparation (sage library) - stage 2 #16052

Closed

gagern mannequin mentioned this issue Dec 29, 2022

Unicode support in reference manual and other docs, e.g. developer manual #18370

Closed

fchapoton mentioned this issue Jan 23, 2017

standardize the utf8 coding declaration #22203

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Unicode support to the doctesting framework #14153

Add Unicode support to the doctesting framework #14153

jdemeyer commented Feb 20, 2013

This comment has been minimized.

nexttime mannequin commented Mar 19, 2013

kiwifb commented Mar 19, 2013

fchapoton commented Nov 10, 2016

fchapoton commented Nov 10, 2016

fchapoton commented Nov 10, 2016

sagetrac-git mannequin commented Jan 18, 2017

sagetrac-git mannequin commented Jan 18, 2017

fchapoton commented Jan 18, 2017

fchapoton commented Jan 18, 2017

sagetrac-git mannequin commented Jul 21, 2017

fchapoton commented Jul 21, 2017

embray commented Jul 21, 2017

fchapoton commented Jul 21, 2017

embray commented Jul 21, 2017

sagetrac-git mannequin commented Jul 21, 2017

sagetrac-git mannequin commented Jul 21, 2017

fchapoton commented Jul 21, 2017

fchapoton commented Jul 25, 2017

fchapoton commented Jul 31, 2017

vbraun commented Aug 5, 2017

sagetrac-git mannequin commented Aug 5, 2017

sagetrac-git mannequin commented Aug 5, 2017

fchapoton commented Aug 5, 2017

fchapoton commented Aug 6, 2017

roed314 commented Aug 6, 2017

roed314 commented Aug 6, 2017

vbraun commented Aug 9, 2017

roed314 commented Aug 11, 2017

vbraun commented Aug 11, 2017

Add Unicode support to the doctesting framework #14153

Add Unicode support to the doctesting framework #14153

Comments

jdemeyer commented Feb 20, 2013

This comment has been minimized.

nexttime mannequin commented Mar 19, 2013

kiwifb commented Mar 19, 2013

fchapoton commented Nov 10, 2016

fchapoton commented Nov 10, 2016

fchapoton commented Nov 10, 2016

sagetrac-git mannequin commented Jan 18, 2017

sagetrac-git mannequin commented Jan 18, 2017

fchapoton commented Jan 18, 2017

fchapoton commented Jan 18, 2017

sagetrac-git mannequin commented Jul 21, 2017

fchapoton commented Jul 21, 2017

embray commented Jul 21, 2017

fchapoton commented Jul 21, 2017

embray commented Jul 21, 2017

sagetrac-git mannequin commented Jul 21, 2017

sagetrac-git mannequin commented Jul 21, 2017

fchapoton commented Jul 21, 2017

fchapoton commented Jul 25, 2017

fchapoton commented Jul 31, 2017

vbraun commented Aug 5, 2017

sagetrac-git mannequin commented Aug 5, 2017

sagetrac-git mannequin commented Aug 5, 2017

fchapoton commented Aug 5, 2017

fchapoton commented Aug 6, 2017

roed314 commented Aug 6, 2017

roed314 commented Aug 6, 2017

vbraun commented Aug 9, 2017

roed314 commented Aug 11, 2017

vbraun commented Aug 11, 2017