doctest: add cmdline parameters

[techtonik]

BPO	11390
Nosy	@warsaw, @vstinner, @ezio-melotti, @bitdancer
Files	README.txt doctest_cli.patch doctest_cli.patch doctest_term_dummy.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-06-25.12:13:38.725>
created_at = <Date 2011-03-03.20:33:46.151>
labels = ['type-feature']
title = 'doctest: add cmdline parameters'
updated_at = <Date 2013-06-25.12:13:38.724>
user = 'https://bugs.python.org/techtonik'

bugs.python.org fields:

activity = <Date 2013-06-25.12:13:38.724>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2013-06-25.12:13:38.725>
closer = 'r.david.murray'
components = []
creation = <Date 2011-03-03.20:33:46.151>
creator = 'techtonik'
dependencies = []
files = ['20989', '30655', '30667', '30696']
hgrepos = []
issue_num = 11390
keywords = ['patch']
message_count = 16.0
messages = ['130002', '191521', '191522', '191528', '191639', '191645', '191716', '191717', '191824', '191825', '191826', '191843', '191844', '191846', '191850', '191851']
nosy_count = 6.0
nosy_names = ['barry', 'vstinner', 'techtonik', 'ezio.melotti', 'r.david.murray', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue11390'
versions = ['Python 3.4']

[techtonik]

doctest module need to parse -h/--help parameters, and accept flags like ELLIPSIS from command line. Otherwise it very hard to debug tests like contained in the attached README.

It is also worth to make it parameter compatible with unittest.main()

Usage: tests.py [options] [test] [...]

Options:
-h, --help Show this message
-v, --verbose Verbose output
-q, --quiet Minimal output
-f, --failfast Stop on first failure
-c, --catch Catch control-C and display results
-b, --buffer Buffer stdout and stderr during test runs

[bitdancer]

Here is a patch that implements the ones I'm interested in (-o for any option, and -f as a shortcut for -o FAIL_FAST). It's a good question whether doctest should support other unittest options, but doing so is more complicated than supporting -o and -f.

Adding a couple people to nosy who have an interest in testing and doctest, hoping for a patch review :)

[bitdancer]

Oh, a documentation update is still needed here, so I guess I'm not really ready for review yet :(

[bitdancer]

Patch updated with documentation.

[bitdancer]

I will probably commit this tomorrow if there are no objections.

[bitdancer]

Patch updated per Barry's review comments.

[python-dev]

New changeset ae802dc4dcd4 by R David Murray in branch 'default':
bpo-11390: convert doctest CLI to argparse and add -o and -f options.
http://hg.python.org/cpython/rev/ae802dc4dcd4

[bitdancer]

Committed. Thanks for the review, Barry.

[vstinner]

The new tests added in changeset ae802dc4dcd4 are failing on some buildbots. I can reproduce the issue on my Linux box (Fedora 18). First failure of "./python -m test test_doctest":

[1/1] test_doctest

---

File "/home/haypo/prog/python/default/Lib/test/test_doctest.py", line 2627, in test.test_doctest.test_CLI
Failed example:
rc1, out1, err1
Expected:
(0, b'', b'')
Got:
(0, b'\x1b[?1034h', b'')

The b'\x1b[?1034h' sequence it written by the readline module when it is imported. The readline module is imported by pdb, which is used by doctest.

The following article proposes set to TERM environment variable to "linux" to workaround the issue:
http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html

It works:

$ echo $TERM  # yeah, Fedora supports 256 colors!
xterm-256color
$ ./python -c 'import readline'|hexdump -C
00000000  1b 5b 3f 31 30 33 34 68                           |.[?1034h|
00000008
$ TERM=linux ./python -c 'import readline'|hexdump -C

This issue was reported to OpenSuse in 2009:
http://lists.opensuse.org/opensuse-bugs/2009-05/msg01633.html

The issue is not specific to Python. Just one another example with Octave (issue closed as "not a bug"):
https://bugzilla.redhat.com/show_bug.cgi?id=304181

The issue was also reported upstream (bug-readline mailing list) a few days ago:
http://lists.gnu.org/archive/html/bug-readline/2013-06/msg00000.html

[vstinner]

Attached doctest_term_dummy.patch: workaround the issue by setting TERM env var to "dummy".

Why is test_CLI() implemented using a doctest? Why not reusing the unittest module to benefit from test.support and test.regrtest? (Hint: I don't like doctests :-))

[vstinner]

test_doctest.test_CLI() is also failing on Windows:

http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%203.x/builds/1078/steps/test/logs/stdio

It looks like an issue with Windows newline (\r\n) versus UNIX newline (\n). test_CLI() decodes the bytes output from UTF-8, but do not normalize newlines.

The universal_newlines=True option can be passed to subprocess, which is not possible currently using script_helper. Is there function somewhere else to normalize newlines?

[bitdancer]

It is implemented as a doctest because everything else in that test module is implemented as a doctest. And clearly this reveals a bug of some sort. Perhaps the dummy trick needs to be done by doctest itself? (I will need to read your post over more carefully and understand the issue better.)

As for the windows failures, my apologies. I knew I would need to check windows after the commit when I wrote the code, but I forgot to actually do it.

[vstinner]

As for the windows failures, my apologies. I knew I would need to check windows after the commit when I wrote the code, but I forgot to actually do it.

No problem, it's so easy to forget Windows :-) And we have buildbots
for this task (detect regressions).

2013/6/25 R. David Murray <report@bugs.python.org>:

R. David Murray added the comment:

It is implemented as a doctest because everything else in that test module is implemented as a doctest. And clearly this reveals a bug of some sort. Perhaps the dummy trick needs to be done by doctest itself? (I will need to read your post over more carefully and understand the issue better.)

As for the windows failures, my apologies. I knew I would need to check windows after the commit when I wrote the code, but I forgot to actually do it.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue11390\>

---

[vstinner]

Perhaps the dummy trick needs to be done by doctest itself?

The issue really comes from the readline module. The workaround is to set an environment variable, which affects the whole process (all threads).

Only setting TERM for test_CLI() is fine, because it's the only test running doctest in a subprocess.

Hum, we may set TERM=dummy in script_helper directly, because this module tries to run Python in a fresh environment. It uses "python -E" for example.

[python-dev]

New changeset 8f22e03f5f07 by R David Murray in branch 'default':
bpo-11390: fix test failures due to readline and windows lineneds.
http://hg.python.org/cpython/rev/8f22e03f5f07

[bitdancer]

Fixed for this test. It would probably be worthwhile to improve script_helpers, I'll open a new issue for that.