test_argparse failure in interactive mode

[terryjreedy]

BPO	11906
Nosy	@terryjreedy, @ezio-melotti, @merwok, @voidspace, @Trundle

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2012-01-16.09:09:36.546>
created_at = <Date 2011-04-22.03:26:10.285>
labels = ['easy', 'type-bug', 'library']
title = 'test_argparse failure in interactive mode'
updated_at = <Date 2012-01-16.09:09:36.545>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2012-01-16.09:09:36.545>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2012-01-16.09:09:36.546>
closer = 'terry.reedy'
components = ['Library (Lib)']
creation = <Date 2011-04-22.03:26:10.285>
creator = 'terry.reedy'
dependencies = []
files = []
hgrepos = []
issue_num = 11906
keywords = ['easy']
message_count = 18.0
messages = ['134259', '134284', '134286', '134291', '137068', '137076', '137085', '137087', '137088', '137094', '137099', '137112', '137141', '150923', '150948', '150988', '150990', '151333']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'bethard', 'ezio.melotti', 'eric.araujo', 'michael.foord', 'Trundle', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11906'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[terryjreedy]

(3.1 not checked because it seems not to have test_argparse)
Python 2.7 or 3.2, WinXP these pass:
C:\Programs\Python27>python -m test.regrtest test_argparse
C:\Programs\Python32>python -m test test_argparse
[1/1] test_argparse
1 test OK.

C:\Programs\Python32>python -m test -v test_argparse
C:\Programs\Python32>python -c "from test import test_argparse as t; t.test_main()"
Ran 1536 tests in 6.188s
OK

but interactively (command window interpreter or IDLE shell, 3.2 or 2.7)
>>> from test import test_argparse as t; t.test_main()

gives two failures related to extra spaces in usage string (I added a couple of newlines for clarity). I have no idea if problem is with argparse, test_argparse, regrtest, unittest, or interactive versus batch mode.

======================================================================
FAIL: test_groups_parents (test.test_argparse.TestParentParsers)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\Programs\Python32\lib\test\test_argparse.py", line 2217, in test_groups_parents
    '''.format(self.main_program)))
  File "C:\Programs\Python32\lib\test\test_argparse.py", line 29, in assertEqual
    super(TestCase, self).assertEqual(obj1, obj2)
AssertionError:
'usage: [-h] [-w W] [-x X] [-y Y | -z Z]\n\noptional arguments:\n  -h, --help  s [truncated]... != 
'usage:  [-h] [-w W] [-x X] [-y Y | -z Z]\n\noptional arguments:\n  -h, --help   [truncated]...
- usage: [-h] [-w W] [-x X] [-y Y | -z Z]
+ usage:  [-h] [-w W] [-x X] [-y Y | -z Z]
?        +
  
  optional arguments:
    -h, --help  show this help message and exit
    -y Y
    -z Z
  
  g:
    gd
  
    -w W
    -x X

======================================================================
FAIL: test_parent_help (test.test_argparse.TestParentParsers)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\Programs\Python32\lib\test\test_argparse.py", line 2188, in test_parent_help
    '''.format(self.main_program)))
  File "C:\Programs\Python32\lib\test\test_argparse.py", line 29, in assertEqual
    super(TestCase, self).assertEqual(obj1, obj2)
AssertionError:
'usage: [-h] [-b B] [--d D] [--w W] [-y Y] a z\n\npositional arguments:\n  a\n   [truncated]... != 
'usage:  [-h] [-b B] [--d D] [--w W] [-y Y] a z\n\npositional arguments:\n  a\n  [truncated]...
- usage: [-h] [-b B] [--d D] [--w W] [-y Y] a z
+ usage:  [-h] [-b B] [--d D] [--w W] [-y Y] a z
?        +
  
  positional arguments:
    a
    z
  
  optional arguments:
    -h, --help  show this help message and exit
    -b B
    --w W
  
  c:
    --d D
  
  x:
    -y Y

---

Ran 1536 tests in 29.438s

FAILED (failures=2)
Traceback (most recent call last):
  File "<pyshell#11>", line 1, in <module>
    from test import test_argparse as t; t.test_main()
  File "C:\Programs\Python32\lib\test\test_argparse.py", line 4423, in test_main
    support.run_unittest(__name__)
  File "C:\Programs\Python32\lib\test\support.py", line 1145, in run_unittest
    _run_suite(suite)
  File "C:\Programs\Python32\lib\test\support.py", line 1128, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: multiple errors occurred

[terryjreedy]

If I put the same line I ran interactively in a file and run it

from test import test_argparse as t; t.test_main()

all tests pass. So it is specifically a problem from the interactive prompt.

[Trundle]

That happens because argparse uses os.basename(sys.argv[0]) (per default) as program name, but sys.argv[0] is usually a string of length 0 at interactive sessions. The tests use "usage: {} ...".format(program_name) (note that there will be two spaces for an empty program_name) for the expected output, while argparse only outputs one space in that case.

[terryjreedy]

Thanks for the diagnosis. I am glad it is something simple.

[merwok]

I don’t know if we should go out of our way to support running tests in interactive mode. Running them from regrtest is the recommended way.

[voidspace]

Unless Terry wants to contribute a fix I suggest closing this.

[terryjreedy]

Ahem. Interactive mode is an approved method of running Python code, along with batch mode. The core interpreter and stdlib modules should run correctly in both modes. So the entire test suite should pass in both modes too. If the tests are written correctly, failure would indicate a bug in the tested component.

That aside, the doc for test/ does not contain 'recommended' and does not discuss running a single test. What I did is the easiest way on Windows.

In this case, following Andreas' remark, the bug is in the test, not the module, in that it miscalculates the expected output in the corner case of a null program name. At lines 2172 and 2205 in the 3.2.0 version of test_argparse.py, changing

'''usage: {} ...
...
'''.format(self.main_program)

to

prog = self.main_program
...
'''usage: {}{}...
...
'''.format(prog, ' ' if prog else '')

fixes the problem.

[merwok]

Interactive mode is an approved method of running Python code, along
with batch mode. The core interpreter and stdlib modules should run
correctly in both modes. So the entire test suite should pass in both
modes too.
You are right.

That aside, the doc for test/ does not contain 'recommended' and does
not discuss running a single test.
Such guidelines belong more in the devguide than the library doc: http://docs.python.org/devguide/runtests

What I did is the easiest way on Windows.
I cannot argue with you on that :)

In this case, following Andreas' remark, the bug is in the test, not
the module, in that it miscalculates the expected output in the
corner case of a null program name.
Since it’s an easy patch in the test file, +1.

[voidspace]

Interactive mode is an approved method of running Python code, along with batch mode.

That is not guaranteed for any particular piece of Python code in the standard library. In particular it is not amenable to test automation, so it is certainly not a requirement that tests pass in this mode. (What is the *use case* for running a command line argument parser in interactive mode?)

I'm certainly not opposed to fixing tests so that they do pass when run like this, but I disagree that it is a "bug".

[terryjreedy]

Unless the doc for a module explicitly diclaims interactive mode (as does multiproccessing), it should run interactively as documented. Batch and interactive are not mutually exclusive; python -i runs a file in batch mode and switches to interactive mode. IDLE *always* runs files this way!

Interactive exploration is a recommended way to learn Python. I agree that it would be tedious to explore the usage of argparse, for instance, by typing everything at the interactive prompt. But one could, for instance, write a file that puts fake content into sys.argv, sets up option and arg specs, and parses. After running the file in IDLE (or with python -i), one might interactively modify sys.argv or the specs and reparse to see what changes.

In any case, using a module interactively and running its test interactively are different things. If a test cannot run interactively, it should be marked as 'skip if interactive' just as with all the other skip conditions. (Skip if not self.program_name might do it.) But this is all moot for this issue.

[voidspace]

it should run interactively as documented.

Where is it documented that all tests will run from the IDLE prompt?

I have *never* heard this claim before. I have nothing against tests supporting this, but those who want it to happen will have to do the work.

[ezio-melotti]

Terry, I think you can apply the patch you proposed in msg137085 and close this issue.
If the recommended structure of test files is not documented, a section in the devguide should be added, but that's another issue. (FWIW I'm not even sure there's a recommended structure, if that's the case we should figure out one, document it, and use it in all the tests.)

[terryjreedy]

I will when I get setup to do that again.

[terryjreedy]

http://docs.python.org/devguide/runtests.html
"If you don’t have easy access to a command line, you can run the test suite from a Python or IDLE shell:
>>> from test import autotest"

However, argparse is the least of the test suite problems on Windows.
(bpo-9116, bpo-11732, other problems)

[merwok]

"If you don’t have easy access to a command line, you can run the test
suite from a Python or IDLE shell:
>>> from test import autotest"

I discovered that after our discussion in this report and added it to the devguide in c18fd0ee23ed.

BTW I agree with Ezio that the fix you proposed should be committed.

[python-dev]

New changeset b950267efd59 by Terry Jan Reedy in branch '3.2':
bpo-11906 Make test_argparse work interactively by removing extra space
http://hg.python.org/cpython/rev/b950267efd59

[python-dev]

New changeset ec32e6ec16fc by Terry Jan Reedy in branch '2.7':
bpo-11906 Make test_argparse work interactively by removing extra space
http://hg.python.org/cpython/rev/ec32e6ec16fc

[ezio-melotti]

This was committed on py3k in 4f8c24830a5c. Terry, can the issue be closed?