New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Doctest should handle situations when test files are not readable #59383
Comments
Hi, Thanks for considering! |
In general in Python we let exceptions speak for themselves. It would, however, probably be reasonable to add the try/except and error message in _test, which should only be called when the module is run as a script. |
Thanks for the comment David. I'm attaching second version that does the same in the _test function. |
Great, thanks. There's a loop around testfiles. I think it would be nicer to not call sys.exit. You could at the same time fix what appears to be a bug in the loop code. It looks like 1 is currently returned only if the last file fails. Hmm, and given that there is a bug, adding tests would be great if you are up for it. I don't think there are currently any tests of the doctest command line functionality. That's more complicated than writing the fix, though :) |
Sure, if you give me some time, I'll try to do everything as you suggest. |
So I figured it might be best to first agree on the actual behaviour (what the patch will look like) and then I can write the tests. So here is my 3rd version:
Does this look acceptable? If not, I will happily work further :) (and provide the tests once the behaviour is clear). Thanks! |
Ah, right, I misread the code when I looked at it. There is usually a reason why something is done the way it is...though not always. But, running some example command lines, it looks to me like the current behavior is there because without it, you won't notice if the tests in the middle of your list of files failed. So, I think your second patch is more what we need. However, rather than calling sys.exit, I think it would be better to just print the message and do a return 1. It is also possible we shouldn't change it at all. unittest (which has had some thought put in to the commnd line interface) just ends in a traceback in a similar situation (except that in unittest's case it is an ImportError...) |
Ok, attaching 4th version :) I think it is nice when the testing library can react in situations like this one and not show a traceback. IMHO it is always nicer to display a pretty user-readable message than ending with a traceback. |
Failure modes tend to get less attention that successful behavior. If I wrote a program that used doctest/unittest to test multiple files, I should like it to run an many as possible. If a filename is bad, print name as usual, say 'aborted', and '0/0 tests' run. Then at the end tell me via return value or exception about problems. In any case, having error behavior documented and consistent between modules would be good. I am not sure how much of this would be a bugfix versus enhancement. |
The problem with running all the files as things stand is that the errors get lost in the other output. Changing that would definitely be an enhancement. |
So maybe an optimal solution would be to add a note summarizing this to the end of output? I mean that this: X passed and Y failed. Could be replaced by: X passed and Y failed, Z files were not loaded. Then the user will know that the files were not loaded and he can search the test output for the reasons. |
I think that sounds reasonable. The message should say files in all three cases', since the individual test file summary method will be similar but talking about numbers of tests. |
Fifth version :)
This is a sample output: $ python3 -m doctest -v spam c.py beans
Cannot read 'spam': [Errno 2] No such file or directory: 'spam'
Trying:
2 * 2
Expecting:
4
ok
1 items passed all tests:
1 tests in c
1 tests in 1 items.
1 passed and 0 failed.
Test passed.
Cannot read 'beans': [Errno 2] No such file or directory: 'beans'
Test files read successfully: 1
Unreadable files: 2 Does this look better? |
@david can you pick this up given your previous involvement? |
Added few nitpicks on Rietveld. |
Attaching a new version of patch:
Hopefully this is the final version :) |
Doctest still failed with backtrace if argument is a file name. $ ./python -m doctest aaa.py
Traceback (most recent call last):
File "/home/serhiy/py/cpython/Lib/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/home/serhiy/py/cpython/Lib/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/serhiy/py/cpython/Lib/doctest.py", line 2795, in <module>
sys.exit(_test())
File "/home/serhiy/py/cpython/Lib/doctest.py", line 2773, in _test
m = __import__(filename[:-3])
ImportError: No module named 'aaa' It would be good to have tests. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: