Unittest output drives developers to avoid docstrings #90284
Comments
|
In python/importlib_metadata#302, I learned that the way unittest reports failures in tests is incentivizing the replacement of docstrings with comments in order not to make resolution of the relevant failing test more difficult to locate. I presume I don't need to explain why docstrings are nice and preferable over comments. Better would be for unittest to provide an option to ignore the docstrings or to emit the test path regardless of whether a docstring was present and to employ that option in CPython, allowing for docstrings in tests. |
|
Could you give a couple examples for those of us not familiar with the problem? |
|
Absolutely. I was thinking to do just that. |
|
I created this diff: diff --git a/Lib/test/test_importlib/test_metadata_api.py b/Lib/test/test_importlib/test_metadata_api.py
index e16773a7e8..92aacd5ad5 100644
--- a/Lib/test/test_importlib/test_metadata_api.py
+++ b/Lib/test/test_importlib/test_metadata_api.py
@@ -90,8 +90,11 @@ def test_entry_points_distribution(self):
self.assertEqual(ep.dist.version, "1.0.0")
def test_entry_points_unique_packages(self):
- # Entry points should only be exposed for the first package
- # on sys.path with a given name.
+ """
+ Entry points should only be exposed for the first package
+ on sys.path with a given name.
+ """
+ raise ValueError("Failing on purpose")
alt_site_dir = self.fixtures.enter_context(fixtures.tempdir())
self.fixtures.enter_context(self.add_sys_path(alt_site_dir))
alt_pkg = {And then ran the tests, but the output is easy to totally scrutable: So there must be some other test invocation that doesn't provide the clarity of which test is failing. |
|
My guess is the tests were run with something like this: In that case, the location of the test is replaced with the first line of the docstring. If there's no docstring, however, the location of the failure is easier to detect: |
|
Looks like the code, I believe here's where the reporting happens: cpython/Lib/unittest/runner.py Lines 48 to 51 in 9b52920 I see a "description" property there that may already provide the feature. |
|
It looks like that 'descriptions' attribute is passed through from TextTestRunner, which sets it to True by default ( cpython/Lib/unittest/runner.py Line 163 in 9b52920 |
|
After some investigation (and tracing calls through a dozen or more layers), I found that Python's own regression tests can disable this behavior thus: Combined with the above diff (with a docstring), the output is now as desired: It seems it would be wise to enable this setting by default, given that without it, every test module is forced to replace docstrings with comments. |
Actually, can you? Docstrings to document regular modules, classes or functions are a fine tool, especially with tools that extract them (pydoc, sphinx). I don’t see the same need for test functions. |
|
I also use comments in lieu of docstrings for text_xyx methods because I find the addition of the first line of the docstring in error reports to be useless, distracting, and sometimes, depending on the content, confusing. If the error is in the test method, the full comment is available when looking back at the method code. If the error is in the tested code, the comment/docstring is inapplicable. When I edit test files, I run them directly from an IDLE editor via the 'if __name__' clause. When I run all IDLE tests from Command Prompt, I usually run 'python -m test.test_idle'. (Similar but not necessarily identical to 'python -m -ugui test_idle'.) So fiddling with regrtest will not help in either case. Based on the above, the following seems to work. |
I searched around and didn't find any good treatise or thorough overview of reasons _why_ docstrings should be preferred, so I performed an internal deconstruction of my motivations and published this article on my blog. |
|
I've merged the fix for regrtest and I'll explore Terry's concerns and see what I can devise for those concerns as well. |
|
In an attempt to replicate Terry's usage, I added the following patch: When I did, I then ran the tests using the As you can see, the location of the failing test in the log is masked, and instead the description is present. I've pushed https://github.com/python/cpython/tree/bpo-46126/disable-descriptions-unittest-m, which when applied disables the descriptions. This approach works as a surgical intervention, wrapping Terry, would you review the concept? Does it meet your needs? If so, I can apply it more broadly and prepare a PR. |
Could you elaborate? I can see |
|
Inada, you're right. Good catch. I think I missed that behavior from before because I used I'm now suspicious that my report in https://bugs.python.org/issue46126#msg408882 was also flawed as it uses I now believe that there are no concerns relating to Terry's concern, and likely not Brett's concern despite my best efforts to replicate. I'll bring back the original example and see if it in fact has the reported defect. |
|
And indeed, after removing the I'm effectively unable to replicate the behavior reported by Brett in the other issue (unless At this point, I propose one of two options:
And then simply declare that docstrings are acceptable for CPython tests. Given that descriptions are on by default, I'm slightly inclined toward (1). Thoughts? |
|
I think the situation and the discussion should be summarized on python-dev! |
|
I've confirmed that prior to the patch in PR 30194, the location of the failure was indeed reported. It was just not reported on the same line: The description is given _in addition_ to the location of the test. That means the concern reported in #302 was actually invalid. I also was under the false impression that the description was masking the test location, but instead, it's just enhancing it (while also injecting a newline). Given this renewed understanding, I believe it's appropriate to back out the PR.
Great suggestion. Will do. |
|
Perhaps we could make an enhancement then? Having the extra information on a separate line is, at least for me, very jarring -- as in, I hadn't figured out that that was the way it was done until Inadasan pointed it out. Perhaps a command-line switch to enable having it all on one line? That would also help when grepping. (I would propose it to be the default behavior, but I can see that that would result in very long lines.) |
|
My goal with this issue is to simply unblock the use of docstrings in Python tests. I believe with the work above and the proposed published post, I've accomplished that goal. I've already spent more time than I'd hoped on this issue and would like to resolve it at its original scope. I do see how this issue reveals some deficiencies with the current UI/UX, and I'd welcome a separate feature request to address those deficiencies. I'll even help draft the bug report on request. |
|
That makes sense. bpo-47133 created. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: