-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add py.test-like "-k" test selection to unittest #76252
Comments
I'd like to add test selection based on parts of the test class/method name to unittest. Similar to py.test's "-k" option: https://docs.pytest.org/en/latest/example/markers.html#using-k-expr-to-select-tests-based-on-their-name Here's a proof of concept implementation: jonashaag/cpython@master...unittest-select Is this something others find useful as well? If so, I'd like to work on getting this into Python stdlib proper. This is my first time contributing to the unittest framework; is the general approach taken in my PoC implementation correct in terms of abstractions? How can I improve the implementation? Jonas |
Just to be clear, the current implementation is limited to substring matches. It doesn't support py.test like "and/or" combinators. (Actually, py.test uses 'eval' to support arbitrary patterns.) So say we have test case SomeClass Then
The -k option may be used multiple times, combining the patterns with "or":
It's also possible to use glob-style patterns, like -k "spam_*_eggs". |
I think this would be a useful enhancement. Feel free to open a PR. I haven't looked at your implementation, but you'll also need to add tests for it. |
Thanks Antoine. I will need some guidance as to what are the correct places to make these changes. I'm not quite sure about the abstractions here (runner, loader, suite, case, etc.) My PoC (see GitHub link in first post) uses a TestSuite subclass. (The subclass is only so that it's easier to assess the general implementation approach; I guess it should be put into the main class instead.) Things I'm unsure of:
Thanks |
First, I should clarify that I'm not a unittest maintainer. However, as far as I can tell, the maintainers have not been very active lately. Also, this is a reasonably simple enhancement (at least conceptually), so I think can do without a maintainer's formal approval. To your questions:
At its core, TestSuite is really a glorified collection of tests. The selection logic is inside the TestLoader, and indeed the TestLoader docstring says: """This class is responsible for loading tests according to various criteria and returning them wrapped in a TestSuite""" So I would recommend putting this in TestLoader.
I think the hardcoded approach is ok. Though to benefit readability you may want to use a predicate function instead.
I think this is the wrong approach. A test that isn't selected shouldn't be skipped, it should not appear in the output at all. Another reason for putting this in TestLoader :-)
I'd ask the question differently: do you need to call .id() to do the matching at all? Intuitively, the TestLoader probably knows about the test names already, even before it instantiates TestCases for them. TestCases shouldn't be instantiated for tests that are not selected. |
My first implementation actually was mostly the test loader. Two things made me change my mind and try to make the changes in the suite code:
Are you still saying this should go to the test loader? |
Le 21/11/2017 à 00:23, Jonas H. a écrit :
Take a look at TestLoader.loadTestsFromName(). It does much more than
I'm not sure what the point of displaying the number of "deselected" But in any case, marking them skipped feels wrong to me. Most often,
IMHO, yes. Looking at your posted implementation, at least I don't |
If it helps, think of TestLoader as collecting tests, rather than simply loading Python modules. |
The internal Python test runner "regrtest" already implements the feature as the -m MATCH option and --matchfile FILENAME option. It's implemented in test.support._match_file(). See bpo-31324 for a recent issue on this feature: performance issue (it's going to be fixed). |
Interesting, Victor. I've had a look at the code you mentioned, but I'm afraid it doesn't really make sense to re-use any of the code. Here's a new patch, implemented in the loader as suggested by Antoine, and with tests. I'm happy to write documentation etc. once we're through with code review. |
The enhancement is now pushed to git master. Thank you Jonas for contributing this! |
IMHO it's a big enhancement for the base unittest test runner and deserves to be documend in What's New in Python 3.7! Jonas: do you want to write a PR to patch Doc/whatsnew/3.7.rst? |
Sure! |
This is appearing as a backwards incompatible change for Django because test case class attributes are now evaluated at load time before setUpClass runs (which initializes some things that those class attributes use). It's possible to adapt Django for this change, but it might affect other projects as well. Traceback from Django's tests: $ python -Wall tests/runtests.py
Testing against Django installed in '/home/tim/code/django/django' with up to 3 processes
Traceback (most recent call last):
File "tests/runtests.py", line 477, in <module>
options.exclude_tags,
File "tests/runtests.py", line 282, in django_tests
extra_tests=extra_tests,
File "/home/tim/code/django/django/test/runner.py", line 598, in run_tests
suite = self.build_suite(test_labels, extra_tests)
File "/home/tim/code/django/django/test/runner.py", line 513, in build_suite
tests = self.test_loader.discover(start_dir=label, **kwargs)
File "/home/tim/code/cpython/Lib/unittest/loader.py", line 346, in discover
tests = list(self._find_tests(start_dir, pattern))
File "/home/tim/code/cpython/Lib/unittest/loader.py", line 403, in _find_tests
full_path, pattern, namespace)
File "/home/tim/code/cpython/Lib/unittest/loader.py", line 457, in _find_test_path
return self.loadTestsFromModule(module, pattern=pattern), False
File "/home/tim/code/cpython/Lib/unittest/loader.py", line 124, in loadTestsFromModule
tests.append(self.loadTestsFromTestCase(obj))
File "/home/tim/code/cpython/Lib/unittest/loader.py", line 90, in loadTestsFromTestCase
testCaseNames = self.getTestCaseNames(testCaseClass)
File "/home/tim/code/cpython/Lib/unittest/loader.py", line 234, in getTestCaseNames
testFnNames = list(filter(shouldIncludeMethod, dir(testCaseClass)))
File "/home/tim/code/cpython/Lib/unittest/loader.py", line 227, in shouldIncludeMethod
testFunc = getattr(testCaseClass, attrname)
File "/home/tim/code/django/django/utils/decorators.py", line 172, in __get__
return self.fget(cls)
File "/home/tim/code/django/django/test/testcases.py", line 1269, in live_server_url
return 'http://%s:%s' % (cls.host, cls.server_thread.port)
AttributeError: type object 'AdminSeleniumTestCase' has no attribute 'server_thread' |
Ah, the problem isn't that it's running getattr() on test methods, but that it runs getattr() on all methods. Former code: attrname.startswith(prefix) and \ New code: testFunc = getattr(testCaseClass, attrname) This is trivial to fix. @core devs: Should I revert to original behaviour with the order of the prefix check and the getattr() call, and add a regression test that guarantees this behaviour? |
Yes, that sounds reasonable to me. |
|
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: