Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
A simple example of my proposed "how to package tests" standard. What do you think? --titus -- To run the tests, you can do: % python -m somepackage.tests.__main__ under Python 2.5 or above, and % python setup.py test if you have setuptools or Distribute installed. This mechanism will also be supported under distutils2, or distutils under 3.3 and above. You can also run % nosetests with nose. How about py.test? Other test runners? --- Rules, in brief: ---------------- - for a given package, 'python -m somepackage.tests.__main__' should run its tests, if any. - no code outside of 'somepackage.tests' should depend on 'somepackage.tests'. This is so that distrib packagers (debian, redhat) can omit the sub-package from the non-devel install. Rules, with a bit more exposition: ---------------------------------- Tests should generally be run at build time, but may not be runnable after installation. Nonetheless we want to specify how to do it in case people want to install them. As long as distrib packagers can remove the <package>/tests directory from their distributed version of your package, they will be happy. Therefore the rest of the package should not depend on anything in <package>/tests. --- Results of testing-in-python discussion, 2/27: - Matt Harrison: explicitly suggest placing test data (data used ONLY for tests) under the tests package directory. - Michael Foord: tests/ should be a package (contain an __init__.py), so that it is discoverable and Python can handle it. - Michal Kwiatkowski: explicitly suggest putting other tests in tests/ top-level directory. - Michal Kwiatkowski: also look at http://time-loop.tumblr.com/post/49578950/where-python-developers-keep-their-test-directories to see where people put their test directories. - Michael Foord: make the mechanism framework agnostic. - Titus Brown: rely on something other than specific module runtime semantics; so python setup.py test or something in the style of python -m somepackage.tests is better than 'python -m unittest somepackage' or whatnot. - All right thinking people agree that supporting suites from test tools other than unittest in setup.py would be nice. - Michael Foord: the 'suite' callable mentioned in unittest documentation isn't actually used or enforced, so it does not clash, collide, or make redundant the 'test_suite' collable convention defined in setuptools. - Michael Foord: for modules loaded with a TestLoader the load_tests function is better. For executing suites directly you can *now* do: python -m unittest package.module.suitename. - Michael Ford in response to Robert Collins: "So the real requirement is that the package / module specified should either be a standard unittest package - or provide a load_tests function that returns a unittest compatible TestSuite-like object, capable of being run with a TextTestRunner. If all test frameworks can adhere to this then I'm fine with dropping the 'run' module requirement." - Michael Foord in response to Titus Brown: "load_tests is already a convention. This particular hook allows you to interoperate with unittest whilst avoiding most of the snakes. If you have *specific* problems with this approach then I'd like to hear about them. :-)" - Titus Brown: I should write up examples of using load_tests, test_suite (in setuptools), test_runner (in setuptools), discover (in unittest2/3rd-party package), and '-m unittest' (which may be redundant with one of the above?) - Titus Brown: I also don't really understand what the arguments to 'setup.py test' are. Longer e-mail from Michael: > Perhaps the convention could be that you provide an executable run module > only if the distutils2 defaults aren't adequate. In this case the > arguments you supply in setup.py tests/testrunner arguments will have to > do the same job as your run module (duplication). > > We could come up with a more complex scheme for distutils2 that meets all > these usecases: > > If no arguments are specified in setup.py then "setup.py test" does the > following: > > For each package in the distribution that supplies a test sub-package > including a run module, execute the equivalent of: > > python -m package.test.run > > For all packages that don't provide a test.run sub-package and module > attempt test discovery. > > If tests/testrunner arguments are supplied to setup.py then those will be > used and the programmer should also provide a test.run if he wants tests to > be able to be run without setup.py being used. > This kind of makes the setup.py arguments redundant, or at least slightly > harmful to use them without providing test.run as well. --- Concerns: how flexible is 'python setup.py test'? Can it be framework agnostic, or do frameworks have to expose a unittest-like framework (e.g. what nose has done)? Another point: I like the idea of being able to specify other test runners, with unittest being the default one. Once we have that level of API compliance then we can do all sorts of nice things; more once I work through some examples. @@CTB revisit test_suite / test_runner in setuptools