Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Bump zope.testrunner / fix test isolation problems #155
@do3cc nailed why a minor version bump of zope.testrunner makes our tests fail: mixing ZopeTestCase and plone.app.testing within the same distribution renders all its tests unstable.
Note that fixing these test isolation problems not only has the benefit of having more robust tests, but at the same time our test infrastructure will be way more easy (and with that making it far easier to reproduce bugs that are only triggered in Jenkins).
Another nice bonus is that unit tests will run first and thus early feedback if you are already expecting failures.
Lastly a slight increase on speed, due to less layer setup and teardown should be observed as well.
The list of packages affected are:
please create separate tickets for each of them and link them back here so we can keep track of it
Ok, so nothing wrong: jenkins does something like:
On the pull request job it already gives an (unrelated) error on the third command (bin/alltest --all) and for whichever reason the build stops and does not run the last command.
My finding so far on what's failing: the errors are while running Products.CMFDiffTool and plone.app.testing tests.
It looks like there are layer setup/teardown mistakes in CMFDiffTool:
On zope.testrunner 4.4.4 the AT layer is the first layer to be run, works perfectly and the rest of the layers run fine as well.
On zope.testrunner > 4.4.4 the AT layer is ran after other layers have been executed and it fails with a
Minimal way to trigger it:
We see these kind of problems all the time. Most of our old tests just pass because they are ran in a specific order. Changing the order when older packages are involved almost always leads to problems. We have tons of test isolation issues hidden in our setup. So far we were just able to fix the most obvious ones.
The error "ValueError: undefined property 'content_meta_type'" is also well known. See plone/Products.CMFPlone#382 for details.
Important progress update.
I am giving up here for now.
Without a real counting, just by looking, it seems that about 10% of packages use both systems. I can't just kick them all out :-( They all need to be converted to plone.app.testing, step by step.
The packages with problems and the script to find them. Only on mac
@gforcada moved the list of packages on the first message, so that github show the nice progress bar
changed the title from
Bump zope.testrunner / fix test isolation problems
Jan 30, 2016
@do3cc could we add a command line switch to plone.app.testing to log somewhere which tests use ZopeTestCase? That would make things much easier to know what has to be fixed/changed, as it is not as simple as to grep through a package looking for ZopeTestCase.
plone.app.testing has a bbb module that uses ZopeTestCase...
You mean the test runner itself, no?
You can find out from the output of the testrunner. The patching happens in the zopelite Layer. So you look for setup ZopeLite and then for the next occurrence of Running...tests above that line
Von meinem iPhone gesendet
@do3cchttps://github.com/do3cc could we add a command line switch to plone.app.testing to log somewhere which tests use ZopeTestCase? That would make things much easier to know what has to be fixed/changed, as it is not as simple as to grep through a package looking for ZopeTestCase.
plone.app.testing has a bbb modulehttps://github.com/plone/plone.app.testing/blob/master/plone/app/testing/bbb.py that uses ZopeTestCase...
Reply to this email directly or view it on GitHubhttps://github.com/plone/buildout.coredev/pull/155#issuecomment-177044675.
________________________________________________________ The contents of this e-mail and any attachments are confidential to the intended recipient. They may not be disclosed to or used by or copied in any way by anyone other than the intended recipient. If this e-mail is received in error, please immediately notify the sender and delete the e-mail and attached documents. Please note that neither the sender nor the sender's company accept any responsibility for viruses and it is your responsibility to scan or otherwise check this e-mail and any attachments.
But still: A registered type should not leak to other tests.
added a commit
this pull request
Feb 15, 2016
I just thought, that specially for the tests that are not directly importing/deriving from ZopeTestCase, but instead doing it through plone.app.testing.bbb module, this branch can be really helpful identifying all tests that need to be updated: plone/plone.app.testing@7259887