Skip to content
This repository

xunit vs multiprocess #2

Open
jpellerin opened this Issue December 14, 2011 · 33 comments
JP
Owner

What steps will reproduce the problem?
1. Run xunit unit test w/ --processes=2
2. See fails

What is the expected output? What do you see instead?

All of nose's tests should pass under --processes=2

Please use labels and text to provide additional information.

Problem is, xunit creates a file and writes to it when it sees reportable events. For that to work
under mp, all the reporting has to happen in the main process. That will require a major redesign of
how mp worker processes report their results to the main process.

Google Code Info:
Issue #: 267
Author: jpelle...@gmail.com
Created On: 2009-05-16T13:04:45.000Z
Closed On:

JP
Owner

Any update to this issue? I would love to see a fix soon as this issue is plaguing me.

Google Code Info:
Author: boneill...@gmail.com
Created On: 2010-12-22T22:50:58.000Z

JP
Owner

I don't have time to work on nose myself right now, but would happily accept a patch (with tests) that fixes this issue.

Google Code Info:
Author: jpelle...@gmail.com
Created On: 2010-12-23T14:19:35.000Z

JP
Owner

I wish I had the knowledge to fix it myself.

Google Code Info:
Author: boneill...@gmail.com
Created On: 2010-12-27T18:59:50.000Z

JP
Owner

I'm attaching a new version of xunitmultiprocess.py that accurately measures the timings.

rosen diankov,

Google Code Info:
Author: ffxvzero@gmail.com
Created On: 2011-02-23T12:02:42.000Z

JP
Owner

I was able to do a quick fix using multiprocessing.Queue and multiprocessing.Array to manage passing data across the processes. Because the multiprocess plugin also uses the python multiprocessing module, this seemed like the most natural extension.

In order for this work, the xunit plugin also needs to be loaded into each new process, this is currently not being done in the mulitprocess plugin.

I'm attaching two files: a new multiprocess plugin and xunit plugin called Xunitmp:

The multiprocess plugin also fixes the timeouts, generator issues as covered in Issue 399:

http://code.google.com/p/python-nose/issues/detail?id=399

The Xunitmp plugin can be enabled with --with-xunitmp

The only disadvantage here is that the queues are global variables. I don't think this will be a problem...

rosen diankov,

Google Code Info:
Author: ffxvzero@gmail.com
Created On: 2011-02-23T11:18:28.000Z

JP
Owner

Google Code Info:
Author: jpelle...@gmail.com
Created On: 2011-02-23T13:21:05.000Z

JP
Owner

I'm attaching a new version of xunitmultiprocess

After doing a little more testing, it became clear that multiprocessing.Manager is a better alternative than multiprocessing.Queue since it doesn't block on exiting processes.

I also made it play nicer with the Capture plugin, now it will fill the XML tags instead of embedding it into the error stream.

Google Code Info:
Author: ffxvzero@gmail.com
Created On: 2011-02-25T05:35:22.000Z

JP
Owner

Please get the latest xunitmultiprocess.py from:

https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/test/noseplugins/xunitmultiprocess.py

In order to use it with the new multiprocess.py, have to do:

multiprocess._instantiate_plugins = [xunitmultiprocess.Xunitmp]

Google Code Info:
Author: rosen.di...@gmail.com
Created On: 2011-03-25T05:51:09.000Z

JP
Owner

Rosen:

I'm glad to see you're using Manager.Queue rather than the raw Queue.

It's troubling to see coupling between the multiprocess and xunit plugins. They shouldn't need to be aware of each other.

Earlier, I added a fix to the plugintest plugin which may have some bearing here. The cleanest solution was to always use multiprocessing-aware interface whenever the the multiprocessing module was available. This meant that the plugintest system didn't need to be aware of the multiprocess plugin, and vice versa. It would continue to "just work" regardless of whether multiprocessing was used or not.

Would an analogous approach be useful here?

Google Code Info:
Author: workitha...@gmail.com
Created On: 2011-05-03T04:09:15.000Z

JP
Owner

I tried it today but the problem is still there and was not fixed by Issue 399.

As far as I understand it after trying a few things each test process creates its own
Xunit instances, then report, addSucess, addFailure and so on gets called on different
instances. As suggested in previous comments I guess it could be solved by sharing
same instance of the errorlist. I don't see the need of this affecting the multiprocess plugin at all, but maybe I am missing something?

Google Code Info:
Author: anders.r...@gmail.com
Created On: 2011-11-22T14:07:00.000Z

JP
Owner

does the new multiprocess code already address this issue? Some (maybe all) of the patches here have been applied already in Issue 399

Google Code Info:
Author: kumar.mcmillan
Created On: 2011-05-03T14:38:58.000Z

JP
Owner

Ok, I see now why you need to add things in the multiprocess plugin, I thought the multiprocessing module worked differently.

Google Code Info:
Author: anders.r...@gmail.com
Created On: 2011-11-22T14:40:12.000Z

Santiago Suarez Ordoñez

What's the status of this bug? I do see lots of updates in mp.
I would love to help if that's needed. Is there anything else for this one to happen?

Prasanna Santhanam

I'm trying the use of Rosen's plugin for some of my integration tests. The first problem I had was that its options --xunit-file conflicts with xunit plugin's option of the same name. So I changed Rosen's plugin to have --xml-file instead and got it working. Secondly, my tests do not have an attribute capturedOutput which the plugin refers to:

    if test.capturedOutput is not None:
        systemout = '<system-out><![CDATA['+escape_cdata(str(test.capturedOutput))+']]></system-out>'
    xml = """<testcase classname=%(cls)s name=%(name)s time="%(taken)f">

I've commented out the if condition for now and kicked off a run. Will report back any issues I see. Would love to very much see this plugin/ this issue fixed. The multiprocess runs saved 1/3rd of our run time.

Would it be hard to just have xunit plugin send each of its run reports into a seperate file? say moduleName-report.xml? Because this is how unittest-xml-reporting splits the xUnit style reports and they are rendered fine by jenkins CI.

Also - anyone actively working on this issue?

Prasanna Santhanam

Ok the xunitmp plugin works for me on my setup. I've been able to run tests in parallel again. Any chance of it being merged with nose mainline? I'm happy to provide any testing data if reqd.

Prasanna Santhanam

The nose version I am using the plugin with is 1.2.1

$nosetests -p
Plugin xunit
Plugin deprecated
Plugin skip
Plugin multiprocess
Plugin failuredetail
Plugin capture
Plugin logcapture
Plugin xunitmp
Plugin coverage
Plugin marvin
Plugin attributeselector
Plugin doctest
Plugin profile
Plugin id
Plugin allmodules
Plugin collect-only
Plugin isolation
Plugin pdb

Prasanna Santhanam

ok - i'm back. there seem to be a couple of problems with the xunitmp plugin

1) It doesn't show test classes in the XML of the fixtures that were run

2) I often see the following awkward error that doesn't tell me much. I followed some threads on nose-devs that had tests with fixtures created using generators. But I have no such tests:

nose.plugins.multiprocess: ERROR: Worker 1 error running test or returning results
Traceback (most recent call last):
File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 699, in runner
test(result)
File "/tests/lib/python2.7/site-packages/nose/suite.py", line 176, in __call

return self.run(*arg, **kw)
File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 796, in run
test(orig)
File "/tests/lib/python2.7/site-packages/nose/suite.py", line 176, in call
return self.run(*arg, **kw)
File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 796, in run
test(orig)
File "/tests/lib/python2.7/site-packages/nose/suite.py", line 176, in call
return self.run(*arg, **kw)
File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 777, in run
result.addError(self, self.exc_info())
File "/tests/lib/python2.7/site-packages/nose/proxy.py", line 134, in addError
plugins.addError(self.test, err)
File "/tests/lib/python2.7/site-packages/nose/plugins/manager.py", line 99, in __call_

return self.call(*arg, **kw)
File "/tests/lib/python2.7/site-packages/nose/plugins/manager.py", line 167, in simple
result = meth(*arg, **kw)
File "/tests/lib/python2.7/site-packages/nose/plugins/manager.py", line 334, in addError
return self.plugin.addError(test.test, err, capt)
AttributeError: 'NoSharedFixtureContextSuite' object has no attribute 'test'
Failure: AttributeError ('NoSharedFixtureContextSuite' object has no attribute 'test') ... ERROR

John Szakmeister
Collaborator

Oooh. That looks like plugins/manager.py is expecting the test attribute to exist but it doesn't. On a plugin I'm working on, I've seen this be the case when there is an error is something like setUpClass(). I'm not sure about this particular trace though.

Donald Ngo

Hey guys I think I am having the same problem. When I run
nosetests -s -v --xml-report-file=donald.xml --xml-accumulate --with-xunit --processes=3

I always get the following output in my xml file.
<?xml version="1.0" encoding="UTF-8"?>

It looks like my tests are being run in parallel but the xml file is always like that. Is this the same problem you guys are having?

Wes Winham

@donald-ngo Yup. That's the problem.

twiggy

Is there a way we can vote on this? processes gives us a huge speed up, but having no xunit output blows when the build fails.

Donald Ngo

I will gladly vote to get this fixed

Prasanna Santhanam
vogxn commented April 09, 2013
vultron81

Also vote to get this issue fixed

Aaron Daubman
daubman commented May 09, 2013

+1, critical for builds with very large numbers of tests

kylejmcintyre

I was asked to join github just to vote for this issue being fixed. This has a very negative effect on our CI.

Santiago Suarez Ordoñez

+1 for getting this fixed. I'll happily buy dinner for whoever takes on this.

Jorge Niedbalski

+1 for fix this issue, would be a great milestone for large tests batteries runners!

kahunacohen

Hi, don't mean to nag, but I am just curious if this is actively being worked on by someone. Thanks for any info!

John Szakmeister
Collaborator

@kahunacohen Not that I know of. For one, that's a particular hard area of nose to work with. Second, I think most of the maintainers (including myself) having a growing interest in Nose 2. I'm not sure what changes need to be made to use Nose 2 (I haven't dived deeply into it just yet), but it does support multiple processes.

twiggy

Thanks Ignas!!!!!!!!!!!!!!

senseysensor

Thanks!!!

Blair Morrison blrm referenced this issue in omaciel/robottelo December 17, 2013
Closed

Jenkins job to start testing the UI #123

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.