xunit vs multiprocess

[jpellerin]

What steps will reproduce the problem?

1. Run xunit unit test w/ --processes=2
2. See fails

What is the expected output? What do you see instead?

All of nose's tests should pass under --processes=2

Please use labels and text to provide additional information.

Problem is, xunit creates a file and writes to it when it sees reportable events. For that to work
under mp, all the reporting has to happen in the main process. That will require a major redesign of
how mp worker processes report their results to the main process.

Google Code Info:
Issue #: 267
Author: jpelle...@gmail.com
Created On: 2009-05-16T13:04:45.000Z
Closed On:

[jpellerin]

Any update to this issue? I would love to see a fix soon as this issue is plaguing me.

Google Code Info:
Author: boneill...@gmail.com
Created On: 2010-12-22T22:50:58.000Z

[jpellerin]

I don't have time to work on nose myself right now, but would happily accept a patch (with tests) that fixes this issue.

Google Code Info:
Author: jpelle...@gmail.com
Created On: 2010-12-23T14:19:35.000Z

[jpellerin]

I wish I had the knowledge to fix it myself.

Google Code Info:
Author: boneill...@gmail.com
Created On: 2010-12-27T18:59:50.000Z

[jpellerin]

I was able to do a quick fix using multiprocessing.Queue and multiprocessing.Array to manage passing data across the processes. Because the multiprocess plugin also uses the python multiprocessing module, this seemed like the most natural extension.

In order for this work, the xunit plugin also needs to be loaded into each new process, this is currently not being done in the mulitprocess plugin.

I'm attaching two files: a new multiprocess plugin and xunit plugin called Xunitmp:

The multiprocess plugin also fixes the timeouts, generator issues as covered in Issue 399:

http://code.google.com/p/python-nose/issues/detail?id=399

The Xunitmp plugin can be enabled with --with-xunitmp

The only disadvantage here is that the queues are global variables. I don't think this will be a problem...

rosen diankov,

Google Code Info:
Author: ffxvzero@gmail.com
Created On: 2011-02-23T11:18:28.000Z

[jpellerin]

I'm attaching a new version of xunitmultiprocess.py that accurately measures the timings.

rosen diankov,

Google Code Info:
Author: ffxvzero@gmail.com
Created On: 2011-02-23T12:02:42.000Z

[jpellerin]

Google Code Info:
Author: jpelle...@gmail.com
Created On: 2011-02-23T13:21:05.000Z

[jpellerin]

I'm attaching a new version of xunitmultiprocess

After doing a little more testing, it became clear that multiprocessing.Manager is a better alternative than multiprocessing.Queue since it doesn't block on exiting processes.

I also made it play nicer with the Capture plugin, now it will fill the XML tags instead of embedding it into the error stream.

Google Code Info:
Author: ffxvzero@gmail.com
Created On: 2011-02-25T05:35:22.000Z

[jpellerin]

Please get the latest xunitmultiprocess.py from:

https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/test/noseplugins/xunitmultiprocess.py

In order to use it with the new multiprocess.py, have to do:

multiprocess._instantiate_plugins = [xunitmultiprocess.Xunitmp]

Google Code Info:
Author: rosen.di...@gmail.com
Created On: 2011-03-25T05:51:09.000Z

[jpellerin]

Rosen:

I'm glad to see you're using Manager.Queue rather than the raw Queue.

It's troubling to see coupling between the multiprocess and xunit plugins. They shouldn't need to be aware of each other.

Earlier, I added a fix to the plugintest plugin which may have some bearing here. The cleanest solution was to always use multiprocessing-aware interface whenever the the multiprocessing module was available. This meant that the plugintest system didn't need to be aware of the multiprocess plugin, and vice versa. It would continue to "just work" regardless of whether multiprocessing was used or not.

Would an analogous approach be useful here?

Google Code Info:
Author: workitha...@gmail.com
Created On: 2011-05-03T04:09:15.000Z

[jpellerin]

does the new multiprocess code already address this issue? Some (maybe all) of the patches here have been applied already in Issue 399

Google Code Info:
Author: kumar.mcmillan
Created On: 2011-05-03T14:38:58.000Z

[jpellerin]

I tried it today but the problem is still there and was not fixed by Issue 399.

As far as I understand it after trying a few things each test process creates its own
Xunit instances, then report, addSucess, addFailure and so on gets called on different
instances. As suggested in previous comments I guess it could be solved by sharing
same instance of the errorlist. I don't see the need of this affecting the multiprocess plugin at all, but maybe I am missing something?

Google Code Info:
Author: anders.r...@gmail.com
Created On: 2011-11-22T14:07:00.000Z

[jpellerin]

Ok, I see now why you need to add things in the multiprocess plugin, I thought the multiprocessing module worked differently.

Google Code Info:
Author: anders.r...@gmail.com
Created On: 2011-11-22T14:40:12.000Z

[santiycr]

What's the status of this bug? I do see lots of updates in mp.
I would love to help if that's needed. Is there anything else for this one to happen?

[vogxn]

I'm trying the use of Rosen's plugin for some of my integration tests. The first problem I had was that its options --xunit-file conflicts with xunit plugin's option of the same name. So I changed Rosen's plugin to have --xml-file instead and got it working. Secondly, my tests do not have an attribute capturedOutput which the plugin refers to:

    if test.capturedOutput is not None:
        systemout = '<system-out><![CDATA['+escape_cdata(str(test.capturedOutput))+']]></system-out>'
    xml = """<testcase classname=%(cls)s name=%(name)s time="%(taken)f">

I've commented out the if condition for now and kicked off a run. Will report back any issues I see. Would love to very much see this plugin/ this issue fixed. The multiprocess runs saved 1/3rd of our run time.

Would it be hard to just have xunit plugin send each of its run reports into a seperate file? say moduleName-report.xml? Because this is how unittest-xml-reporting splits the xUnit style reports and they are rendered fine by jenkins CI.

Also - anyone actively working on this issue?

[vogxn]

Ok the xunitmp plugin works for me on my setup. I've been able to run tests in parallel again. Any chance of it being merged with nose mainline? I'm happy to provide any testing data if reqd.

[vogxn]

The nose version I am using the plugin with is 1.2.1

$nosetests -p
Plugin xunit
Plugin deprecated
Plugin skip
Plugin multiprocess
Plugin failuredetail
Plugin capture
Plugin logcapture
Plugin xunitmp
Plugin coverage
Plugin marvin
Plugin attributeselector
Plugin doctest
Plugin profile
Plugin id
Plugin allmodules
Plugin collect-only
Plugin isolation
Plugin pdb

[vogxn]

ok - i'm back. there seem to be a couple of problems with the xunitmp plugin

1. It doesn't show test classes in the XML of the fixtures that were run
   

2. I often see the following awkward error that doesn't tell me much. I followed some threads on nose-devs that had tests with fixtures created using generators. But I have no such tests:

nose.plugins.multiprocess: ERROR: Worker 1 error running test or returning results
Traceback (most recent call last):
File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 699, in runner
test(result)
File "/tests/lib/python2.7/site-packages/nose/suite.py", line 176, in __call
return self.run(_arg, *_kw)
File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 796, in run
test(orig)
File "/tests/lib/python2.7/site-packages/nose/suite.py", line 176, in call
return self.run(_arg, *_kw)
File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 796, in run
test(orig)
File "/tests/lib/python2.7/site-packages/nose/suite.py", line 176, in call
return self.run(_arg, *_kw)
File "/tests/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 777, in run
result.addError(self, self._exc_info())
File "/tests/lib/python2.7/site-packages/nose/proxy.py", line 134, in addError
plugins.addError(self.test, err)
File "/tests/lib/python2.7/site-packages/nose/plugins/manager.py", line 99, in call
return self.call(_arg, *_kw)
File "/tests/lib/python2.7/site-packages/nose/plugins/manager.py", line 167, in simple
result = meth(_arg, *_kw)
File "/tests/lib/python2.7/site-packages/nose/plugins/manager.py", line 334, in addError
return self.plugin.addError(test.test, err, capt)
AttributeError: 'NoSharedFixtureContextSuite' object has no attribute 'test'
Failure: AttributeError ('NoSharedFixtureContextSuite' object has no attribute 'test') ... ERROR

[jszakmeister]

Oooh. That looks like plugins/manager.py is expecting the test attribute to exist but it doesn't. On a plugin I'm working on, I've seen this be the case when there is an error is something like setUpClass(). I'm not sure about this particular trace though.

[donald-ngo]

Hey guys I think I am having the same problem. When I run
nosetests -s -v --xml-report-file=donald.xml --xml-accumulate --with-xunit --processes=3

I always get the following output in my xml file.

It looks like my tests are being run in parallel but the xml file is always like that. Is this the same problem you guys are having?

[winhamwr]

@donald-ngo Yup. That's the problem.

[twiggy]

Is there a way we can vote on this? processes gives us a huge speed up, but having no xunit output blows when the build fails.

[donald-ngo]

I will gladly vote to get this fixed

[vogxn]

+1, would be nice to have this fixed in the upcoming release.

Prasanna.,

From: Donald Ngo [mailto:notifications@github.com]
Sent: Wednesday, April 10, 2013 05:43 AM
To: nose-devs/nose nose@noreply.github.com
Cc: Prasanna Santhanam
Subject: Re: [nose] xunit vs multiprocess (#2)

I will gladly vote to get this fixed

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-16142306.

[vultron81]

Also vote to get this issue fixed

[daubman]

+1, critical for builds with very large numbers of tests

[kylejmcintyre]

I was asked to join github just to vote for this issue being fixed. This has a very negative effect on our CI.

[santiycr]

+1 for getting this fixed. I'll happily buy dinner for whoever takes on this.

[niedbalski]

+1 for fix this issue, would be a great milestone for large tests batteries runners!

[kahunacohen]

Hi, don't mean to nag, but I am just curious if this is actively being worked on by someone. Thanks for any info!

[jszakmeister]

@kahunacohen Not that I know of. For one, that's a particular hard area of nose to work with. Second, I think most of the maintainers (including myself) having a growing interest in Nose 2. I'm not sure what changes need to be made to use Nose 2 (I haven't dived deeply into it just yet), but it does support multiple processes.

[Ignas]

https://pypi.python.org/pypi/nose_xunitmp/0.2 - works on my machine certified
https://github.com/Ignas/nose_xunitmp - for bugs and fixes

[twiggy]

Thanks Ignas!!!!!!!!!!!!!!

[senseysensor]

Thanks!!!

[castulo]

I'm using the xunitmp plugin and still is not working :(

[Ignas]

Well, you should probably complain about that in xunitmp github, with details like nose version, operating system and what is failing, and not here...

[nichochar]

Thanks Ignas, that's really helpful.

[crusaderky]

Any plan about fixing this problem in the main nose distribution?

[jszakmeister]

@crusaderky If you'd like to see this fixed, I suggest putting together a PR to fix it. It's not likely to climb up the list otherwise.

I'd also suggest taking a look at Nose 2. It's really the future of Nose, and it has both JUnit XML support and MP support.

[kahunacohen]

nose2 hasn't gained much traction. I've stuck with nose1 after trying out
2. nose2 isn't quite there yet and nobody's really picked it up. No
disrespect, just the truth. There's a nose1 plugin that can deal with this
issue more or less.

On Wed, Jan 28, 2015 at 6:09 AM, John Szakmeister notifications@github.com
wrote:

@crusaderky https://github.com/crusaderky If you'd like to see this
fixed, I suggest putting together a PR to fix it. It's not likely to climb
up the list otherwise.

I'd also suggest taking a look at Nose 2. It's really the future of Nose,
and it has both JUnit XML support and MP support.

—
Reply to this email directly or view it on GitHub
#2 (comment).

[jszakmeister]

nose2 hasn't gained much traction.

It won't unless more people use it, and more importantly, help oout.

I've stuck with nose1 after trying out
2. nose2 isn't quite there yet and nobody's really picked it up. No
disrespect, just the truth. There's a nose1 plugin that can deal with this
issue more or less.

None taken. OTOH, Nose will go away at some point. It's pretty crufty as it stands, and fixing some issues would require breaking plugins. Others, such as the multiprocessing brokeness, can only be fixed with some significant effort and likely a number of architecture changes--and I apologize, but it's more time than I can commit to.

So at the end of the day, Nose only grows harder to maintain with time--and quite frankly, with all the stuff I've got going on right now, I'm seriously considering dropping out of the open source scene for a while.

Nose 2 is really where the attention needs to be.

[kahunacohen]

I agree that it's up to users to move nose forward, but at the same time if
nose2 were really that needed it would be getting done. This says to me
that either everyone's leaving nose for other things, or nose 1 suffices
enough for most people that they don't see value in nose2, or a combination
of the two. I'd say the latter. A lot of people are moving on to other
things, and those who continue to use nose1 (I do for many cases) think
it's good enough for most situations. The only real issue I've ever had
with nose1, was this mp issue, which I've generally been able to work
around. We're all busy and generally concentrate on what we need to do to
get our jobs done. We contribute to open-source only when we feel the
benefit of contributing outweighs the cost. The fact that nose1 may be full
of cruft only becomes a problem when it really interferes with our work.
Apparently that's not the case for most people (yet).

On Wed, Jan 28, 2015 at 8:35 AM, John Szakmeister notifications@github.com
wrote:

nose2 hasn't gained much traction.

It won't unless more people use it, and more importantly, help oout.

I've stuck with nose1 after trying out
2. nose2 isn't quite there yet and nobody's really picked it up. No
disrespect, just the truth. There's a nose1 plugin that can deal with this
issue more or less.

None taken. OTOH, Nose will go away at some point. It's pretty crufty as
it stands, and fixing some issues would require breaking plugins. Others,
such as the multiprocessing brokeness, can only be fixed with some
significant effort and likely a number of architecture changes--and I
apologize, but it's more time than I can commit to.

So at the end of the day, Nose only grows harder to maintain with
time--and quite frankly, with all the stuff I've got going on right now,
I'm seriously considering dropping out of the open source scene for a while.

Nose 2 is really where the attention needs to be.

—
Reply to this email directly or view it on GitHub
#2 (comment).

[p00j4]

any idea if its fixed or does it work in nose2 now? or --with-xunit & --processes are still not compatible for any of nose version?

However, @jpellerin : thanks for the great help 👍 xunitmp comes as rescue

[therewillbeblood]

i use nose 1.3.7 inside a virtualenv where install junit_xml==1.6 and nose_xunitmp==0.4.0 and then i can successfully run:
python ${venv_home}/bin/nosetests -vv --processes=8 --process-timeout=21600 --exe --with-xunitmp --xunitmp-file "${xml_file}" ${files_to_check}

The process timeout is necessary because the default value is ridiculously low