Pass subprocess test runners a suitable location for xunit output #1316
Conversation
|
Parallel failures are timeouts on engine connection, which is probably just caused by the Jenkins VM being unusually slow to spin up processes. We can relax the timeouts if failure is frequent. |
|
Beautiful! This is much, much cleaner and simpler, I'm glad there was a way to do it like this. @minrk, what's the easiest way to change those settings just for the test suite? Should we have a profile in the jenkins recipe with different timeouts? |
|
I've just run it again, and they passed, so I wouldn't worry too much about it unless they start failing regularly. |
|
The engine connection timeout is just in the tests, so we can just change it to whatever we want. The default is 5s, which is generally plenty, but we can up it to 10-15s to be extra conservative. |
|
OK, I'm going to merge this then, given both of your inputs above. We can easily adjust it up if we start seeing these failures frequently. |
Pass subprocess test runners a suitable location for xunit output. This enables our Jenkins instance at Shining Panda to display a proper summary instead of just a raw console log.
| @@ -284,6 +284,11 @@ def __init__(self, runner='iptest', params=None): | |||
|
|
|||
| # Assemble call | |||
| self.call_args = self.runner+self.params | |||
|
|
|||
| print self.call_args | |||
There was a problem hiding this comment.
Oops, I missed this print statement during review. Will delete it now, it makes the test suite too verbose.
Pass subprocess test runners a suitable location for xunit output. This enables our Jenkins instance at Shining Panda to display a proper summary instead of just a raw console log.
Producing xunit reports the easy way. We just need to pass the individual test runners a suitable location to put the output file.
This is what the output looks like: https://jenkins.shiningpanda.com/ipython/job/ipython-xunit-test/3/testReport/
I don't know whether the failing tests in parallel are a new failure, a stochastic thing, or somehow caused by this change (I can't see how it would affect parallel).