Configure a RetryAnalizer to retry the non-deterministic test failures

[nacx]

This would allow us to configure the tests that fail in a non deterministic way in Jenkins to be retried automatically. Hopefully this could reduce the noise in our PR builders. This PR already configures the retry policy in one test known to fail randomly, but could be extended to other tests if this approach turns to be useful.

Note that with this change the retried tests will also show the failures in the output, but the build will be marked as succeeded if they succeed after the retry.

Inspired by: http://stackoverflow.com/a/18374673/3540564

/cc @demobox @adriancole

[nacx]

This is to avoid an issue with the duplicate class finder. Scriptbuilder and core have this file in the test resources folder.

[cloudbees-pull-request-builder]

jclouds-pull-requests-java-6 #152 SUCCESS
This pull request looks good

[buildhive]

jclouds » jclouds #1726 SUCCESS
This pull request looks good
(what's this?)

[cloudbees-pull-request-builder]

jclouds-pull-requests #1241 UNSTABLE
Looks like there's a problem with this pull request

[codefromthecrypt]

+1 Thanks. I hope we can eventually not need this, but that damned reality
keeps messing with my plans!

[nacx]

Absolutely. We shouldn't have to merge this, and it is just an experiment to consider, if it turns to be useful. I don't plan to merge it yet, but close/reopen it to trigger several builds and see if the retry thing works.

[buildhive]

jclouds » jclouds #1730 SUCCESS
This pull request looks good
(what's this?)

[cloudbees-pull-request-builder]

jclouds-pull-requests-java-6 #159 SUCCESS
This pull request looks good

[cloudbees-pull-request-builder]

jclouds-pull-requests-java-6 #160 SUCCESS
This pull request looks good

[cloudbees-pull-request-builder]

jclouds-pull-requests #1248 SUCCESS
This pull request looks good

[cloudbees-pull-request-builder]

jclouds-pull-requests #1249 SUCCESS
This pull request looks good

[buildhive]

jclouds » jclouds #1739 SUCCESS
This pull request looks good
(what's this?)

[cloudbees-pull-request-builder]

jclouds-pull-requests-java-6 #163 SUCCESS
This pull request looks good

[buildhive]

jclouds » jclouds #1745 SUCCESS
This pull request looks good
(what's this?)

[cloudbees-pull-request-builder]

jclouds-pull-requests #1252 SUCCESS
This pull request looks good

[cloudbees-pull-request-builder]

jclouds-pull-requests-java-6 #164 SUCCESS
This pull request looks good

[buildhive]

jclouds » jclouds #1746 SUCCESS
This pull request looks good
(what's this?)

[cloudbees-pull-request-builder]

jclouds-pull-requests #1253 UNSTABLE
Looks like there's a problem with this pull request

[nacx]

OK. Finally we have the proof in Jenkins. I've made some experiments with the maven surefire plugin and Jenkins and found no way to configure both to rely on the TestNG results. They record the test result independently, and this the builds are marked unstable even if the tests are retried and the TestNG results don't have those failed tests.

I'm closing the PR and will reopen (eventually) if I manage to find a way to influence the surefire plugin and Jenkins.