Retry Failed Or Skipped Testcases

As we know, Web UI autotest are not as stable as other kinds of tests such as: interface test, unit test.

So we modified TestNg source code to improve the stability by retry failed/skipped testcases, and built a new TestNg jar file, see details.

How to use

Step 1, modify demo.xml as follow:

Here reruntimes="2" means a testcase has 2 chances of retry. A failed/skipped testcase will retry until success or exhaust its chances, and result of the last one will be taken account into TestNg's report. In this way most failed/skipped testcases which actually caused by instability can be filtered out.

Good to know:

The testcases retry in the unit of <test>, such as: TestNg and TestNgClone.

Step 2, use Dagger's testng jar file to run testcases (In Eclipse testcase are run with TestNg plugin's jar file). Here we use Apache Ant and prepare build.xml under Dagger's root path.

<?xml version="1.0"?>
<project default="demo">

	<taskdef resource="testngtasks" classpath="lib/testng-re-html.jar" />

	<path id="cpath">
		<fileset dir="lib">
			<include name="*.jar" />
			<include name="**/*.jar"/>
		</fileset>
	</path>

	<path id="runpath">
		<pathelement location="bin" />
		<path refid="cpath" />
	</path>

	<target name="compile">
		<delete dir="bin" />
		<mkdir dir="bin" />
		<javac destdir="bin" classpathref="cpath" srcdir="src" debug="true" encoding="UTF8">
		</javac>
		<copy file="src/log4j.properties" tofile="bin/log4j.properties" />
	</target>

	<target name="demo" depends="compile">
		<delete dir="test-output" />
		<mkdir dir="test-output" />
		<testng outputDir="test-output" classpathref="runpath" >
			<xmlfileset dir="test-xml" includes="demo.xml" />
		</testng>
	</target>

</project>