Update/rewrite reddeer test suite run with requirements loaded from config file #1586

odockal · 2017-02-13T14:06:32Z

Aim is to be able to run multiple test runs on single test suite based on annotated requirements loaded from single config file. This is especially aiming to use of JRE and Server (only to those?) requirements together defined with one single config xml file. Such loaded requirement configuration would create matrix of combination of possible configurations that would test suite be run with.

rawagner · 2017-01-23T12:02:53Z

not just JRE/Server, those were just examples. This have to work with any other.

odockal · 2017-01-23T13:39:09Z

Expected output:
Once config.xml is loaded, it could has multiple requirement elements that contains JRE and/or Server specific definition. Based on used annotation/requirements that main suite runs with, and based on parameters used on these requirements, test run configuration would take into account all allowed combinations of requirements set up and run test with each such req. configuration.
Examples of annotation usage:

@jre(version="1.6") -> 1 test run
@jre() -> based on how many JRE are defined in config xml file, runs all of them, (or none)
this might be restricted with a param, vesion="ANY" would run all defined JRE otherwise none would be run
@jre(version="1.6", compare="GREATER_THAN") - would run any JRE that fulfills condition such as, search for: JRE is GREATER_THAN (compare param value) 1.6 (version value), default is EQUALS condition.
@server (state=..., type=EAP, version=7.0) -> 1 test run
@server (type=AS, version=7.1, compare=LESS_THAN) -> would run x times where x is the number of defined AS servers under verion 7.1 that occured in config.xml
@server () -> would runamount of the tests with respect to number of defined server runtimes
@jre (conditions) and @server (conditions) -> would run x tests by number of applicable JRE times number of server runtimes

Please, edit this document if necessary.

odockal · 2017-01-23T13:43:30Z

@rawagner @mlabuda @apodhrad @rhopp @psrna I think that this will be quite more complex, than one could expect. Express any ideas, please.

rawagner · 2017-01-23T14:28:19Z

looks good! Another possibility could be to specifically define all versions required for example @jre(version="1,6","1.8") (1.7 is missing).

mlabuda · 2017-01-23T14:57:56Z

Two ideas:

@SomeRequirement() would run all suitable requirements. I would be carefull with ANY and running all of them, because one could expect by providing "ANY" to run any - one and only one. So using ANY for version would trigger only one test run for a first found config.
I would try to do something like @jre(@Version(min="1.7")) to run a test with all JREs higher or equal to JRE 1.7. Then another @jre(@Version(exclude="1.7")) to run all with all specified JRE's expect 1.7. Or @jre(@Version(min="1.6", exclude="1.9")) to run with with all JREs higher than 1.6 but exclude 1.9 from the test run. Default usage would be @jre(@Version("1.6")) which would run a test only with JRE 1.6. Other comparison operators could be "max"...

rhopp · 2017-01-23T15:26:44Z

Guys... keep in mind that we are trying to design something more general. We used server/jre as an example. Another example could be databases and connections. One could have configuration file with X URL's to different databases (and for example even with different credentials) and we should be able to provide "something" in this core implementation which would enable user to create it's own restrictions of which configurations should be running and which not.

rhopp · 2017-01-23T15:28:57Z

In other words: I think that "restriction management" should be done on requiement level (every requirement should be able to decide by itself). But the core implementation (this issue) must provide everything the requiement could need to make that decision.

mlabuda · 2017-01-23T15:56:58Z

If restriction management will be done on requirement level - requirement should decide on itself whether to run or not, we would already get error/failures in execution, because requirement would fail on fulfill/canFulfill. We should address requirements to run as well, e.g. in requirement builder.

But to the point, @SomeReq() should run all defined reqs in xml. @SomeReq() @SomeReq2() should run defined reqs combination (MxN). Restrictions would limit amount of requirements to run tests with, combinations with other requirements should be preserved.

odockal · 2017-02-13T14:14:05Z

rawagner · 2017-02-14T10:15:36Z

We want to support only one config file which means that SuiteConfiguration.getListOfConfigurationsPerFile() should be renamed and should return just one file

rawagner · 2017-02-14T11:06:01Z

check support for nested suites

rawagner · 2017-02-14T11:06:33Z

require requirements .xml file not just folder

mlabuda · 2017-02-14T11:36:24Z

Could you please create more lightweight tests? Using server requirement and JRE requirement seems too much for me. Create 2 simple requirements (or if there are existing for testing, use those) and assert numbers of executed classes/tests for various combination are equal to expected numbers..

odockal · 2017-02-16T06:30:39Z

testPR

odockal · 2017-03-09T11:25:39Z

@rawagner @mlabuda @apodhrad @rhopp @jkopriva @jrichter1 @theJNOVAK
Take a look, amigos. Builds'n'tests seems be almost stable (in case of tests), PR still requires some work but I suggest to do it in the form of another commit/fixup.

rawagner · 2017-03-09T11:32:13Z

...junit/src/org/jboss/reddeer/junit/internal/configuration/RequirementConfigurationObject.java

+ * @author odockal
+ *
+ */
+public class RequirementConfigurationObject {


rename to RequirementConfiguration

rawagner · 2017-03-09T11:32:33Z

....junit/src/org/jboss/reddeer/junit/internal/configuration/RequirementsConfigurationImpl.java

@@ -31,7 +32,7 @@
 * @author Lucia Jelinkova
 * @see TestRunConfiguration
 */
-public class RequirementsConfigurationImpl implements RequirementsConfiguration{


rename to RequirementsConfiguratorImpl and interface to RequirementsConfigurator

I left this classes intact, we agreed on that, right?

rawagner · 2017-03-09T11:53:42Z

...eddeer.junit/src/org/jboss/reddeer/junit/internal/configuration/TestClassRequirementSet.java

+	 * Gets all test classes as an array.
+	 * @return all test classes as an array
+	 */
+	public Class<?>[] getClassesAsArray() {


one liner testClasses.toArray(new Class<?>[testClasses.size()]);

rawagner · 2017-03-09T12:03:33Z

...junit/src/org/jboss/reddeer/junit/internal/configuration/RequirementConfigurationObject.java

+		Requirement<Annotation> requirement = (Requirement<Annotation>) createInstance(this.requirementConfigurationClass.getEnclosingClass());
+
+		if (!(requirement instanceof CustomConfiguration || requirement instanceof PropertyConfiguration)){
+			throw new IllegalArgumentException("The requirement does not implement " + CustomConfiguration.class);


Shouldnt you mention PropertyConfiguration.class too ?

rawagner · 2017-03-09T12:07:12Z

...unit/src/org/jboss/reddeer/junit/internal/configuration/configurator/CustomConfigurator.java

-		if (configs.isEmpty()){
-			throw new RedDeerConfigurationException("No configuration found in XML configuration file for requirement class " + customConfiguration.getClass());
+		// iterate over all configurations and tries to cast the configuration object to expected configuration class
+		// Class cast exception is caught and iteration continues, if none configuration fits, new 


"new" what ? Something is missing in the comment

Comment deleted, Method's javadoc description updated.

rawagner · 2017-03-09T13:48:12Z

...unit/src/org/jboss/reddeer/junit/internal/configuration/configurator/CustomConfigurator.java

+		// Class cast exception is caught and iteration continues, if none configuration fits, new 
+		boolean configurationSet = false;
+		for (Object configuration : this.configurations) {
+			try {


change try-catch to isAssignableFrom

rawagner · 2017-03-09T13:48:33Z

...c/org/jboss/reddeer/junit/internal/configuration/configurator/PropertyBasedConfigurator.java

+
+		boolean configurationSet = false;
+		for (Object configuration : this.configurations) {
+			try {


change try-catch to isAssignableFrom

One catch removed - added if with PropertyBasedConfiguration.class.isAssignableFrom(configuration.getClass()) condition.

rawagner · 2017-03-09T13:51:54Z

...g.jboss.reddeer.junit/src/org/jboss/reddeer/junit/internal/runner/TestsExecutionManager.java

@@ -51,6 +51,7 @@ public int compare(Class<?> clazz0, Class<?> clazz1) {
 	 * 				as test class without a run
 	 */
 	public void addTest(Class<?> testClass) {
+		System.out.println("Adding class without a run: " + testClass.getName());


println deleted

rawagner · 2017-03-09T13:51:58Z

...g.jboss.reddeer.junit/src/org/jboss/reddeer/junit/internal/runner/TestsExecutionManager.java

@@ -62,6 +63,7 @@ public void addTest(Class<?> testClass) {
 	 * 				as test class with a run
 	 */
 	public void addExecutedTest(Class<?> testClass) {
+		System.out.println("Adding class with a run: " + testClass.getName());


no need for output.

rawagner · 2017-03-10T12:03:50Z

plugins/org.jboss.reddeer.requirements/resources/v1/ServerRequirementsBase.xsd

@@ -5,7 +5,7 @@

 	<!-- Import basic RedDeer requirements -->
 	<xs:import namespace="http://www.jboss.org/NS/Req"
-		schemaLocation="http://www.jboss.org/schema/reddeer/v1/RedDeerSchema.xsd" />
+		schemaLocation="RedDeerSchema2.xsd" />


undo the change

rawagner · 2017-03-10T12:05:23Z

...ns/org.jboss.reddeer.requirements/src/org/jboss/reddeer/requirements/jre/JRERequirement.java

@@ -85,6 +86,12 @@ public boolean canFulfill() {
 		}
 	}

+	@Override
+	public boolean isDeclarationAcceptable() {
+		// TODO Delete random generating


remove TODO

I deleted whole method. It is declared with default value in the interface.

rawagner · 2017-03-13T07:37:54Z

...test/src/org/jboss/reddeer/junit/test/integration/runner/order/TestSequenceRedDeerSuite.java

 		int i = 0;
 		while (realSequenceIterator.hasNext() && expectedSequenceIterator.hasNext()){
 			Object real = realSequenceIterator.next();
 			Object expected = expectedSequenceIterator.next();
+			System.out.println(real.toString() + " vs. " + expected.toString());


no need for output...

odockal · 2017-03-13T15:57:40Z

I need to update some unit tests for CustomConfigurator.

rawagner · 2017-03-15T13:44:14Z

testPR

- Test run configuration requirements operates with multiple configurations - Single configuration xml file - Added processing of requirements without configuration - Only one configuration xml file is being work with - Support for nested suites - Support for property based requirements Tests: - Refactored tests: - PropertyBasedConfigurator - CustomConfigurator - TestRunConfigurationImpl - SuiteConfiguration - ParametriziedRequirementTest - RequirementsSequenceTest - InjectRequirementsBeforeRunIfTest - RedDeerSuiteTest - ComplexConfigurationTest - Unit tests for RequirementConfiguration - Unit tests for TestClassRequirementMap - Refactored tests failing due to changes

odockal · 2017-03-16T07:05:40Z

@rawagner commits squashed.

rawagner added this to the 2.0.0 milestone Jan 23, 2017

rawagner added blocker enhancement labels Jan 23, 2017

rawagner assigned odockal Jan 23, 2017

odockal force-pushed the reddeerReqConfig branch 4 times, most recently from 4183081 to 3bcdec9 Compare February 14, 2017 07:58

odockal force-pushed the reddeerReqConfig branch 3 times, most recently from 2a6b1c0 to edc381c Compare February 15, 2017 23:23

odockal force-pushed the reddeerReqConfig branch 5 times, most recently from 9ecdfa1 to 0da34d8 Compare February 23, 2017 16:01

odockal force-pushed the reddeerReqConfig branch 3 times, most recently from a55e021 to 2d2dead Compare March 9, 2017 09:00

rawagner reviewed Mar 9, 2017

View reviewed changes

rawagner reviewed Mar 10, 2017

View reviewed changes

rawagner added a commit to rawagner/reddeer that referenced this pull request Mar 10, 2017

Changes for eclipse#1586

c3893bc

rawagner reviewed Mar 13, 2017

View reviewed changes

odockal force-pushed the reddeerReqConfig branch from 2d2dead to ec68d6c Compare March 13, 2017 12:38

odockal pushed a commit to odockal/reddeer that referenced this pull request Mar 13, 2017

Changes for eclipse#1586

ec68d6c

odockal force-pushed the reddeerReqConfig branch from ec68d6c to 86db3e5 Compare March 13, 2017 12:45

odockal force-pushed the reddeerReqConfig branch 3 times, most recently from f341215 to 73220c5 Compare March 15, 2017 11:53

odockal force-pushed the reddeerReqConfig branch from 73220c5 to e67f9ee Compare March 15, 2017 20:46

rawagner merged commit 67fc981 into eclipse:master Mar 16, 2017

odockal deleted the reddeerReqConfig branch March 16, 2017 09:01

Update/rewrite reddeer test suite run with requirements loaded from config file #1586

Update/rewrite reddeer test suite run with requirements loaded from config file #1586

Conversation

odockal commented Feb 13, 2017

rawagner commented Jan 23, 2017

odockal commented Jan 23, 2017

odockal commented Jan 23, 2017

rawagner commented Jan 23, 2017

mlabuda commented Jan 23, 2017

rhopp commented Jan 23, 2017

rhopp commented Jan 23, 2017

mlabuda commented Jan 23, 2017 • edited

odockal commented Feb 13, 2017 • edited

rawagner commented Feb 14, 2017

rawagner commented Feb 14, 2017

rawagner commented Feb 14, 2017

mlabuda commented Feb 14, 2017

odockal commented Feb 16, 2017

odockal commented Mar 9, 2017

Choose a reason for hiding this comment

rawagner Mar 9, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

odockal commented Mar 13, 2017

rawagner commented Mar 15, 2017

odockal commented Mar 16, 2017

mlabuda commented Jan 23, 2017 •

edited

odockal commented Feb 13, 2017 •

edited

rawagner Mar 9, 2017 •

edited