Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

Combined the functionality of the Combination and Permutation test

suites into the Combination test suite.

The Combination test suite can now use a constructor to pass
parameters to the test or use fields or methods to pass the parameters
to the test class.  The Attributes annotation has been enhanced to
include an attributes field that is an array of string.  This array
is assigned with the names of the attributes that have associated
public field or method access.  If this empty, which is the default
setting, then the Combination test suite will assume that there is a
constructor that has the correct number of parameters and parameter
types.

Prior to this the Combination test suite expected a Map as the
values of the attributes but this has changed and it now expects
a list that contains object arrays for each attribute.  The
map key were used to define the attribute names but now this is
accomplished using the Attributes annotation attributes field.

The CombinationTestTest has been updated to include the tests that
were part of the PermutationTestTest test since the Combination
test suite includes the same feature.

The Permutation test suite has been removed along with the
PermutationTestTest test.
  • Loading branch information...
commit 9efcc65f4186644e3d2532341099aa2c50546ddd 1 parent 13ac6be
ccorsi authored September 04, 2012
445  src/main/java/org/junit/experimental/runners/Combination.java
@@ -30,28 +30,46 @@
30 30
 import org.junit.runners.model.TestClass;
31 31
 
32 32
 /**
33  
- * The custom runner <code>Combination</code> implements attribute based tests.
34  
- * When using the combination test class, the instances are created for different
35  
- * permutations of the given values. The values are defined as a map.  Where the
36  
- * key is the attribute field name and the values is an array of Object instances.
37  
- * This runner will then use the field names to set the values. It will initially
38  
- * determine if the field exists and is public.  If it exists, it will use that
39  
- * field to set the value to.  If not, it will then determine if there is a set 
40  
- * method a la java bean.  If it exists, it will use that method to set the value
41  
- * else it will generate an exception.</p>    
  33
+ * The custom runner <code>Combination</code> implements attribute or
  34
+ * parameter based tests.  When using the combination test class, the
  35
+ * instances are created for different permutations of the given
  36
+ * values. The values are defined as a list.  Where each object array
  37
+ * is associated to a given attribute or test class constructor.  The
  38
+ * developer can use the <code>Attributes</code> annotation
  39
+ * <code>attributes</code> field to define the different attribute
  40
+ * name for each entry in the list. If the attributes field does not
  41
+ * contain any attribute names then this runner assumes that the test
  42
+ * class contains a constructor with the same number of parameters as
  43
+ * the size of the list and the same parameter types that are defined
  44
+ * for each object array. </p>
  45
+ *
  46
+ * If the attributes field of the Attributes annotation contain
  47
+ * attribute names then this runner will use those names to set the
  48
+ * values. It will initially determine if the field exists and is
  49
+ * public.  If it exists, it will use that field to set the value to.
  50
+ * If not, it will then determine if there is a set method a la java
  51
+ * bean.  If it exists, it will use that method to set the value else
  52
+ * it will generate an exception.</p>
42 53
  * 
43  
- * For example, you want to create tests that uses a combinations of inputs.
  54
+ * If the attributes field of the Attributes annotation does not
  55
+ * contain any attribute names then this runner will use the test
  56
+ * class constructor to pass the values.  If no single valid
  57
+ * constructor was found or the constructor parameter type are
  58
+ * incompatible then an exception will be raised. </p>
  59
+ *
  60
+ * For example, you want to create tests that uses a combinations of inputs
  61
+ * that uses the attributes field.
44 62
  * 
45 63
  * <pre>
46 64
  * 
47 65
  * &#064;RunWith(Combination.class)
48 66
  * public class CombinationTest {
49 67
  * 
50  
- *  &#064;Attributes(tests = "test")
51  
- *  public static Map&lt;String,Object[]&gt; inputs() {
52  
- *     Map&lt;String,Object[]&gt; inputs = new HashMap&lt;String,Object[]&gt;();
53  
- *     map.put("input", new Object[] { 1, 2, 3, 4, 5 });
54  
- *     map.put("name",  new Object[] { "A", "B", "C" });
  68
+ *  &#064;Attributes(tests = "test", attributes = { "input", "name" })
  69
+ *  public static List&lt;Object[]&gt; inputs() {
  70
+ *     List&lt;Object[]&gt; inputs = new LinkedList&lt;Object[]&gt;();
  71
+ *     inputs.add(new Object[] { 1, 2, 3, 4, 5 });
  72
+ *     inputs.add(new Object[] { "A", "B", "C" });
55 73
  *     return inputs;
56 74
  *  }
57 75
  *  
@@ -73,18 +91,71 @@
73 91
  * }
74 92
  * </pre>
75 93
  * 
76  
- * Each instance of the <code>CombinationTest</code> test will be passed one of all possible
77  
- * permutations of the two attribute input returned from the call to the inputs static method.
78  
- * </p>
  94
+ * Each instance of the <code>CombinationTest</code> test will be
  95
+ * passed one of all possible permutations of the two attribute input
  96
+ * and name returned from the call of the <code>inputs</code> static
  97
+ * method.  </p>
79 98
  * 
80  
- * The defined tests attribute of the Attributes is a regular expression that is
81  
- * applied to the list of tests associated with the test class.  If the regular expression is
82  
- * satisfied then the test is executed using the set of generated attribute list.</p>
  99
+ * Note that the defined <code>tests</code> attribute of the
  100
+ * <code>Attributes</code> is a regular expression that is applied to
  101
+ * the list of tests associated with the test class.  If the regular
  102
+ * expression is satisfied then the test is executed using the set of
  103
+ * generated attribute list.</p>
83 104
  * 
84  
- * Note also that you can define one or more static annotated Attributes methods within a 
85  
- * given test class and be able to associated each methods to one or more test for the given
86  
- * test class depending on the regular expression defined by the tests field. </p>
  105
+ * Note also that you can define one or more static method annotated
  106
+ * by <code>Attributes</code> within a given test class and be able to
  107
+ * associated each methods to one or more test for the given test
  108
+ * class depending on the regular expression defined by the
  109
+ * <code>tests</code> field. </p>
  110
+ * 
  111
+ * Here is an example that uses the test class constructor to pass the
  112
+ * values from a list of object arrays. </p>
  113
+ * 
  114
+ * <pre>
87 115
  * 
  116
+ * &#064;RunWith(Combination.class)
  117
+ * public class CombinationTest {
  118
+ * 
  119
+ *  &#064;Attributes(tests = "test")
  120
+ *  public static List&lt;Object[]&gt; inputs() {
  121
+ *     List&lt;Object[]&gt; inputs = new LinkedList&lt;Object[]&gt;();
  122
+ *     inputs.add(new Object[] { 1, 2, 3, 4, 5 });
  123
+ *     inputs.add(new Object[] { "A", "B", "C" });
  124
+ *     return inputs;
  125
+ *  }
  126
+ *  
  127
+ *  private int input;
  128
+ *  private String name;
  129
+ *  
  130
+ *  public CombinationTest(int input, String name) {
  131
+ *     this.input = input;
  132
+ *     this.name  = name;
  133
+ *  }
  134
+ *  
  135
+ *  &#064;Test
  136
+ *  public void test() {
  137
+ *     // use input and name to run a test...
  138
+ *    ....
  139
+ *  }
  140
+ * }
  141
+ * </pre>
  142
+ * 
  143
+ * In this example we've decided not to use the
  144
+ * <code>attributes</code> field of the <code>Attributes</code>
  145
+ * annotation and instead decided to just pass the parameters using a
  146
+ * constructor instead. </p>
  147
+ *
  148
+ * You might be wondering when to use the constructor pattern or the
  149
+ * attribute pattern?  The pattern used depends on how many and want
  150
+ * types of parameters are used for each test of a given test class.
  151
+ * If each test uses the same parameter inputs then using the
  152
+ * constructor based approach should suffice.  If each test uses
  153
+ * different combination of inputs then using the attributes based
  154
+ * approach is better.  This approach provides you with the ability to
  155
+ * create a single test instance and perform these different tests
  156
+ * instead of having to create potentially multiple copies of the same
  157
+ * test class or sub-class a base class. </p>
  158
+ *
88 159
  * Note that this test suite was inspired by the 
89 160
  * <a href="http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/CombinationTestSupport.java?view=markup">
90 161
  * CombinationTestSupport.java</a> class that is part of the ActiveMQ distribution. </p>
@@ -184,6 +255,19 @@ public String getKey() {
184 255
 		 * @see MessageFormat
185 256
 		 */
186 257
 		String name() default "{list}";
  258
+
  259
+        /**
  260
+         * This is used to map the parameters to a given test class field or method.
  261
+         * The field has to be public and have the same name defined in this array.
  262
+         * The method should be prefixed by set and the first letter should be a 
  263
+         * capital letter.
  264
+         * The Parameterized test suite will look for a public field first and then
  265
+         * look for a public method if the field is not available.
  266
+         *
  267
+         * @return array of attributes names that will be used to set the attribute
  268
+         *         values with, defaults to empty array.
  269
+         */
  270
+        String[] attributes() default {};
187 271
 	}
188 272
 	
189 273
 	private static class TestClassRunnerForCombination extends BlockJUnit4ClassRunner {
@@ -276,6 +360,92 @@ protected String getName() {
276 360
 		
277 361
 	}
278 362
 
  363
+	private class TestClassRunnerForParameters extends BlockJUnit4ClassRunner {
  364
+
  365
+		private String name;
  366
+		private Object[] parameters;
  367
+		private Attributes annotation;
  368
+
  369
+		public TestClassRunnerForParameters(Class<?> clazz, Object[] parameters, String name, Attributes annotation)
  370
+				throws InitializationError {
  371
+			super(clazz);
  372
+			this.name       = name;
  373
+			this.parameters = parameters;
  374
+			this.annotation = annotation;
  375
+		}
  376
+
  377
+		/* (non-Javadoc)
  378
+		 * @see org.junit.runners.BlockJUnit4ClassRunner#validateConstructor(java.util.List)
  379
+		 */
  380
+		@Override
  381
+		protected void validateConstructor(List<Throwable> errors) {
  382
+			validateOnlyOneConstructor(errors);
  383
+		}
  384
+
  385
+		/* (non-Javadoc)
  386
+		 * @see org.junit.runners.BlockJUnit4ClassRunner#createTest()
  387
+		 */
  388
+		@Override
  389
+		protected Object createTest() throws Exception {
  390
+			return getTestClass().getOnlyConstructor().newInstance(parameters);
  391
+		}
  392
+
  393
+		/* (non-Javadoc)
  394
+		 * @see org.junit.runners.BlockJUnit4ClassRunner#testName(org.junit.runners.model.FrameworkMethod)
  395
+		 */
  396
+		@Override
  397
+		protected String testName(FrameworkMethod method) {
  398
+			return method.getName() + getName();
  399
+		}
  400
+
  401
+		/* (non-Javadoc)
  402
+		 * @see org.junit.runners.ParentRunner#classBlock(org.junit.runner.notification.RunNotifier)
  403
+		 */
  404
+		@Override
  405
+		protected Statement classBlock(RunNotifier notifier) {
  406
+			return childrenInvoker(notifier);
  407
+		}
  408
+
  409
+		/* (non-Javadoc)
  410
+		 * @see org.junit.runners.ParentRunner#getName()
  411
+		 */
  412
+		@Override
  413
+		protected String getName() {
  414
+			return name;
  415
+		}
  416
+
  417
+		/* (non-Javadoc)
  418
+		 * @see org.junit.runners.ParentRunner#getRunnerAnnotations()
  419
+		 */
  420
+		@Override
  421
+		protected Annotation[] getRunnerAnnotations() {
  422
+			return new Annotation[0];
  423
+		}
  424
+
  425
+		/* (non-Javadoc)
  426
+		 * @see org.junit.runners.BlockJUnit4ClassRunner#computeTestMethods()
  427
+		 */
  428
+		@Override
  429
+		protected List<FrameworkMethod> computeTestMethods() {
  430
+			List<FrameworkMethod> methods = super.computeTestMethods();
  431
+			List<FrameworkMethod> remove = new LinkedList<FrameworkMethod>();
  432
+			if (this.annotation != null) {
  433
+				String regex = this.annotation.tests();
  434
+				for (FrameworkMethod method : methods) {
  435
+					String methodName = method.getMethod().getName();
  436
+					if (! methodName.matches(regex)) {
  437
+						remove.add(method);
  438
+					}
  439
+				}
  440
+			}
  441
+			// Removes all not compliant methods with the parameters.
  442
+			methods.removeAll(remove);
  443
+			// Return the resulting test methods, can be empty causing an failure...
  444
+			return methods;
  445
+		}
  446
+		
  447
+	}
  448
+
279 449
 	private static final List<Runner> NO_RUNNERS = Collections.emptyList();
280 450
 	
281 451
 	private List<Runner> runners = new LinkedList<Runner>();
@@ -285,19 +455,27 @@ public Combination(Class<?> clz) throws Throwable {
285 455
 		List<FrameworkMethod> attributesMethods = getAttributesMethods(getTestClass());
286 456
 		Class<?> javaClass = getTestClass().getJavaClass();
287 457
 		for(FrameworkMethod frameworkMethod : attributesMethods) {
288  
-			Map<String,Object[]> attributesMap = getAllAttributes(frameworkMethod);
  458
+            Attributes annotation = frameworkMethod.getAnnotation(Attributes.class);
  459
+            String attributeNames[] = annotation.attributes();
  460
+            String namePattern= annotation.name();
  461
+			List<Object[]> attributesList = getAllAttributes(frameworkMethod);
289 462
 			try {
290 463
 				Map<String, CommandFactory> factories= validateAttributes(
291  
-						javaClass, attributesMap.keySet());
292  
-				for (Map.Entry<Command[], int[]> commands : new AttributesListIterable(
293  
-						factories, attributesMap)) {
294  
-					Command cmds[]= commands.getKey();
295  
-					Attributes attributes= frameworkMethod
296  
-							.getAnnotation(Attributes.class);
297  
-					String namePattern= attributes.name();
298  
-					runners.add(new TestClassRunnerForCombination(javaClass,
299  
-							cmds, nameFor(namePattern, cmds), attributes));
300  
-				}
  464
+                        javaClass, attributeNames);
  465
+                if (factories.isEmpty()) {
  466
+                    for (Map.Entry<Object[], int[]> parameters : new ParametersListIterable(attributesList) ) {
  467
+                        Object objects[] = parameters.getKey();
  468
+                        runners.add(new TestClassRunnerForParameters(getTestClass().getJavaClass(),
  469
+                                objects, nameFor(namePattern, objects, parameters.getValue()), annotation));
  470
+                    }
  471
+                } else {
  472
+                    for (Map.Entry<Command[], int[]> commands : new AttributesListIterable(
  473
+                            factories, attributesList, attributeNames)) {
  474
+                        Command cmds[]= commands.getKey();
  475
+                        runners.add(new TestClassRunnerForCombination(javaClass,
  476
+                            cmds, nameFor(namePattern, cmds), annotation));
  477
+                    }
  478
+                }
301 479
 			} catch (ClassCastException e) {
302 480
 				throw attributesMethodReturnedWrongType(frameworkMethod);
303 481
 			}
@@ -310,11 +488,11 @@ public Combination(Class<?> clz) throws Throwable {
310 488
 	 * @throws Throwable
311 489
 	 */
312 490
 	@SuppressWarnings("unchecked")
313  
-	private Map<String, Object[]> getAllAttributes(
  491
+	private List<Object[]> getAllAttributes(
314 492
 			FrameworkMethod frameworkMethod) throws Throwable {
315 493
 		Object attributes= frameworkMethod.invokeExplosively(null);
316  
-		if (attributes instanceof Map) {
317  
-			return (Map<String, Object[]>) attributes;
  494
+		if (attributes instanceof List) {
  495
+			return (List<Object[]>) attributes;
318 496
 		} else {
319 497
 			throw attributesMethodReturnedWrongType(frameworkMethod);
320 498
 		}
@@ -324,7 +502,7 @@ private Exception attributesMethodReturnedWrongType(FrameworkMethod frameworkMet
324 502
 		String className= getTestClass().getName();
325 503
 		String methodName= frameworkMethod.getName();
326 504
 		String message= MessageFormat.format(
327  
-				"{0}.{1}() must return a Map of String/arrays.",
  505
+				"{0}.{1}() must return a List of Object arrays.",
328 506
 				className, methodName);
329 507
 		return new Exception(message);
330 508
 	}
@@ -337,6 +515,18 @@ private String nameFor(String namePattern, Command[] commands) {
337 515
 		return name;
338 516
 	}
339 517
 	
  518
+	private String nameFor(String namePattern, Object[] objects, int[] indexes) {
  519
+		List<Object> list = new LinkedList<Object>();
  520
+		for(int index : indexes) {
  521
+			list.add(index);
  522
+		}
  523
+		String finalPattern= namePattern.replaceAll("\\{list\\}",
  524
+				list.toString());
  525
+		String name= MessageFormat.format(finalPattern, Arrays.asList(objects).toArray());
  526
+
  527
+		return name;
  528
+	}
  529
+	
340 530
 	static interface CommandFactory {
341 531
 		Command create(Object value);
342 532
 	}
@@ -435,10 +625,16 @@ public String toString() {
435 625
 	
436 626
 	/**
437 627
 	 * @param javaClass
438  
-	 * @param keySet
  628
+	 * @param attributeNames
439 629
 	 * @return returns a attribute name/command factory map
440 630
 	 */
441  
-	private Map<String, CommandFactory> validateAttributes(Class<?> javaClass, Set<String> attributeNames) {
  631
+	private Map<String, CommandFactory> validateAttributes(Class<?> javaClass, String[] attributeNames) {
  632
+        // If no attribute names were passed then we are going to use
  633
+        // a constructor to pass the values to the test.
  634
+        if (attributeNames == null || attributeNames.length == 0) {
  635
+            return new HashMap<String, CommandFactory>();
  636
+        }
  637
+
442 638
 		// Get all reference of public attributes...
443 639
 		Field fields[] = javaClass.getFields();
444 640
 		Map<String, Field> fieldMap = new HashMap<String, Field>();
@@ -476,19 +672,21 @@ public String toString() {
476 672
 	static class AttributesListIterable implements Iterable<Map.Entry<Command[], int[]>> {
477 673
 
478 674
 		private Map<String, CommandFactory> factories;
479  
-		private Map<String, Object[]> attributesMap;
  675
+		private List<Object[]>              attributesList;
  676
+        private String[]                    attributeNames;
480 677
 
481 678
 		AttributesListIterable(Map<String, CommandFactory> factories,
482  
-				Map<String, Object[]> attributesMap) {
483  
-			this.factories     = factories;
484  
-			this.attributesMap = attributesMap;
  679
+                List<Object[]> attributesList, String[] attributeNames) {
  680
+			this.factories      = factories;
  681
+			this.attributesList = attributesList;
  682
+            this.attributeNames = attributeNames;
485 683
 		}
486 684
 		
487 685
 		/* (non-Javadoc)
488 686
 		 * @see java.lang.Iterable#iterator()
489 687
 		 */
490 688
 		public Iterator<Entry<Command[], int[]>> iterator() {
491  
-			return new AttributesListIterator(this.factories, this.attributesMap);
  689
+			return new AttributesListIterator(this.factories, this.attributesList, this.attributeNames);
492 690
 		}
493 691
 		
494 692
 		
@@ -508,32 +706,28 @@ public String toString() {
508 706
 
509 707
 			/**
510 708
 			 * @param factories
511  
-			 * @param attributesMap
  709
+			 * @param attributesList
  710
+             * @param attributeNames;
512 711
 			 */
513 712
 			public AttributesListIterator(Map<String, CommandFactory> factories,
514  
-					Map<String, Object[]> attributesMap) {
  713
+                    List<Object[]> attributesList, String[] attributeNames) {
515 714
 				this.factories = factories;
516  
-				sizes = new int[attributesMap.size()];
  715
+				sizes = new int[attributesList.size()];
517 716
 				startIndexes = new int[sizes.length];
518 717
 				Arrays.fill(startIndexes, 0);
519  
-				entries = new EntryImpl[attributesMap.size()];
520  
-				int index = 0;
521  
-				for(Map.Entry<String, Object[]> entry : attributesMap.entrySet()) {
522  
-					entries[index++] = new EntryImpl(entry.getKey(), entry.getValue());
523  
-				}
524  
-				index = 0;
525  
-				for(Map.Entry<String, Object[]> entry : entries) {
526  
-					sizes[index++] = entry.getValue().length;
  718
+				entries = new EntryImpl[attributesList.size()];
  719
+                for(int index = 0 ; index < attributeNames.length ; index++) {
  720
+                    Object[] values = attributesList.get(index);
  721
+					entries[index] = new EntryImpl(attributeNames[index], values);
  722
+					sizes[index]   = values.length;
527 723
 				}
528 724
 				objects = new Object[sizes.length];
529 725
 				indexes = new int[sizes.length];
530 726
 				// Setup a current set of defaults that start with the zero indexed version....
531  
-				int cnt = 0;
532  
-				for(Map.Entry<String, Object[]> entry : entries) {
533  
-					Object[] values = entry.getValue();
  727
+				for(int cnt = 0 ; cnt < sizes.length ; cnt++) {
  728
+                    Object[] values = entries[cnt].getValue();
534 729
 					objects[cnt] = (values.length > 0) ? values[0] : null;
535 730
 					indexes[cnt] = 0;
536  
-					cnt++;
537 731
 				}
538 732
 			}
539 733
 
@@ -585,6 +779,135 @@ public void remove() {
585 779
 		}
586 780
 	}
587 781
 
  782
+	static class ParametersListIterable implements Iterable<Map.Entry<Object[], int[]>> {
  783
+
  784
+		private Iterable<Object[]> parametersList;
  785
+
  786
+		ParametersListIterable(Iterable<Object[]> parametersList) {
  787
+			this.parametersList = parametersList;
  788
+		}
  789
+		
  790
+		/* (non-Javadoc)
  791
+		 * @see java.lang.Iterable#iterator()
  792
+		 */
  793
+		public Iterator<Entry<Object[], int[]>> iterator() {
  794
+			return new ParametersListIterator(this.parametersList);
  795
+		}
  796
+		
  797
+	}
  798
+	
  799
+	static class ParametersListIterator implements Iterator<Map.Entry<Object[], int[]>> {
  800
+
  801
+		private Object[][] parametersArrs;
  802
+		private int        sizes[];
  803
+		private int        startIndexes[];
  804
+		private Object[]   objects;
  805
+		private int[]      indexes;
  806
+
  807
+		ParametersListIterator(Iterable<Object[]> parametersList) {
  808
+			List<Object[]> list = new LinkedList<Object[]>();
  809
+			for(Object[] objects : parametersList) {
  810
+				list.add(objects);
  811
+			}
  812
+			parametersArrs = list.toArray(new Object[0][]);
  813
+			sizes = new int[parametersArrs.length];
  814
+			startIndexes = new int[sizes.length];
  815
+			Arrays.fill(startIndexes, 0);
  816
+			int index = 0;
  817
+			for(Object[] parameters : parametersList) {
  818
+				sizes[index++] = parameters.length;
  819
+			}
  820
+			objects = new Object[sizes.length];
  821
+			indexes = new int[sizes.length];
  822
+			// Setup a current set of defaults that start with the zero indexed version....
  823
+			for( int curRow = 0 ; curRow < sizes.length ; curRow++ ) {
  824
+				// Defense check against someone passed an empty object array
  825
+				objects[curRow] = (parametersArrs[curRow].length > 0) ? parametersArrs[curRow][0] : null;
  826
+				indexes[curRow] = 0;
  827
+			}
  828
+		}
  829
+		
  830
+		/* (non-Javadoc)
  831
+		 * @see java.util.Iterator#hasNext()
  832
+		 */
  833
+		public boolean hasNext() {
  834
+			return indexes[0] < sizes[0];
  835
+		}
  836
+
  837
+		/* (non-Javadoc)
  838
+		 * @see java.util.Iterator#next()
  839
+		 */
  840
+		public Entry<Object[], int[]> next() {
  841
+			if (hasNext() == false) {
  842
+				throw new NoSuchElementException("No elements remaining");
  843
+			}
  844
+			Object[] curObjects = new Object[sizes.length];
  845
+			int[] curIndexes = new int[sizes.length];
  846
+			System.arraycopy(objects, 0, curObjects, 0, sizes.length);
  847
+			System.arraycopy(indexes, 0, curIndexes, 0, sizes.length);
  848
+			incrementIndexes();
  849
+			return createEntry(curObjects, curIndexes);
  850
+		}
  851
+
  852
+		/**
  853
+		 * 
  854
+		 */
  855
+		private void incrementIndexes() {
  856
+			// Find the next starting entry....
  857
+			for (int curRow = sizes.length - 1; curRow > -1; curRow--) {
  858
+				indexes[curRow]++;
  859
+				if (indexes[curRow] < sizes[curRow]) {
  860
+					objects[curRow] = parametersArrs[curRow][indexes[curRow]];
  861
+					break; // exit while loop we are done...
  862
+				} else if (curRow != 0) {
  863
+					indexes[curRow] = 0;
  864
+					objects[curRow] = parametersArrs[curRow][0];
  865
+				}
  866
+			}
  867
+		}
  868
+
  869
+		/* (non-Javadoc)
  870
+		 * @see java.util.Iterator#remove()
  871
+		 */
  872
+		public void remove() {
  873
+			throw new UnsupportedOperationException("remove method is not supported");
  874
+		}
  875
+		
  876
+	}
  877
+	
  878
+	/**
  879
+	 * @param objects
  880
+	 * @param indexes
  881
+	 * @return
  882
+	 */
  883
+	private static Entry<Object[], int[]> createEntry(Object[] objects, int[] indexes) {
  884
+		return new Map.Entry<Object[], int[]>() {
  885
+			
  886
+			private Object[] key;
  887
+			private int[] value;
  888
+
  889
+			public Object[] getKey() {
  890
+				return key;
  891
+			}
  892
+
  893
+			public Entry<Object[], int[]> init(Object[] curObjects,
  894
+					int[] curIndexes) {
  895
+				this.key   = curObjects;
  896
+				this.value = curIndexes;
  897
+				return this;
  898
+			}
  899
+
  900
+			public int[] getValue() {
  901
+				return value;
  902
+			}
  903
+
  904
+			public int[] setValue(int[] arg0) {
  905
+				return value;
  906
+			}
  907
+		}.init(objects, indexes);
  908
+	}
  909
+
  910
+
588 911
 	/**
589 912
 	 * @param factories 
590 913
 	 * @param curObjects
405  src/main/java/org/junit/experimental/runners/Permutation.java
... ...
@@ -1,405 +0,0 @@
1  
-package org.junit.experimental.runners;
2  
-
3  
-import java.lang.annotation.Annotation;
4  
-import java.lang.annotation.ElementType;
5  
-import java.lang.annotation.Retention;
6  
-import java.lang.annotation.RetentionPolicy;
7  
-import java.lang.annotation.Target;
8  
-import java.lang.reflect.Modifier;
9  
-import java.text.MessageFormat;
10  
-import java.util.Arrays;
11  
-import java.util.Collections;
12  
-import java.util.Iterator;
13  
-import java.util.LinkedList;
14  
-import java.util.List;
15  
-import java.util.Map;
16  
-import java.util.Map.Entry;
17  
-import java.util.NoSuchElementException;
18  
-
19  
-import org.junit.runner.Runner;
20  
-import org.junit.runner.notification.RunNotifier;
21  
-import org.junit.runners.BlockJUnit4ClassRunner;
22  
-import org.junit.runners.Suite;
23  
-import org.junit.runners.model.FrameworkMethod;
24  
-import org.junit.runners.model.InitializationError;
25  
-import org.junit.runners.model.Statement;
26  
-import org.junit.runners.model.TestClass;
27  
-
28  
-/**
29  
- * The custom runner <code>Permutation</code> implements parameter based tests.
30  
- * When using the permutation test class, the instances are created for different
31  
- * permutations of the given values. The values are defined as two dimensional array
32  
- * where each ith index is an array of values for the ith parameter of the tests
33  
- * constructor. </p>    
34  
- * 
35  
- * For example, you want to create tests that uses a combinations of inputs.
36  
- * 
37  
- * <pre>
38  
- * 
39  
- * &#064;RunWith(Permutation.class)
40  
- * public class CombinationTest {
41  
- * 
42  
- *  &#064;Parameters(tests = "test")
43  
- *  public static List&lt;Object[]&gt; inputs() {
44  
- *     return Arrays.asList(new Object[][] {
45  
- *         { 1, 2, 3, 4, 5 },
46  
- *         { "A", "B", "C" },
47  
- *       });
48  
- *  }
49  
- *  
50  
- *  private int input;
51  
- *  
52  
- *  private String name;
53  
- *  
54  
- *  public CombinationTest(int input, String name) {
55  
- *     this.input = input;
56  
- *     this.name  = name;
57  
- *  }
58  
- *  
59  
- *  &#064;Test
60  
- *  public void test() {
61  
- *     // use input and name to run a test...
62  
- *    ....
63  
- *  }
64  
- * }
65  
- * </pre>
66  
- * 
67  
- * Each instance of the <code>Permutation</code> test will be passed one of all possible
68  
- * permutations of the two argument input returned from the call to the inputs static method.
69  
- * </p>
70  
- * 
71  
- * The defined tests attribute of the Parameters is a regular expression that is
72  
- * applied to the list of tests associated with the test.  If the regular expression is
73  
- * satisfied then the test is executed using the set of generated parameter list.
74  
- * 
75  
- * @author Claudio Corsi
76  
- *
77  
- */
78  
-public class Permutation extends Suite {
79  
-	
80  
-	/**
81  
-	 * This annotation is used for a static method which will return an array of arrays
82  
-	 * with input used to create each instance of the test class.  These parameters are
83  
-	 * used to create every permutation possible and are then passed to the test
84  
-	 * constructor. 
85  
-	 * 
86  
-	 * @author Claudio Corsi
87  
-	 *
88  
-	 */
89  
-	@Retention(RetentionPolicy.RUNTIME)
90  
-	@Target(ElementType.METHOD)
91  
-	public static @interface Parameters {
92  
-		/**
93  
-		 * This is used to define the regular expression used to determine which tests will
94  
-		 * use the give permutations of the given parameters.
95  
-		 * 
96  
-		 * @return regular expression used to contain tests
97  
-		 */
98  
-		String tests() default ".*";
99  
-
100  
-		/**
101  
-		 * <p>
102  
-		 * Optional pattern to derive the test's name from the parameters. Use
103  
-		 * numbers in braces to refer to the parameters or the additional data
104  
-		 * as follows:
105  
-		 * </p>
106  
-		 * 
107  
-		 * <pre>
108  
-		 * {list} - the parameter list
109  
-		 * {0} - the first parameter value
110  
-		 * {1} - the second parameter value
111  
-		 * etc...
112  
-		 * </pre>
113  
-		 * 
114  
-		 * @return {@link MessageFormat} pattern string, except the index
115  
-		 *         placeholder.
116  
-		 * @see MessageFormat
117  
-		 */
118  
-		String name() default "{list}";
119  
-	}
120  
-	
121  
-	private class TestClassRunnerForPermutations extends BlockJUnit4ClassRunner {
122  
-
123  
-		private String name;
124  
-		private Object[] parameters;
125  
-		private Parameters annotation;
126  
-
127  
-		public TestClassRunnerForPermutations(Class<?> clazz, Object[] parameters, String name, Parameters annotation)
128  
-				throws InitializationError {
129  
-			super(clazz);
130  
-			this.name       = name;
131  
-			this.parameters = parameters;
132  
-			this.annotation = annotation;
133  
-		}
134  
-
135  
-		/* (non-Javadoc)
136  
-		 * @see org.junit.runners.BlockJUnit4ClassRunner#validateConstructor(java.util.List)
137  
-		 */
138  
-		@Override
139  
-		protected void validateConstructor(List<Throwable> errors) {
140  
-			validateOnlyOneConstructor(errors);
141  
-		}
142  
-
143  
-		/* (non-Javadoc)
144  
-		 * @see org.junit.runners.BlockJUnit4ClassRunner#createTest()
145  
-		 */
146  
-		@Override
147  
-		protected Object createTest() throws Exception {
148  
-			return getTestClass().getOnlyConstructor().newInstance(parameters);
149  
-		}
150  
-
151  
-		/* (non-Javadoc)
152  
-		 * @see org.junit.runners.BlockJUnit4ClassRunner#testName(org.junit.runners.model.FrameworkMethod)
153  
-		 */
154  
-		@Override
155  
-		protected String testName(FrameworkMethod method) {
156  
-			return method.getName() + getName();
157  
-		}
158  
-
159  
-		/* (non-Javadoc)
160  
-		 * @see org.junit.runners.ParentRunner#classBlock(org.junit.runner.notification.RunNotifier)
161  
-		 */
162  
-		@Override
163  
-		protected Statement classBlock(RunNotifier notifier) {
164  
-			return childrenInvoker(notifier);
165  
-		}
166  
-
167  
-		/* (non-Javadoc)
168  
-		 * @see org.junit.runners.ParentRunner#getName()
169  
-		 */
170  
-		@Override
171  
-		protected String getName() {
172  
-			return name;
173  
-		}
174  
-
175  
-		/* (non-Javadoc)
176  
-		 * @see org.junit.runners.ParentRunner#getRunnerAnnotations()
177  
-		 */
178  
-		@Override
179  
-		protected Annotation[] getRunnerAnnotations() {
180  
-			return new Annotation[0];
181  
-		}
182  
-
183  
-		/* (non-Javadoc)
184  
-		 * @see org.junit.runners.BlockJUnit4ClassRunner#computeTestMethods()
185  
-		 */
186  
-		@Override
187  
-		protected List<FrameworkMethod> computeTestMethods() {
188  
-			List<FrameworkMethod> methods = super.computeTestMethods();
189  
-			List<FrameworkMethod> remove = new LinkedList<FrameworkMethod>();
190  
-			if (this.annotation != null) {
191  
-				String regex = this.annotation.tests();
192  
-				for (FrameworkMethod method : methods) {
193  
-					String methodName = method.getMethod().getName();
194  
-					if (! methodName.matches(regex)) {
195  
-						remove.add(method);
196  
-					}
197  
-				}
198  
-			}
199  
-			// Removes all not compliant methods with the parameters.
200  
-			methods.removeAll(remove);
201  
-			// Return the resulting test methods, can be empty causing an failure...
202  
-			return methods;
203  
-		}
204  
-		
205  
-	}
206  
-
207  
-	private static final List<Runner> NO_RUNNERS = Collections.emptyList();
208  
-	
209  
-	private List<Runner> runners = new LinkedList<Runner>();
210  
-
211  
-	public Permutation(Class<?> clz) throws Throwable {
212  
-		super(clz, NO_RUNNERS);
213  
-		List<FrameworkMethod> parametersMethods = getParametersMethods(getTestClass());
214  
-		for(FrameworkMethod frameworkMethod : parametersMethods) {
215  
-			Parameters annotation = frameworkMethod.getAnnotation(Parameters.class);
216  
-			String namePattern = annotation.name();
217  
-			Iterable<Object[]> parametersList = getAllParameters(frameworkMethod);
218  
-			for (Map.Entry<Object[], int[]> parameters : new ParametersListIterable(parametersList) ) {
219  
-				Object objects[] = parameters.getKey();
220  
-				runners.add(new TestClassRunnerForPermutations(getTestClass().getJavaClass(),
221  
-						objects, nameFor(namePattern, objects, parameters.getValue()), annotation));
222  
-			}
223  
-		}
224  
-	}
225  
-
226  
-	/**
227  
-	 * @param frameworkMethod
228  
-	 * @return
229  
-	 * @throws Throwable
230  
-	 */
231  
-	@SuppressWarnings("unchecked")
232  
-	private List<Object[]> getAllParameters(FrameworkMethod frameworkMethod)
233  
-			throws Throwable {
234  
-		return (List<Object[]>) frameworkMethod.invokeExplosively(
235  
-				null);
236  
-	}
237  
-
238  
-	private String nameFor(String namePattern, Object[] objects, int[] indexes) {
239  
-		List<Object> list = new LinkedList<Object>();
240  
-		for(int index : indexes) {
241  
-			list.add(index);
242  
-		}
243  
-		String finalPattern= namePattern.replaceAll("\\{list\\}",
244  
-				list.toString());
245  
-		String name= MessageFormat.format(finalPattern, Arrays.asList(objects).toArray());
246  
-
247  
-		return name;
248  
-	}
249  
-	
250  
-	static class ParametersListIterable implements Iterable<Map.Entry<Object[], int[]>> {
251  
-
252  
-		private Iterable<Object[]> parametersList;
253  
-
254  
-		ParametersListIterable(Iterable<Object[]> parametersList) {
255  
-			this.parametersList = parametersList;
256  
-		}
257  
-		
258  
-		/* (non-Javadoc)
259  
-		 * @see java.lang.Iterable#iterator()
260  
-		 */
261  
-		public Iterator<Entry<Object[], int[]>> iterator() {
262  
-			return new ParametersListIterator(this.parametersList);
263  
-		}
264  
-		
265  
-	}
266  
-	
267  
-	static class ParametersListIterator implements Iterator<Map.Entry<Object[], int[]>> {
268  
-
269  
-		private Object[][] parametersArrs;
270  
-		private int        sizes[];
271  
-		private int        startIndexes[];
272  
-		private Object[]   objects;
273  
-		private int[]      indexes;
274  
-
275  
-		ParametersListIterator(Iterable<Object[]> parametersList) {
276  
-			List<Object[]> list = new LinkedList<Object[]>();
277  
-			for(Object[] objects : parametersList) {
278  
-				list.add(objects);
279  
-			}
280  
-			parametersArrs = list.toArray(new Object[0][]);
281  
-			sizes = new int[parametersArrs.length];
282  
-			startIndexes = new int[sizes.length];
283  
-			Arrays.fill(startIndexes, 0);
284  
-			int index = 0;
285  
-			for(Object[] parameters : parametersList) {
286  
-				sizes[index++] = parameters.length;
287  
-			}
288  
-			objects = new Object[sizes.length];
289  
-			indexes = new int[sizes.length];
290  
-			// Setup a current set of defaults that start with the zero indexed version....
291  
-			for( int curRow = 0 ; curRow < sizes.length ; curRow++ ) {
292  
-				// Defense check against someone passed an empty object array
293  
-				objects[curRow] = (parametersArrs[curRow].length > 0) ? parametersArrs[curRow][0] : null;
294  
-				indexes[curRow] = 0;
295  
-			}
296  
-		}
297  
-		
298  
-		/* (non-Javadoc)
299  
-		 * @see java.util.Iterator#hasNext()
300  
-		 */
301  
-		public boolean hasNext() {
302  
-			return indexes[0] < sizes[0];
303  
-		}
304  
-
305  
-		/* (non-Javadoc)
306  
-		 * @see java.util.Iterator#next()
307  
-		 */
308  
-		public Entry<Object[], int[]> next() {
309  
-			if (hasNext() == false) {
310  
-				throw new NoSuchElementException("No elements remaining");
311  
-			}
312  
-			Object[] curObjects = new Object[sizes.length];
313  
-			int[] curIndexes = new int[sizes.length];
314  
-			System.arraycopy(objects, 0, curObjects, 0, sizes.length);
315  
-			System.arraycopy(indexes, 0, curIndexes, 0, sizes.length);
316  
-			incrementIndexes();
317  
-			return createEntry(curObjects, curIndexes);
318  
-		}
319  
-
320  
-		/**
321  
-		 * 
322  
-		 */
323  
-		private void incrementIndexes() {
324  
-			// Find the next starting entry....
325  
-			for (int curRow = sizes.length - 1; curRow > -1; curRow--) {
326  
-				indexes[curRow]++;
327  
-				if (indexes[curRow] < sizes[curRow]) {
328  
-					objects[curRow] = parametersArrs[curRow][indexes[curRow]];
329  
-					break; // exit while loop we are done...
330  
-				} else if (curRow != 0) {
331  
-					indexes[curRow] = 0;
332  
-					objects[curRow] = parametersArrs[curRow][0];
333  
-				}
334  
-			}
335  
-		}
336  
-
337  
-		/* (non-Javadoc)
338  
-		 * @see java.util.Iterator#remove()
339  
-		 */
340  
-		public void remove() {
341  
-			throw new UnsupportedOperationException("remove method is not supported");
342  
-		}
343  
-		
344  
-	}
345  
-	
346  
-	/**
347  
-	 * @param objects
348  
-	 * @param indexes
349  
-	 * @return
350  
-	 */
351  
-	private static Entry<Object[], int[]> createEntry(Object[] objects, int[] indexes) {
352  
-		return new Map.Entry<Object[], int[]>() {
353  
-			
354  
-			private Object[] key;
355  
-			private int[] value;
356  
-
357  
-			public Object[] getKey() {
358  
-				return key;
359  
-			}
360  
-
361  
-			public Entry<Object[], int[]> init(Object[] curObjects,
362  
-					int[] curIndexes) {
363  
-				this.key   = curObjects;
364  
-				this.value = curIndexes;
365  
-				return this;
366  
-			}
367  
-
368  
-			public int[] getValue() {
369  
-				return value;
370  
-			}
371  
-
372  
-			public int[] setValue(int[] arg0) {
373  
-				return value;
374  
-			}
375  
-		}.init(objects, indexes);
376  
-	}
377  
-
378  
-	private List<FrameworkMethod> getParametersMethods(TestClass testClass)
379  
-			throws Exception {
380  
-		List<FrameworkMethod> parametersMethods = new LinkedList<FrameworkMethod>();
381  
-		List<FrameworkMethod> methods = testClass
382  
-				.getAnnotatedMethods(Parameters.class);
383  
-		for (FrameworkMethod each : methods) {
384  
-			int modifiers= each.getMethod().getModifiers();
385  
-			if (Modifier.isStatic(modifiers) && Modifier.isPublic(modifiers))
386  
-				parametersMethods.add(each);
387  
-		}
388  
-
389  
-		if (parametersMethods.isEmpty()) {
390  
-			throw new Exception("No public static parameters methods on class "
391  
-					+ testClass.getName());
392  
-		}
393  
-		
394  
-		return parametersMethods;
395  
-	}
396  
-
397  
-	/* (non-Javadoc)
398  
-	 * @see org.junit.runners.Suite#getChildren()
399  
-	 */
400  
-	@Override
401  
-	protected List<Runner> getChildren() {
402  
-		return runners;
403  
-	}
404  
-
405  
-}
3  src/test/java/org/junit/tests/experimental/ExperimentalTests.java
@@ -7,7 +7,6 @@
7 7
 import org.junit.tests.experimental.results.ResultMatchersTest;
8 8
 import org.junit.tests.experimental.runners.CombinationTestTest;
9 9
 import org.junit.tests.experimental.runners.ParameterizedTestTest;
10  
-import org.junit.tests.experimental.runners.PermutationTestTest;
11 10
 import org.junit.tests.experimental.theories.ParameterSignatureTest;
12 11
 import org.junit.tests.experimental.theories.ParameterizedAssertionErrorTest;
13 12
 import org.junit.tests.experimental.theories.extendingwithstubs.StubbedTheoriesTest;
@@ -26,7 +25,7 @@
26 25
 		ParameterSignatureTest.class, WhenNoParametersMatch.class,
27 26
 		WithExtendedParameterSources.class, StubbedTheoriesTest.class, 
28 27
 		WithOnlyTestAnnotations.class, ParameterizedTestTest.class,
29  
-        CombinationTestTest.class, PermutationTestTest.class })
  28
+        CombinationTestTest.class })
30 29
 public class ExperimentalTests {
31 30
 
32 31
 }
505  src/test/java/org/junit/tests/experimental/runners/CombinationTestTest.java
@@ -4,9 +4,11 @@
4 4
 import static org.junit.Assert.fail;
5 5
 
6 6
 import java.util.HashMap;
  7
+import java.util.HashSet;
7 8
 import java.util.LinkedList;
8 9
 import java.util.List;
9 10
 import java.util.Map;
  11
+import java.util.Set;
10 12
 
11 13
 import org.junit.Test;
12 14
 import org.junit.experimental.runners.Combination;
@@ -55,12 +57,12 @@ private Result executeTest(Class<?> testClass) {
55 57
 	@RunWith(Combination.class)
56 58
 	public static class MissingAttributeTest {
57 59
 		
58  
-		@Attributes
59  
-		public static Map<String, Object[]> inputs() {
60  
-			Map<String, Object[]> map = new HashMap<String, Object[]>();
61  
-			map.put("nonExistentAttribute", new Object[] { 1, 2 });
62  
-			map.put("aValue", new Object[] { 1 });
63  
-			return map;
  60
+		@Attributes(attributes = {"nonExistentAttribute", "aValue"})
  61
+		public static List<Object[]> inputs() {
  62
+			List<Object[]> list = new LinkedList<Object[]>();
  63
+			list.add(new Object[] { 1, 2 });
  64
+			list.add(new Object[] { 1 });
  65
+			return list;
64 66
 		}
65 67
 
66 68
 		public void setAValue(int aValue) {
@@ -87,11 +89,11 @@ public void testMissingAttributeTestCount() {
87 89
 	@RunWith(Combination.class)
88 90
 	public static class IncompatibleAttributesTest {
89 91
 		
90  
-		@Attributes
91  
-		public static Map<String, Object[]> inputs() {
92  
-			Map<String, Object[]> map = new HashMap<String, Object[]>();
93  
-			map.put("value", new Object[] { 1.0, 2.0, 3.0 });
94  
-			return map;
  92
+		@Attributes(attributes = { "value" })
  93
+		public static List<Object[]> inputs() {
  94
+			List<Object[]> list = new LinkedList<Object[]>();
  95
+			list.add(new Object[] { 1.0, 2.0, 3.0 });
  96
+			return list;
95 97
 		}
96 98
 
97 99
 		public void setValue(int value) {
@@ -117,11 +119,11 @@ public void testIncompatibleAttributesTestFailureCount() {
117 119
 	@RunWith(Combination.class)