Test suite partitioning

[dataclouder]

Experimental method to partition the tests from the acceptance test suite.
In short, it works this way: you run the test with two parameters:

• -vcd-partitions=3 // this is the number of partitions
• -vcd-partition-node=1 // this is the identified of the current node

In the other two VCDs, you run the same command, but with -vcd-partition-node=2 and -vcd-partition-node=3. The tests get split in three parts, and each node gets one piece.
The partitioning works by collecting the test names from the current directory, sorting them, and assigning a node number to each name, before the test starts. The operation should give the same result in every VCD, provided that all nodes work on the same branch and commit.

Prerequisites:

1. Every test must have a preTestCheck and postTestCheck function calls
2. All VCDs participating in the test must use the same branch and commit
3. We must not run two or more partitions in the same VCD: it will probably lead to conflicts

Possible problems:

• Every node in the partition is independent, and has no connection of other nodes.
• There is no completion check: we can't be sure that all nodes have run their share.
• Users who run partitioned tests must make sure that:
  • They use the same number of partitions in every test call
  • They use different node numbers in each VCD participating to the distributed run (it is possible to run the same partition on multiple nodes, which would be a total waste of resources)
  • They start as many nodes as the number of partitions declared (starting fewer nodes will leave tests out of the suite run)

Possible enhancements:

• Every node could produce a log of the tests it ran and the ones it skipped (DONE);
• The results could be consolidated after completion;
• The node logs could be compared to make sure that all tests were considered by any node;
• A sentinel resource could be created at the start of the test suite to make sure that no calls to the test suite will run on the same VCD.

Preparation for parallel handling
This feature would work better if a centralized process were to start all the nodes, and collect the results.
For this reason, the following features were added:

• there is an option -vcd-partition-tests-file with the name of the file containing the tests that will run in the current node. If such an option was provided, the node will not attempt to create its own list of tests
• The names of all the test files produced by the node are standardized. If running under a Jenkins job, the names will contain the build number.
• A sentinel file is created at the end of the test suite, containing the exit code of the run.

[lvirbalas]

Looks great! Only there's no changelog entry.

[dataclouder]

Looks great! Only there's no changelog entry.

Added

[Didainius]

First few comments on a first look. I think it would be good to have a small paragraph about how to execute core functionality with bare hands (using newly introduced flags) in TESTING.md.

[Didainius]

I have done a few runs on smaller set of tests and it worked - thanks!

[adezxc]

Ran a small subset of tests on my own machine and worked great! Thanks for this improvement