Figure out how to parallelize tests

[Ichimonji10]

Pulp Smash takes a while to execute, and it'd be nice if we could reduce the running time. One way to accomplish this is to parallelize tests. If parallel tests are desired, then two problems need to be resolved:

• Which test cases can be run in parallel?
• How does the test runner actually run parallel tests?

As for the first question, it's good to realize that many test cases perform global actions. For example, a test case might:

• Completely reset Pulp, including resetting services, recreating a database and removing files from the filesystem. (See reset_pulp().)
• Reset individual services. (See reset_squid() and the ServiceManager class.)
• Remove all orphans.
• Munge the filesystem of the target system.
• More?

Any test case which is to be run in parallel must either be refactored to not do these things or must be marked as requiring serial execution.

As for the second case, the unittest test runner from the standard lib doesn't provide any facilities for parallelization. Maybe it can be done with unittest, but I'm unclear on how. We might need to whip up something interesting with a library like testtools, or we might need to pick a different test runner.

[elyezer]

As for the second case, the unittest test runner from the standard lib doesn't provide any facilities for parallelization. Maybe it can be done with unittest, but I'm unclear on how. We might need to whip up something interesting with a library like testtools, or we might need to pick a different test runner.

I think we should consider picking py.test. It provide some interesting features other than being able to run in parallel (on a single or multiple machines as well as single and multiple python processes). Features like py.test fixtures may help on ensuring the system state for those tests.

I will leave another option which could be having multiple Pulp servers using libvirt or docker so we could make each test in parallel act on its own Pulp server and then some reset function may become irrelevant on this setup.