Parallel test runs from libvirt

[sathnaga]

Idea:
How about having tests run in parallel for different flavours of guests(say, rhel64, rhel65, rhel70)??

E:g:- Running a test (e:g:- virsh schedinfo or virsh dominfo or unattended install) for virt-tests-vm1(rhel64) and virt-tests-vm2(rhel65) at same time.

Needs to be looked at:
1.Separate the guest centric tests( have a flag).
2. Restrict libvirt daemon restarts while running parallel mode
3. Skipping tests(running tests one time) which are Host level, which does not need any VM input.(e:g:- virsh hostname, virsh nodeinfo etc)
4. Might need a rework on log collection as many jobs write at same time.
5.

I remember there was parallel mechanism available from qemu earlier, I guess that was from same image using same base image (snapshot option) and different tests at same time??

@lmr @ldoktor @cevich @yumingfei : Provide your thoughts, comments and point me if such cases are already possible :-)

Thanks in advance.

[cevich]

Though it may not be that relevant here, my team tried this very briefly in Docker Autotest. It resulted in a horrible mess! Even once you get beyond implementing all the required synchronization between processes, you're still in a situation where tests can unpredictably clash with each-other. These problems take up a LOT of time to debug, and it's impossible to catch all of them since you can almost never predict the entire, exact sequencing of events. So IMHO, the simplest, lowest-variable count way is much more efficient (in development hours). Esp. since you can always split the entire test load between multiple test boxes, where there is no chance for contention and no synchronization is required.

[yumingfei]

@sathnaga
As @cevich said, parallel jobs may mess things up.
Because we never know what conflicts in two different test module.
Take an example, "virsh reboot" and "virsh start" shouldn't work in parallel if we can't make sure its cleanup.
So my idea is that you can design parallel jobs in one test module since we can control it.

[lmr]

Parallel testing is of course valuable, but hard to do right. @cevich has a point, although this is something we want to tackle in avocado. In the early days, pre virt-test we had parallel execution of tests, but today the functionality bit rotted.

[yumingfei]

And the idea you want to run jobs for different guests on single host is good(In fact, we already have similar cases, even not unified :) ).
So I advise that you should design new cases for these parallel jobs.

[sathnaga]

@cevich @lmr @yumingfei : Thanks for the comments.
@cevich: Sure, I agree with your points, and It would be good to have it a try, anyways as @lmr mentioned we can take this to avocado as we would have more flexibility I believe?

@yumingfei : Not different tests at same time, same test say, "virsh start" for different VMs.
Design new cases would be one of the solution:+1: but difficult to maintain and the framework support would sustain. We can have available tests get benefit of it as well.:-)

@lmr Sure, Please let me know once the parallel test idea gets evolved in avocado.
I hope, we will be able to use the existing tests in virt-test in avocado in future as it is or with a minimal change.

[ldoktor]

Well if I understand the requirement correctly, you might create custom scenarios in virt-test inside a single job. The only cons is you'd have to rewrite the test with this in mind. For example let's try installing 5 different guests simultaneously. All you need is to define vms = "vm1 vm2 vm3 vm4 vm5" and double all the differences with_vmXappendix (eg:image_name_image1_vm1,smp_vm2, kickstart_vm3, ...).

You wouldn't be able to benefit from multiplexer, so you can't rely on kickstart, cdrom, kernel, append being filled for you because you choose RHEL.7.0 so you'd have to hardcode this into your config (or parse the files yourself in the test, which is IMO a bit overhead).

Anyway if such test makes sense to you, just create couple of scenarios, virt-test supports synchronization primitives so you're free to do that. (at least in qemu version, I'm not all that familiar with libvirt, but as far as you define single job, it should work too).

Last but not least, don't forget that single job doesn't mean single thread. You're free to spawn couple of workers with different parameters and periodically poll for results.

[ldoktor]

@sathnaga true, it would be hardcoded, so harder to maintain (still with proper configuration, simple jobs should be bearable).

About avocado the answer is yes, but not at this moment.