add a --deterministic option

[lionelperrin]

I'm going to implement a --deterministic option. Before, I would appreciate your opinion on this idea.

I would like that, when launched several times, pabot distributes the test suites in a deterministic manner.

This will require to add a new distribution function that split the list of suite names into N parts and then launch N processes, one for each part.

What do you think ?

[mkorpela]

Is this for re-running problems that are related to specific test execution orders?

I bet that it is very difficult to get the same specific execution orders as it is the responsibility of operating systems scheduler to give CPU time to different processes.. So I'm against this feature, if you can't do it in a way that really guarantees same timings - and I think it is impossible to do it.

[lionelperrin]

Yes, this aims at re-running using the same execution order. In my case, I'm going to use 4 processes and only the execution order of each of this process matters.

But I agree with your point of view, 'deterministic' is too promising, I plan to specify the execution order of each process and not the overall execution order with the same timings. Does it make sense for you ?

[laurentbristiel]

@lionelperrin if the deterministic option if for a given process, shouldn't it be a Robot Framework option rather than a pabot one?

[lionelperrin]

@laurentbristiel Robotframework now has the ability to re-run in the same execution order, even in random mode (https://code.google.com/p/robotframework/issues/detail?id=1673)
For pabot, the difficulty comes from the test distribution among the subprocesses. This distribution is computed on the fly by the multiprocessing python library. In my case, I would like to compute the distribution of the tests in a first step, and distribute the test on the subprocesses in a deterministic manner.

[mkorpela]

Current logic is based on a worker pool - the thread that is first ready will get to execute the next suite.
Actually it will create a new process for each suite, but the number of concurrent processes is limited.

So currently the only thing that controls the execution order is the timings of executed suites - and in most cases they are changing a bit.

I have to say that for a parallel tool like pabot, the tool must expect that tests (or at least test suites) that are executed are independent of each other OR the dependencies between tests must be handled in the test cases. So there shouldn't be any reason for this kind of functionality - and I also think it would be very hard to implement it.

[mkorpela]

If the tests are failing because of an interference between them (two test cases somehow affect each other), re-running that problem with the same timings would be very convenient. I would be interested in first getting an idea how your going to implement this?

I think the solution would have to be somehow based on the recordings (output.xmls) to get the timings correct. Maybe a new tool altogether - not pabot?

[lionelperrin]

I think you're right about the complexity of implementing something based on the recordings. I have the feeling that it would add too much complexity to pabot and it's not suitable to implement it in pabot.

A '--deterministic' option (or a not so promising option name) is quite easy to implement (see https://github.com/lionelperrin/pabot/commits/deterministic) and very useful in my use case.

Nevertheless, I can understand that this need is not shared by the community.

[mkorpela]

@lionelperrin: I checked the code. I'm just not understanding the benefit of this - so could you explain it to me?

For example in a situation where we have 3 suites (s1, s2, s3) distributed between two workers.

Currently they would be executed in this order:
s1 and s2 start at the same time
s3 starts to execute when first one of s1 and s2 is ready.
Total execution time: E1 = max(min(timeof(s1), timeof(s2)) + timeof(s3), max(timeof(s1), timeof(s2)))

In your case they would be executed in the following way:
s1 and s2 start at the same time.
s3 starts when s1 is finished.
Total execution time: E2 = max(timeof(s1) + timeof(s3), timeof(s2))

And the ugly truth is that E1 <= E2

So why would you want to do it?

[lionelperrin]

@mkorpela: You're right about the execution times. With the option enabled, there is no load balancing anymore and the total time is longer than without.

If you have 10 suites and 2 workers, with the option enabled, you can garantee that
worker1 will process s1, s3, s5, s7, s9 while worker2 will process s2, s4, s6, s8, s10.

The added value is that if your worker is buggy and can't process s3 after s1, then you'll see it and have a way to reproduce it.
Without this option, in this case, some run might work (if s2 is shorter than s1), some other might fail.

[mkorpela]

First of all the deterministic option only gives one possible execution order - thus it will reduce the possibility to detect this kind of errors (IMHO bad thing - more detected bugs is better than less). This is very much discussed area of test automation (see for example http://www.satisfice.com/repeatable.shtml ).

Second point is about probability of this mechanism working as expected.
If we have N parallel executors the probability of two interfering suites ending up to the same is 1/N.
So in 1-1/N cases the mechanism will fail. N >= 2 so this number is alway greater than 50% - so most times the deterministic option will fail.

[mkorpela]

@lionelperrin I've created a small script to visualise the execution order - might be that something based on it could be used to really re-generate the same specific execution order.
https://gist.github.com/mkorpela/d4954e094db810aa7fb4

[mkorpela]

Closing this issue as there hasn't been any activity or arguments against my points at the end of discussion