Why:

The new default behavior is confusing for users. It is also difficult to
determine a good default max_trials, so having not enough or to many
trials sampled by default at the start of HPO can be annoying for many
users. Using inf by default and iterating with pool-size may be the best
alternative.

Now that we have a support for n-workers, the argument pool-size we
previously deprecated actually make sense. By default, pool-size
should be equal to number of workers. We have n-workers set to 1 by
default, so by default we are back to previous behavior; sampling 1
trial at a time, until max_trials.

How:

The producer now takes a pool size as argument when producing. The same
applies to ExperimentClient.suggest() and ExperimentClient.workon(). The
pool size is used to sample multiple trials at a time and increase I/O
efficiency.

The producer now keeps track of number of new trials so that
if multiple workers are producing new trials with a non-seed algorithm
(hence they produce different trials and there are no conflicts leading
to backoff) they will stop if they generated together up to `pool_size`
trials.

Note:

Pool-size is moved to to worker configuration instead. Since pool-size
relates to n-workers, which is part of worker configuration, having
pool-size in worker configuration makes more sense.

Why:

There was a bug in the tests. The functions to generate trials would
generate more than requested because of the new behavior of producer
attempting to produce all trials at once, once the value of `max_trials`
was conflicting with the number of trials requested to the trial
generating function for tests (`orion.testing.evc.generate_trials`).

Fortunately the bug in the tests did not seem to miss any bugs in the
code they were testing.

How:

Adjust the expected numbers based on the corrected behiavor. The numbers
make indeed more sense now.

Co-authored-by: Lin Dong <michaeltunglin@gmail.com>

… into feature/back_to_pool_size