Cannot run same experiment twice if executor is not specified.

[bouthilx]

This

experiment.workon(main, max_trials=10)

# Need to force new max_trials because max_trials in workon overwrote the value in experiment and algo. That should also be fixed.
experiment._experiment.max_trials = 20
experiment._experiment.algorithms.algorithm.max_trials = 20
experiment.workon(main, max_trials=20)

leads to

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/orion/executor/multiprocess_backend.py](https://localhost:8080/#) in submit(self, function, *args, **kwargs)
    202         try:
--> 203             return self._submit_cloudpickle(function, *args, **kwargs)
    204         except ValueError as e:

6 frames
[/usr/local/lib/python3.7/dist-packages/orion/executor/multiprocess_backend.py](https://localhost:8080/#) in _submit_cloudpickle(self, function, *args, **kwargs)
    216         payload = cloudpickle.dumps((function, args, kwargs))
--> 217         return _Future(self.pool.apply_async(_couldpickle_exec, args=(payload,)), True)
    218 

[/usr/lib/python3.7/multiprocessing/pool.py](https://localhost:8080/#) in apply_async(self, func, args, kwds, callback, error_callback)
    361         if self._state != RUN:
--> 362             raise ValueError("Pool not running")
    363         result = ApplyResult(self._cache, callback, error_callback)

ValueError: Pool not running

The above exception was the direct cause of the following exception:

ExecutorClosed                            Traceback (most recent call last)
[<ipython-input-38-06c445e130ba>](https://localhost:8080/#) in <module>()
      2 experiment._experiment.max_trials = 20
      3 experiment._experiment.algorithms.algorithm.max_trials = 20
----> 4 experiment.workon(main, max_trials=20)
      5 print(experiment.stats)

[/usr/local/lib/python3.7/dist-packages/orion/client/experiment.py](https://localhost:8080/#) in workon(self, fct, n_workers, pool_size, reservation_timeout, max_trials, max_trials_per_worker, max_broken, trial_arg, on_error, idle_timeout, **kwargs)
    805             if self._executor is None or self._executor_owner:
    806                 with self.executor:
--> 807                     rval = runner.run()
    808             else:
    809                 rval = runner.run()

[/usr/local/lib/python3.7/dist-packages/orion/client/runner.py](https://localhost:8080/#) in run(self)
    245                     # Scatter the new trials to our free workers
    246                     with self.stat.time("scatter"):
--> 247                         scattered = self.scatter(new_trials)
    248 
    249                     # Gather the results of the workers that have finished

[/usr/local/lib/python3.7/dist-packages/orion/client/runner.py](https://localhost:8080/#) in scatter(self, new_trials)
    316 
    317             future = self.client.executor.submit(
--> 318                 _optimize, trial, self.fct, self.trial_arg, **self.kwargs
    319             )
    320             self.pending_trials[future] = trial

[/usr/local/lib/python3.7/dist-packages/orion/executor/multiprocess_backend.py](https://localhost:8080/#) in submit(self, function, *args, **kwargs)
    204         except ValueError as e:
    205             if str(e).startswith("Pool not running"):
--> 206                 raise ExecutorClosed() from e
    207 
    208             raise

ExecutorClosed:

[Delaunay]

Need to force new max_trials because max_trials in workon overwrote the value in experiment and algo. That should also be fixed.

The code does this pretty explicitly

        if self.max_trials > max_trials:
            self._experiment.max_trials = max_trials
            assert self._experiment.algorithms is not None
            self._experiment.algorithms.max_trials = max_trials

The runner is only using max_trials_per_worker
The max_trials being the total number of trials; it can only be fully constrained if it is sync in the db for other workers to pick it up as well.