[tune] Exception raised when there is no more trials

[old-bear]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 10.11
• Ray installed from (source or binary): binary
• Ray version: 0.5.3
• Python version: 2.7
• Exact command to reproduce:

Describe the problem

For SeachAlgorithm, if next_trials returns empty and is_finished yields True. The following happens in TrialRunner:

   def step(self):
        ...
        next_trial = self._get_next_trial()      <----- no more trial, which yields None here
        if next_trial is not None:
            self.trial_executor.start_trial(next_trial)
        elif self.trial_executor.get_running_trials():      <--- all trials completes, so no more running trials
            self._process_events()
        else:
            ....
            raise TuneError("Called step when all trials finished?")    <--- reach here

Source code / logs

See above

For the solution, I think we can add

if not self._search_alg.is_finished():
    raise(...)

[richardliaw]

What is the situation where this occurs? Do you have an example for reproducing?

…

On Tue, Oct 16, 2018 at 2:33 AM old-bear ***@***.***> wrote: System information - *OS Platform and Distribution (e.g., Linux Ubuntu 16.04)*: MacOS 10.11 - *Ray installed from (source or binary)*: binary - *Ray version*: 0.5.3 - *Python version*: 2.7 - *Exact command to reproduce*: Describe the problem For SeachAlgorithm, if next_trials returns empty and is_finished yields True. The following happens in TrialRunner: def step(self): ... next_trial = self._get_next_trial() <----- no more trial, which yields None here if next_trial is not None: self.trial_executor.start_trial(next_trial) elif self.trial_executor.get_running_trials(): <--- all trials completes, so no more running trials self._process_events() else: .... raise TuneError("Called step when all trials finished?") <--- reach here Source code / logs See above For the solution, I think we can add if not self._search_alg.is_finished(): raise(...) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3069>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEUc5TDMU6F7o93IkEQSC7JmnLn8uqGaks5ulafMgaJpZM4Xd883> .

[ericl]

We already check search_alg.is_finished() in runner.is_finished(), so I don't think we should even be entering step() if the search algo is finished.

[old-bear]

The situation may be a little tricky:

class MySearchAlgorithm(SearchAlgorithm):
    def next_trials(self):
        # doing some calcuation
        # here is_finished is still False
        ... 
        # and decide we should stop here
        self._is_finished = true
        return []

    def is_finished(self):
        return self._is_finished 

When using this algorithm, the check in tune.py before will pass:

    while not runner.is_finished():
        runner.step()     <-- run step here

, and thus trigger this problem.

As for the search algorithm, although we can move all the calculation into on_trial_result and just return the calculated ones in next_trials to avoid this problem, I think that would be quite unnatural

[ericl]

I see, that makes sense!