EXP_STOP operation removes both completed and incomplete experiment

[kusumachalasani]

EXP_STOP is intended to cancel the experiment when the experiment is incomplete.

Right now, it deletes the experiment irrespective of if an experiment is complete or not.
Because of this, the plots generated for the completed experiment become inaccessible.

@johnaohara

[kusumachalasani]

@johnaohara
As discussed, shall we rename 'EXP_STOP' to 'EXP_REMOVE' as it deletes the experiment?
If this operation is used, it means we are deleting the experiment irrespective of whether it is complete or not. It even deletes the plots the completed experiment has generated.

If the user wants to restart the same experiment, the user has to do EXP_REMOVE and then EXP_START. The caveat of this operation would be the user will lose the available data for that experiment.
The user can view the experiment status before applying this operation.
For REST service, the user can view the listExperiments page (http://localhost:8085/listexperiments) to view the status of the experiment. We can have similar functionality for gRPC as well.

[johnaohara]

@kusumachalasani , thanks for opening this issue.

The original intention for EXP_STOP was to stop a running experiment. As the service was stateless, and an experiment was essentially paused until the next EXP_TRIAL_GENERATE_NEW call, stopping an experiment semantically meant removing it from the running experiments. I agree that the naming should be clarified to represent what the action is, i.e. removing it from HPO, e.g EXP_REMOVE

Now that HPO retains state, and has functions to generate charts after the trials have been completed. What is the expected behavior if a user wants to start a new set of trials, i.e using the same configuration? i.e. what is the expected lifecycle of a set of results? if someone starts a new experiment, using the same configuration, is the old data erased? if not, how would the experiments be uniquly identified?

[kusumachalasani]

@johnaohara
experiment_name is the primary key for experiments. If we want to run a new set of trials using the same configuration,

1. user can change the experiment_name in the search space and start the experiment. The configuration of the experiment remains the same.
2. delete the experiment and start the same experiment again (experiment_name can be the same as previous). All the previous data available for that experiment gets inaccessible.

The data for an experiment is available until that experiment exists in HPO.

[kusumachalasani]

@johnaohara I updated the operation from EXP_STOP to EXP_DELETE and also updated the function names to use delete instead of stop. I did update the function names for gRPC as well.
All the changes are part of the PR #117. Can you please review the same?

[kusumachalasani]

Closing this as the changes required went in as part of PR #117.