Provide api call to Stop experiment

[johnaohara]

There are use cases that require stopping an already running experiment. This PR provides REST & gRPC endpoints to allow users to stop a running experiment.
The PR includes sanity checks for remove removing running tests.

One commit that is not directly related to the main functionality of the PR is some changes to speed up running the test suite. Running the full testsuite went from 2562 seconds to 107 seconds on my machine, and allowed me to find a couple of race/locking issues. If it is deemed appropriate to separate out that change, I can revert that commit in this PR and open a separate PR

[dinogun]

@johnaohara Thank you for adding this, should make it easier for scripts to integrate HPO into workflows thanks to this feature. Just one comment, can you update the API doc for the new API

[johnaohara]

@dinogun np, I will update the API doc

[johnaohara]

@dinogun I have updated the API Doc, and modified the response code to be consistent with the other POST commands. Quick question, is there a reason that the rest api's return -1 with a http 4xx response, and not return a message that is meaningful for the client?

[dinogun]

@dinogun I have updated the API Doc, and modified the response code to be consistent with the other POST commands. Quick question, is there a reason that the rest api's return -1 with a http 4xx response, and not return a message that is meaningful for the client?

@johnaohara Yes, that was convenient for us when HPO was part of Autotune and returning specific error codes. This needs to be changed going forward.

[dinogun]

@chandrams Are you done with your test changes?

[chandrams]

@dino - Yes, I 'm done with the test changes.

[dinogun]

LGTM