More flexible restart with presets

[stefankraft]

Current behavior

1. When running "polyaxon ops restart --uid= -f <my_preset>.yml" the --eager flag is not supported so that the restart cannot trigger a hyperparameter tuning run in Polyaxon CE. Trying to do so leads to
   Could not restart run 2921e5a2cfc34530983e22045558bed6.
   Error message: (400)
   Reason: Bad Request
   HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.14.2', 'Date': 'Wed, 21 Oct 2020 11:50:54 GMT', 'Content-Type': 'application/json', 'Content-Length': '129', 'Connection': 'keep-alive', 'Allow': 'POST, OPTIONS', 'X-Frame-Options': 'DENY', 'Vary': 'Origin, Cookie', 'X-Content-Type-Options': 'nosniff'})
   HTTP response body: ["Cloning was not successful, error: You cannot create this operation, your account does not support operations of kind: matrix"]
   .
   Statuts: 400. One or more request parameters is incorrect

2. When running "polyaxon ops restart --uid= -f <my_preset>.yml" only the input params which existed in the original run can be modified in the preset. Examples:
   Restart a run with a new dataset (e.g. cifar10), but the original run limited datasets via options as {name: dataset_name, type: str, value: mnist, isOptional: true, options: [mnist, imnet]} leads to PolyaxonRunFailed with message "Run compilation error: Compilation Error: Could not parse value cifar10, an error was encountered: The value cifar10 provided for key dataset_name is not one of the possible values."
   If a new parameter is introduced, e.g. to resize the data which did not exist in the original run, the restart leads to PolyaxonRunFailed with message "Run compilation error: Compilation Error: Received unexpected params {'resize_data'}"

Enhancement

1. Allow the --eager flag for polyaxon ops restart in the same way as with polyaxon run
2. Allow for arbitrary modifications or additions of params of original runs when restarting with presets

[mmourafiq]

Hi Stefan, thank you for opening this issue.

For the first question this is an interesting feature enhancement, and we will add it to the roadmap.
For the second question:

• Compilation Error: Could not parse value cifar10, an error was encountered: The value cifar10 provided for key dataset_name is not one of the possible values." is a valid error because the original component has a restriction and defines the possible options, i.e. the author expects Polyaxon to raise an error if the value is not one of the values defined.
• For additional params similar to your example resize_data, you just need to set the params as contextOnly which will tell the CLI to not validate against the Inputs/Outputs. I am not sure if this will work on the current CLI/Server you are using, but I am sure that it works on v1.2 because we improved the context only validation for params.

[stefankraft]

Hi Mourad,
great to hear that you will add enhancement 1 to your roadmap.
Concerning enhancement 2:
I can confirm that contextOnly works using v1.2. Thanks for the hint. However, I noticed that if I add a new param with contextOnly which did not exist in the original run then this new parameter is not documented in the restarted run under "Inputs". If I change an existing parameter using contextOnly then this also effects the documentation (as it should).
Concering the restrictions via "option" I would argue that it might be nice to be able to overwrite the restrictions in a restart with preset. Just like in my example I may have a parameter to select a dataset with some restricted options which may however change in the future if I support further datasets. Maybe "contextOnly" could also be used to overwrite "options". I however also see that this may be dangerous and may lead to unexpected behavior.

[mmourafiq]

I think the contextOnly flag covers, or hide for that matter, 2 important aspects that might be exposed individually, skipValidation and isHidden.

I think the isHidden is something that was discussed before, that might make sense in situation where a component has several inputs/outputs and some of them do not make too much sense to be on viewed in the UI/CLI, or at least not by default.

For skipValidation that's something that could make sense by introducing a warning when the user use such flag, but I am not sure when a user would want to disable the validation system apart from the restart/resume use-case.

Will think about this and see if we can introduce such features to enable advanced or not straightforward situations.