[feature] Support NAS

[gaocegege]

We support some parameter search algorithms such as random search as grid search. Personally, I think it is better if we could support neural architecture search based on our system. I am not sure if it is feasible.

@YujiOshima We have some discussion on slack and I am glad to move it here.

/cc @weiweijiuzaizhe

[YujiOshima]

I agree supporting NAS is great. NAS is so important for architecture search.
But NAS suggestion service needs to return an architecture information.
The architecture may be not as simple as hyperparameter.
We should discuss how to express the architecture.

[YujiOshima]

I'm going to implement POC code for architecture search with NAS.
How about discussing the general architecture search interface with it?

[gaocegege]

SGTM! Thanks!

[YujiOshima]

/assign @YujiOshima

[jlewi]

@gaocegege Why is this P1? For P1 I'd like to use everything that's needed in order to have an initial release of Katib.

I don't think NAS is required for our initial release. So I'd suggest making this P2.

As @YujiOshima implementing NAS is actually quite difficult and in practice I'm not sure how many people will benefit from it because it requires huge amounts of compute resources which is probably too expensive. (The NAS paper used hundreds of GPUs for weeks).

[gaocegege]

OK, we can set it to p2/p3 since it is hard to implement 🤔

WDYT @YujiOshima

[YujiOshima]

Yes. In the initial release, I think that support of random, grid, BO, HyperBand and CMA-ES is sufficient.
I'm focusing to refine API now.

[gaocegege]

As discussed, we hope we could implement the experimental feature before kubeflow 1.0.

[gaocegege]

dup with #257