Support multi-objective optimization

[anencore94]

/kind feature

Describe the solution you'd like

• katib currently use only single-objective optimization, (even though katib could display additional metrics, the optimizing process focused on single objective function)

• However, as you might know well, optuna support multi-objective optimization

  • which means, for example, we can optimize the hyperparameters to maximize accuracy and minimize time-cost simultaneously.

• In such case, I know we can define a new objective function which combines two objectives, such as maximize accuracy / time-cost or something else more complicated.

• However, I wonder is there any plan to support such feature natively in katib.

[andreyvelich]

Thank you for creating this @anencore94.
I think that feature will require some API modification since type, goal and objectiveMetricName can be unique: https://github.com/kubeflow/katib/blob/master/pkg/apis/controller/common/v1beta1/common_types.go#L94-L103.

@anencore94 @gaocegege @johnugeorge Any ideas how we can implement this ?

@c-bata Does Goptuna support Multi-object optimization ?

[c-bata]

@c-bata Does Goptuna support Multi-object optimization ?

No. I'm not planning to implement multi-objective support in Goptuna because it requires some breaking changes.

As @anencore94 said, I think Optuna is a good choice if Katib will support multi-objective optimizations.
Optuna supports a lot of multi-objective optimization algorithms like NSGA-II, multi-objective TPE, and the algorithms provided by BoTorch. And it is not difficult to implement Optuna based suggestion service since its internal design is almost similar with Goptuna.

[g-votte]

Hi, I'm one of the maintainers of Optuna. Thank you for referring to Optuna as a possible backend of the multi-objective API.

As @c-bata mentions, Optuna supports several algorithms for multi-objective optimization, with almost the same interface as single objective optimization.

I think we can go forward by (1) implementing the Optuna suggestion service, and (2) fixing the interface of Katib's multi-objective functionality.

For (1), I created the PR #1613 to add the suggestion service. I appreciate your taking a look and providing any feedback.

[anencore94]

Can we close this issue by #1613 ?

[andreyvelich]

Can we close this issue by #1613 ?

This issue still is progress. If we want to support multi-objective optimisation, we have to modify our Katib Experiment APIs.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

/lifecycle frozen
/help

[google-oss-prow]

@andreyvelich:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/lifecycle frozen
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Padarn]

I saw this issue mentioned on the 2023 roadmap, but it looks like the PR above resolves it? There is an example in the PR https://github.com/kubeflow/katib/pull/1613/files#diff-1057ae2ec900a6062a9c0c6e12e9845a4ae2c29f5cf4ae1bf9265ab6b227155c

[andreyvelich]

I saw this issue mentioned on the 2023 roadmap, but it looks like the PR above resolves it? There is an example in the PR https://github.com/kubeflow/katib/pull/1613/files#diff-1057ae2ec900a6062a9c0c6e12e9845a4ae2c29f5cf4ae1bf9265ab6b227155c

@Padarn We haven't implemented multi-objective optimization using Optuna yet.
We are looking for contributors to support this feature.
If you are interested, I am happy to discuss this in one of our upcoming Community Calls.