Python SDK for Katib and Kubeflow Pipelines can not be installed at the same time

[thesuperzapper]

What happened?

Currently, If you try install the latest versions of the kfp and kubeflow-katib libraries at the same time, there is a dependency version conflict.

This is caused by a number of conflicts:

1. kfp==2.7.0 needs protobuf>=4.21.1,<5 but kubeflow-katib needs protobuf>=3.19.5,<=3.20.3
2. kfp==2.7.0 needs kubernetes>=8.0.0,<27 but kubeflow-katib needs kubernetes>=27.2.0

See for yourself by trying the following command:

pip install kfp==2.7.0 kubeflow-katib==0.17.0rc0

What did you expect to happen?

The latest versions of both kfp and kubeflow-katib can be installed at the same time.

Environment

Kubernetes version:

$ kubectl version

Katib controller version:

$ kubectl get pods -n kubeflow -l katib.kubeflow.org/component=controller -o jsonpath="{.items[*].spec.containers[*].image}"

Katib Python SDK version:

$ pip show kubeflow-katib

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[thesuperzapper]

For those wondering, the latest versions which will work together are:

pip install kfp==2.5.0 kubeflow-katib==0.16.0

[thesuperzapper]

Can the @kubeflow/pipelines and @kubeflow/wg-automl-leads please work together to try and prevent these kind of dependency conflicts in the python packages, so we can always install the latest versions of both in Kubeflow Notebooks?

[tenzen-y]

IIRC, This protobuf version limitation is brought us from #2141.
So, Unless we resolve the #2141, we can not update the protobuf version to v4.

[thesuperzapper]

I have raised a similar issue for KServe, which also has a dependency conflict with protobuf: kserve/kserve#3730

[tenzen-y]

/assign
After #2344 is merged, I will open a PR to resolve this conflict.