Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubeflow not ready for K8s 1.22 #6098

Closed
DomFleischmann opened this issue Aug 18, 2021 · 17 comments
Closed

Kubeflow not ready for K8s 1.22 #6098

DomFleischmann opened this issue Aug 18, 2021 · 17 comments

Comments

@DomFleischmann
Copy link

/kind bug

With the new Kubernetes 1.22 release, some APIs have been removed: https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/

The kubeflow project still uses some of these APIs and will therefore not work on k8s 1.22.

For example, the following CRDs are still using apiextensions.k8s.io/v1beta1:

poddefaults.kubeflow.org
notebooks.kubeflow.org
tensorboards.tensorboard.kubeflow.org
@jbottum
Copy link
Contributor

jbottum commented Aug 18, 2021

@kubeflow/wg-manifests-leads

@jbottum
Copy link
Contributor

jbottum commented Aug 18, 2021

/priority p2
/area manifests

@google-oss-robot
Copy link

@jbottum: The label(s) area/manifests cannot be applied, because the repository doesn't have them.

In response to this:

/priority p2
/area manifests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@bmccord2
Copy link

bmccord2 commented Aug 27, 2021

It looks like this may be preventing me from setting up kubeflow with microk8s entirely (at least with gpu support). With microk8s 1.21 the gpu-operator fails to start, it goes into a crash loop complaining that no runtime for "nvidia" is configured. I have nvidia drivers already installed on this machine and I see in the release notes for microk8s 1.22 there is "Nvidia operator v1.7.0 can now detect pre-installed drivers.".

With microk8s 1.22 the gpu addon starts up just fine. What's weird is I can't find anyone else talking about this, so I assumed its just because I didn't know what I was doing (which I don't really). But the fact that it works with 1.22 and not with 1.21 makes me think this is in fact due to an issue with microk8s <=1.21. I can't find the link now, but I did see someone mention that they had to uninstall existing nvidia drivers, which isn't really an option for me on this machine.

@DomFleischmann
Copy link
Author

@bmccord2 There are some known issues with the nvidia operator and microk8s 1.21 you can try with microk8s 1.20 which uses the older and a bit more limited nvidia operator. The installation instructions would be something like this:

sudo snap install microk8s --channel=1.20/stable
microk8s enable kubeflow --bundle=cs:kubeflow

@bmccord2
Copy link

@DomFleischmann thanks, I might give that a try later, for right now we just switched to k3d for our initial kubeflow environment.

@Enzo90910
Copy link

I just had this issue while trying to install KFP on recent versions (>= 4.3) of Docker for Desktop, which include k8s 1.22. I don't think I can offer a MR, but I am very willing to help test one since this will be a major problem for us soon.

@reach4nk
Copy link

reach4nk commented Feb 7, 2022

Is someone looking at this or is there any update?

@Talador12
Copy link

Talador12 commented Jun 15, 2022

v1.21 of Kubernetes is reaching end of supported lifetime for some cloud providers/companies. This should be an item with regular community updates and an expected time of arrival. Is there a roadmap for v1.22 support posted anywhere?

@khteh
Copy link

khteh commented Jul 5, 2022

+1

@padrian2s
Copy link

I think is possible to manually install some Kubeflow components that are 1.22 ready.

Istio 1.11
Kserve
Knative
Profile
etc..

you have to try them out

@davidxia davidxia moved this from To Do to High priority in Needs Triage Jul 20, 2022
@davidxia davidxia moved this from High priority to To Do in Needs Triage Jul 20, 2022
@davidxia
Copy link

I just found the Slack 1.6 announcement with support for K8s 1.22. And this release page. This can probably be closed.

@davidxia davidxia moved this from To Do to Closed in Needs Triage Aug 16, 2022
@jyoder-deloitte
Copy link

I just found the Slack 1.6 announcement with support for K8s 1.22. And this release page. This can probably be closed.

I don't think it can be closed until after release day. They have already missed 2 expected release dates.

@davidxia
Copy link

davidxia commented Sep 1, 2022

@jyoder-deloitte ah, where do you see them missing the dates?

@jinnovation
Copy link

I don't think it can be closed until after release day

Kubeflow 1.6 has been released: https://github.com/kubeflow/kubeflow/releases/tag/v1.6.0

@juliusvonkohout
Copy link
Member

/close

There has been no activity for a long time. Please reopen if necessary.

@google-oss-prow
Copy link

@juliusvonkohout: Closing this issue.

In response to this:

/close

There has been no activity for a long time. Please reopen if necessary.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kubeflow-bot kubeflow-bot removed this from Closed in Needs Triage Aug 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests