Error When Installing Kubeflow on Azure AKS

[snowneji]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]
I'm following the instructions here: https://www.kubeflow.org/docs/azure/deploy/install-kubeflow/ to install kubeflow on Aks. Everything works until this step kfctl apply -V -f ${CONFIG_URI}.

This Error keeps pop up with retries and eventually failed:

WARN[0008] Encountered error applying application istio: (kubeflow.error): Code 500 with message: Apply.Run : unable to recognize "/tmp/kout554230921": no matches for kind "ClusterRbacConfig" in version "rbac.istio.io/v1alpha1" filename="kustomize/kustomize.go:266"
WARN[0008] Will retry in 2 seconds. filename="kustomize/kustomize.go:267"

What did you expect to happen:
I'm expecting Kubeflow to be correctly installed on AKS

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Would it be any version issue?

Environment:

• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): Not able to install yet
• kfctl version: (use kfctl version): kfctl v1.1.0-0-g9a3621e
• Kubernetes platform: (e.g. minikube) AKS
• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.13", GitCommit:"30d651da517185653e34e7ab99a792be6a3d9495", GitTreeState:"clean", BuildDate:"2020-10-20T16:22:33Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release): macOS Catalina

• ISTIO VERSION: tried 1.7.3,1.7.5

[berndverst]

/assign berndverst

[berndverst]

@snowneji currently it only works with 1.6.X (if you edit the manifest file you'll see that as a comment) -- I tested with 1.6.14. We're looking into why it doesn't work with Istio 1.7.X. I encountered similar problems myself. Will also make sure we call this out more in the documentation.

@sdake @howardjohn FYI. I too was seeing issues like this with deploying Kubeflow to AKS with Istio 1.7.3 (which I installed via Istio operator)

[berndverst]

/cc sdake howardjohn

[snowneji]

Thank you, I'll try with 1.6.X

[sdake]

can you file any information you have about a 1.7 failure. this would better enable us to reproduce.

…

On Tue, Dec 1, 2020 at 8:06 PM Yifan Wang ***@***.***> wrote: Thank you, I'll try with 1.6.X — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5434 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFYRCLLFDMSTDV24TWNKPLSSWVKDANCNFSM4UB75LJA> .

[snowneji]

The program will keep retrying and receiving the following error:

WARN[0008] Encountered error applying application istio: (kubeflow.error): Code 500 with message: Apply.Run : unable to recognize "/tmp/kout554230921": no matches for kind "ClusterRbacConfig" in version "rbac.istio.io/v1alpha1" filename="kustomize/kustomize.go:266"
WARN[0008] Will retry in 2 seconds. filename="kustomize/kustomize.go:267"

[snowneji]

Also just found a similar issue: #5391

[sdake]

RBAC v1 has been replaced by a beta implementation. I am on mobile atm - please consult with @howardjohn

…

On Tue, Dec 1, 2020 at 8:56 PM Yifan Wang ***@***.***> wrote: Also just found a similar issue: #5391 <#5391> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5434 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFYRCNP7Y4KSV5QH5SZ6VTSSW3F3ANCNFSM4UB75LJA> .

[snowneji]

It turned out my issue was due to resources under cert-manager namespace was incorrectly installed. It disappeared once I fixed the installation issue in cert-manager namespace

[berndverst]

/close

[k8s-ci-robot]

@berndverst: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[berndverst]

@snowneji out of curious in which namespace was cert-manager installed? I double-checked though it appears it should install in cert-manager namespace automatically.