Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubeflow Install Error #4642

Closed
jobvarkey opened this issue Jan 9, 2020 · 75 comments
Closed

Kubeflow Install Error #4642

jobvarkey opened this issue Jan 9, 2020 · 75 comments

Comments

@jobvarkey
Copy link

@jobvarkey jobvarkey commented Jan 9, 2020

Trying to install Kubeflow v.70 on on-prem cluster (kubernetes v15.7/Ubuntu 18.04) but getting below error:

root@0939-jdeml-m03:/opt/kubeflow/kf-test# kfctl apply -V -f ${CONFIG_FILE}
INFO[0000]


Notice anonymous usage reporting enabled using spartakus
To disable it
If you have already deployed it run the following commands:
cd $(pwd)
kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus

For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/


filename="coordinator/coordinator.go:120"
INFO[0000] Deleting cachedir /opt/kubeflow/kf-test/.cache/manifests because Status.ReposCache is out of date filename="kfconfig/types.go:464"
INFO[0000] Fetching https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz to /opt/kubeflow/kf-test/.cache/manifests filename="kfconfig/types.go:485"
Error: failed to build kfApp from URI /opt/kubeflow/kf-test/kfctl_existing_arrikto.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter093015857/archive: EOF
Usage:
kfctl apply -f ${CONFIG} [flags]

Flags:
-f, --file string Static config file to use. Can be either a local path:
export CONFIG=./kfctl_gcp_iap.yaml
or a URL:
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_existing_arrikto.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.0.yaml
kfctl apply -V --file=${CONFIG}
-h, --help help for apply
-V, --verbose verbose output default is false

failed to build kfApp from URI /opt/kubeflow/kf-test/kfctl_existing_arrikto.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter093015857/archive: EOF

The Kubeflow installation was working fin when tried it a few days ago.
Trying to install it in a new cluster and getting this error.
Can you please help?

Thanks,
Job Varkey

@issue-label-bot issue-label-bot bot added the kind/bug label Jan 9, 2020
@issue-label-bot

This comment has been minimized.

Copy link

@issue-label-bot issue-label-bot bot commented Jan 9, 2020

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.82. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

@kubeflow-bot kubeflow-bot added this to To Do in Needs Triage Jan 9, 2020
@haghabozorgi

This comment has been minimized.

Copy link

@haghabozorgi haghabozorgi commented Jan 9, 2020

I am having this same issue which seems to have started some time today. I was working with kfctl apply earlier with no issues now it fails.
output sample of a new build attempt

$ kfctl build -V -f ${CONFIG_URI}
INFO[0000] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.1.yaml to /tmp/927987910/tmp.yaml  filename="utils/k8utils.go:169"
INFO[0001] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.1.yaml to /tmp/245628269/tmp_app.yaml  filename="configconverters/converters.go:71"
INFO[0001] App directory /home/haghabozorgi/terraform-configs/env/dev/kf-test already exists  filename="coordinator/coordinator.go:274"
INFO[0001] Writing KfDef to kfctl_aws.0.7.1.yaml         filename="coordinator/coordinator.go:277"
INFO[0001] Running `eksctl version` ...                  filename="utils/awsutil.go:54"
INFO[0002] output: [ℹ]  version.Info{BuiltAt:"", GitCommit:"", GitTag:"0.10.2"}  filename="utils/awsutil.go:62"
INFO[0002] 
****************************************************************
Notice anonymous usage reporting enabled using spartakus
To disable it
If you have already deployed it run the following commands:
  cd $(pwd)
  kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus

For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/
****************************************************************
  filename="coordinator/coordinator.go:120"
INFO[0002] Creating directory /home/haghabozorgi/terraform-configs/env/dev/kf-test/.cache  filename="kfconfig/types.go:437"
INFO[0002] Fetching https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz to /home/haghabozorgi/terraform-configs/env/dev/kf-test/.cache/manifests  filename="kfconfig/types.go:485"
Error: failed to build kfApp from URI https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.1.yaml: couldn't generate KfApp:  (kubeflow.error): Code 500 with message: could not sync cache. Error:  (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter291586024/archive: EOF
Usage:
  kfctl build [flags]

Flags:
  -f, --file string   Static config file to use. Can be either a local path:
                                export CONFIG=./kfctl_gcp_iap.yaml
                        or a URL:
                                export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.0.yaml
                                export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_existing_arrikto.0.7.0.yaml
                                export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.0.yaml
                                export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.0.yaml
                        kfctl build -V --file=${CONFIG}
  -h, --help          help for build
  -V, --verbose       verbose output default is false

failed to build kfApp from URI https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.1.yaml: couldn't generate KfApp:  (kubeflow.error): Code 500 with message: could not sync cache. Error:  (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter291586024/archive: EOF
@wdhorton

This comment has been minimized.

Copy link

@wdhorton wdhorton commented Jan 9, 2020

I'm also running into the same issue

@srinjay-paul

This comment has been minimized.

Copy link

@srinjay-paul srinjay-paul commented Jan 9, 2020

Running into the same issue. Started around an hour ago.

@bpoti

This comment has been minimized.

Copy link

@bpoti bpoti commented Jan 9, 2020

Running into same issue.

@Zhihan1996

This comment has been minimized.

Copy link

@Zhihan1996 Zhihan1996 commented Jan 9, 2020

Same isuue.

@andersonvc

This comment has been minimized.

Copy link

@andersonvc andersonvc commented Jan 9, 2020

I tried reverting kfctl to version 0.7.0 (https://github.com/kubeflow/kubeflow/releases/download/v0.7.0/kfctl_v0.7.0_darwin.tar.gz)

and the config file to 0.7.0
(https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.0.yaml)

Seeing a EOF error similar to haghabozorgi:

couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/c0e81bedec9a4df8acf568cc5ccacc4bc05a3b38.tar.gz Error Error opening a gzip reader for /var/folders/0h/7sy0pqj150l2193wncsjhd900000gp/T/getter685306777/archive: EOF

@wronk

This comment has been minimized.

Copy link

@wronk wronk commented Jan 10, 2020

Same here. Also tried to revert to 0.7.0 without success

@slowlove1995

This comment has been minimized.

Copy link

@slowlove1995 slowlove1995 commented Jan 10, 2020

i have same issue

@nschenone

This comment has been minimized.

Copy link

@nschenone nschenone commented Jan 10, 2020

Same

@alimhanif

This comment has been minimized.

Copy link

@alimhanif alimhanif commented Jan 10, 2020

it was fine yesterday.
today i got the same issue

@raviranjan0309

This comment has been minimized.

Copy link

@raviranjan0309 raviranjan0309 commented Jan 10, 2020

Yes, It was working fine for me till last night. Today I am getting the same issue as discussed above.

@waelCh

This comment has been minimized.

Copy link

@waelCh waelCh commented Jan 10, 2020

I am also facing the same issue

Error: failed to build kfApp from URI /home/ec2-user/KF_DIR/aicluster2020/kfctl_aws.0.7.1.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter085444705/archive: EOF

@swjung94

This comment has been minimized.

Copy link

@swjung94 swjung94 commented Jan 10, 2020

i have same issue

@anirudhkm

This comment has been minimized.

Copy link

@anirudhkm anirudhkm commented Jan 10, 2020

same here.. :(

@prsrepo

This comment has been minimized.

Copy link

@prsrepo prsrepo commented Jan 10, 2020

I am also facing the same issue.

`/home/kubernetes/kubeflow-poc/kfctl apply -V -f ${CONFIG_URI}
INFO[0000] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.1.yaml to /tmp/706389512/tmp.yaml filename="utils/k8utils.go:169"
INFO[0000] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.1.yaml to /tmp/719011783/tmp_app.yaml filename="configconverters/converters.go:71"
INFO[0000] App directory /home/kubernetes/kubeflow-poc/kf-poc already exists filename="coordinator/coordinator.go:274"
INFO[0000] Writing KfDef to kfctl_k8s_istio.0.7.1.yaml filename="coordinator/coordinator.go:277"
INFO[0000] No name specified in KfDef.Metadata.Name; defaulting to kf-poc based on location of config file: /home/kubernetes/kubeflow-poc/kf-poc/kfctl_k8s_istio.0.7.1.yaml. filename="coordinator/coordinator.go:202"
INFO[0000]


Notice anonymous usage reporting enabled using spartakus
To disable it
If you have already deployed it run the following commands:
cd $(pwd)
kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus

For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/


filename="coordinator/coordinator.go:120"
INFO[0000] Creating directory /home/kubernetes/kubeflow-poc/kf-poc/.cache filename="kfconfig/types.go:437"
INFO[0000] Fetching https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz to /home/kubernetes/kubeflow-poc/kf-poc/.cache/manifests filename="kfconfig/types.go:485"
Error: failed to build kfApp from URI https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.1.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter659136634/archive: EOF
Usage:
kfctl apply -f ${CONFIG} [flags]

Flags:
-f, --file string Static config file to use. Can be either a local path:
export CONFIG=./kfctl_gcp_iap.yaml
or a URL:
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_existing_arrikto.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.0.yaml
kfctl apply -V --file=${CONFIG}
-h, --help help for apply
-V, --verbose verbose output default is false

failed to build kfApp from URI https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.1.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter659136634/archive: EOF`

@zoyun

This comment has been minimized.

Copy link

@zoyun zoyun commented Jan 10, 2020

I have the same issue

@mukulp

This comment has been minimized.

Copy link

@mukulp mukulp commented Jan 10, 2020

I am having the same issue. I even tried using kfctl_v0.7.1-2-g55f9b2a.

@vpavlin

This comment has been minimized.

Copy link

@vpavlin vpavlin commented Jan 10, 2020

Hi, I tried to use my PR branch in the KFDef and got the same fail - specifically:

uri:  https://github.com/vpavlin/manifests/archive/openshift/kfdef.zip
Code 400 with message: couldn't download URI https://github.com/vpavlin/manifests/archive/openshift/kfdef.zip Error zip: not a valid zip file

If I click the link the zip is downloaded normally and the contents seem to be ok

@ruanxingbaozi

This comment has been minimized.

Copy link

@ruanxingbaozi ruanxingbaozi commented Jan 10, 2020

+1

1 similar comment
@razumv

This comment has been minimized.

Copy link

@razumv razumv commented Jan 10, 2020

+1

@vpavlin

This comment has been minimized.

Copy link

@vpavlin vpavlin commented Jan 10, 2020

I tried to find how to get a tarball from github properly and found

https://blog.abelotech.com/posts/how-download-github-tarball-using-curl-wget/

based on that I used: https://github.com/vpavlin/manifests/tarball/openshift/kfdef

And that passed, but it fails elswhere - probably because of the branch name (as it contains /) and there is a sub-dir created in .cache/manifests. So I don't see the above described issue anymore...sorry:)

EDIT: But something is really fishy there, because the cache now contains one file which has the name of the branch and is empty (instead of a dir or content of the branch)

@descampsk

This comment has been minimized.

Copy link

@descampsk descampsk commented Jan 10, 2020

Same issue here with kfctl v0.7.0

@valentin-stefan-popa

This comment has been minimized.

Copy link

@valentin-stefan-popa valentin-stefan-popa commented Jan 10, 2020

Same here also with 0.7.0 and 0.7.1 Yesterday was working.

@leileiwan

This comment has been minimized.

Copy link

@leileiwan leileiwan commented Jan 10, 2020

+1

2 similar comments
@cyrano5614

This comment has been minimized.

Copy link

@cyrano5614 cyrano5614 commented Jan 10, 2020

+1

@ronsingler

This comment has been minimized.

Copy link

@ronsingler ronsingler commented Jan 10, 2020

+1

@nepalineupane

This comment has been minimized.

Copy link

@nepalineupane nepalineupane commented Jan 14, 2020

Confirmed as working, thank you @kunmingg

@mukulp

This comment has been minimized.

Copy link

@mukulp mukulp commented Jan 15, 2020

I built kfctl from https://github.com/kubeflow/kfctl/releases/tag/v1.0-rc.1 using go install. Is there a way of building it using make?

@zimmertr

This comment has been minimized.

Copy link

@zimmertr zimmertr commented Jan 15, 2020

Any reason you can't use one of the compiled binaries in the Darwin and Linux tarballs?

@mukulp

This comment has been minimized.

Copy link

@mukulp mukulp commented Jan 15, 2020

When I did tar -xvf on the Linux tarball it did not create the kfctl executable.

@zimmertr

This comment has been minimized.

Copy link

@zimmertr zimmertr commented Jan 15, 2020

Screen Shot 2020-01-15 at 10 55 42 AM

wget https://github.com/kubeflow/kfctl/releases/download/v1.0-rc.1/kfctl_v1.0-rc.1-0-g963c787_darwin.tar.gz
tar -xzvf kfctl_v1.0-rc.1-0-g963c787_darwin.tar.gz
./kfctl

@mukulp this works for the MacOS client. ^

@mukulp

This comment has been minimized.

Copy link

@mukulp mukulp commented Jan 15, 2020

I used https://github.com/kubeflow/kfctl/releases/tag/v1.0-rc.1 as the tarball. I did not know there was a tarball at https://github.com/kubeflow/kfctl/releases/download/v1.0-rc.1/kfctl_v1.0-rc.1-0-g963c787_linux.tar.gz. Now I know how to build using go, I will experiment with the go language and see if I can make any sense of kfctl. At some point I need to test this on ubuntu 19 and k8s above version 1.15.7.

@superkjp

This comment has been minimized.

Copy link

@superkjp superkjp commented Jan 16, 2020

Hi guys,

Trying to install Kubeflow v.70 on on-prem cluster (kubernetes v15.7/Ubuntu 18.04) but getting below error:

root@0939-jdeml-m03:/opt/kubeflow/kf-test# kfctl apply -V -f ${CONFIG_FILE}
INFO[0000]

Notice anonymous usage reporting enabled using spartakus
To disable it
If you have already deployed it run the following commands:
cd $(pwd)
kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus

For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/

filename="coordinator/coordinator.go:120"
INFO[0000] Deleting cachedir /opt/kubeflow/kf-test/.cache/manifests because Status.ReposCache is out of date filename="kfconfig/types.go:464"
INFO[0000] Fetching https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz to /opt/kubeflow/kf-test/.cache/manifests filename="kfconfig/types.go:485"
Error: failed to build kfApp from URI /opt/kubeflow/kf-test/kfctl_existing_arrikto.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter093015857/archive: EOF
Usage:
kfctl apply -f ${CONFIG} [flags]

Flags:
-f, --file string Static config file to use. Can be either a local path:
export CONFIG=./kfctl_gcp_iap.yaml
or a URL:
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_existing_arrikto.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.0.yaml
kfctl apply -V --file=${CONFIG}
-h, --help help for apply
-V, --verbose verbose output default is false

failed to build kfApp from URI /opt/kubeflow/kf-test/kfctl_existing_arrikto.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter093015857/archive: EOF

The Kubeflow installation was working fin when tried it a few days ago.
Trying to install it in a new cluster and getting this error.
Can you please help?

Thanks,
Job Varkey

Hi,
I also met this error when I was trying to re-install the kubeflow, and here is my idea:
You can first download this file via wget:
wget https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz
And then unzip the file
tar -xvf v0.7-branch.tar.gz
I guess you're trying to install the kubeflow via this file: kfctl_k8s_istio.0.7.1.yaml
So after you open the file and scroll down to the end, you will find this information
repos:

And finally, try to run this command again: kfctl build -V -f kfctl_k8s_istio.0.7.1.yaml
That should work.

@waelCh

This comment has been minimized.

Copy link

@waelCh waelCh commented Jan 16, 2020

Hi @superkjp,
I tried that solution and I remember there was some other issue that rise.

@kishoreallwynraj

This comment has been minimized.

Copy link

@kishoreallwynraj kishoreallwynraj commented Jan 19, 2020

Hi @superkjp,
I tried your steps as well and i got logs like below. But i did not see any components getting deployed to my cluster. "kubectl -n kubeflow get all" shows no resources.

INFO[0000] No name specified in KfDef.Metadata.Name; defaulting to ochack1 based on location of config file: kfctl_k8s_istio.0.7.1.yaml. filename="coordinator/coordinator.go:202"
INFO[0000]


Notice anonymous usage reporting enabled using spartakus
To disable it
If you have already deployed it run the following commands:
cd $(pwd)
kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus

For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/


filename="coordinator/coordinator.go:120"
INFO[0000] Deleting cachedir .cache/manifests because Status.ReposCache is out of date filename="kfconfig/types.go:464"
INFO[0000] Fetching /home/ec2-user/kubeflow/ochack1/manifests-0.7-branch to .cache/manifests filename="kfconfig/types.go:485"
INFO[0000] Fetch succeeded; LocalPath .cache/manifests filename="kfconfig/types.go:526"
INFO[0000] Processing application: application-crds filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: application filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: metacontroller filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: argo filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: kubeflow-roles filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: centraldashboard filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: bootstrap filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: webhook filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: jupyter-web-app filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: metadata filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: notebook-controller filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: pytorch-job-crds filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: pytorch-operator filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: knative-crds filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: knative-install filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: kfserving-crds filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: kfserving-install filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: spartakus filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: tensorboard filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: tf-job-crds filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: tf-job-operator filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: katib-crds filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: katib-controller filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: api-service filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: minio filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: mysql filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: persistent-agent filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: pipelines-runner filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: pipelines-ui filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: pipelines-viewer filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: scheduledworkflow filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: pipeline-visualization-service filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: profiles filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: seldon-core-operator filename="kustomize/kustomize.go:340"

@superkjp

This comment has been minimized.

Copy link

@superkjp superkjp commented Jan 20, 2020

Hi @kishoreallwynraj
I apologize that I miss the last step which is running:

kfctl apply -V -f kfctl_k8s_istio.0.7.1.yaml

Because "kfctl build" will make sure kubeflow creates the series of kustomized folder that contains the necessary components it needs.
So, try the command above again and let's see what happens.

Here are two things I'd like to mention:

  1. You may be stuck on the warning when your k8s is trying to create the 'cert-manager'. That's because the time it generates the pod: cert-manager-webhook takes some time, so just wait and it will take care of it.
  2. I cannot remember the name exactly but the statement is about missing one part of the components, just run the command: kfctl apply -V -f kfctl_k8s_istio.0.7.1.yaml again.

Hope this helps you.

@kishoreallwynraj

This comment has been minimized.

Copy link

@kishoreallwynraj kishoreallwynraj commented Jan 20, 2020

Thanks @superkjp.. that worked.

@sk1395

This comment has been minimized.

Copy link

@sk1395 sk1395 commented Jan 22, 2020

@superkjp pods are running fine but I am not able to access the central dashboard
and i checked ping or curl which am not able to do for any of the resources

@sk1395

This comment has been minimized.

Copy link

@sk1395 sk1395 commented Jan 22, 2020

@superkjp I am following the steps suggested but i am getting warning on applying the config_file and once i try to list the resources , I am getting only few pods but not all

kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.11", GitCommit:"25074a190ef2a07d8b0ed38734f2cb373edfb868", GitTreeState:"clean", BuildDate:"2019-09-18T14:34:46Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

Kfctl used:
wget https://github.com/kubeflow/kubeflow/releases/download/v0.7.1/kfctl_v0.7.1-2-g55f9b2a_linux.tar.gz

In kfctl_k8s_istio.0.7.1.yaml.yaml:

  • kustomizeConfig:
    overlays:
    • application
      repoRef:
      name: manifests
      path: seldon/seldon-core-operator
      name: seldon-core-operator
      repos:
  • name: manifests
    uri: /home/ubuntu/mydir/deploymentdir/mykubeflow/manifests-0.7-branch
    version: v0.7.1

Manifest used:[[extracted in local and uri updated in kfctl_k8s_istio.0.7.1.yaml.yaml]
wget https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz
Logs after applying the Config_file:
kfctl apply -V -f ${CONFIG_FILE}
INFO[0000] No name specified in KfDef.Metadata.Name; defaulting to mykubeflow based on location of config file: /home/ubuntu/mydir/deploymentdir/mykubeflow/kfctl_k8s_istio.0.7.1.yaml. filename="coordinator/coordinator.go:202"
INFO[0000]


Notice anonymous usage reporting enabled using spartakus
To disable it
If you have already deployed it run the following commands:
cd $(pwd)
kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus

For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/


filename="coordinator/coordinator.go:120"
INFO[0000] Creating directory /home/ubuntu/mydir/deploymentdir/mykubeflow/.cache filename="kfconfig/types.go:437"
INFO[0000] Fetching /home/ubuntu/mydir/deploymentdir/mykubeflow/manifests-0.7-branch to /home/ubuntu/mydir/deploymentdir/mykubeflow/.cache/manifests filename="kfconfig/types.go:485"
INFO[0000] Fetch succeeded; LocalPath /home/ubuntu/mydir/deploymentdir/mykubeflow/.cache/manifests filename="kfconfig/types.go:526"
INFO[0000] Processing application: istio-crds filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: istio-install filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: istio filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: application-crds filename="kustomize/kustomize.go:340"
"
INFO[0000] Processing application: pipeline-visualization-service filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: profiles filename="kustomize/kustomize.go:340"
INFO[0000] Processing application: seldon-core-operator filename="kustomize/kustomize.go:340"
INFO[0000] /home/ubuntu/mydir/deploymentdir/mykubeflow/.cache/manifests exists; not resyncing filename="kfconfig/types.go:460"
INFO[0000] namespace: kubeflow filename="utils/k8utils.go:425"
WARN[0000] Encountered error during apply: (kubeflow.error): Code 500 with message: Apply.Run Error [unable to recognize "/tmp/kout085395191": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1", unable to recognize "/tmp/kout085395191": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1", unable to recognize "/tmp/kout809367297": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1", unable to recognize "/tmp/kout809367297": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1", unable to recognize "/tmp/kout809367297": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1", unable to recognize "/tmp/kout809367297": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1", unable to recognize

If I try to list the resources ,I could see only few of the resources not all and some of those are going into containerconfigerror:
kubectl -n kubeflow get po,svc
NAME READY STATUS RESTARTS AGE
pod/katib-db-dd4c6b94b-jp4cv 0/1 Pending 0 41m
pod/katib-manager-7b8bf9b977-f6m9f 0/1 CreateContainerConfigError 0 41m
pod/kfserving-controller-manager-0 0/2 ContainerCreating 0 44m
pod/metadata-db-d88b6666c-pxsmb 0/1 Pending 0 41m
pod/metadata-deployment-55bc8bb977-z7fm5 0/1 CreateContainerConfigError 0 41m
pod/metadata-envoy-deployment-65bb86654c-nkshd 1/1 Running 0 41m
pod/metadata-grpc-deployment-86d98cf5c6-2fss5 0/1 CreateContainerConfigError 0 41m
pod/minio-d96d4f4cf-br6pn 0/1 Pending 0 41m
pod/ml-pipeline-ml-pipeline-visualizationserver-d494fcb84-ltfbh 1/1 Running 0 37m
pod/mysql-74578b646b-44694 0/1 Pending 0 41m
pod/tensorboard-55d9cd67bc-c7hwr 1/1 Running 0 37m

@kunmingg

This comment has been minimized.

Copy link
Contributor

@kunmingg kunmingg commented Jan 22, 2020

Close this one as fix has been released.

@kunmingg kunmingg closed this Jan 22, 2020
@superkjp

This comment has been minimized.

Copy link

@superkjp superkjp commented Jan 23, 2020

Hi @sk1395,

The relationship between the version of Kubeflow and Kubernetes is:

Kubernetes Versions Kubeflow 0.4 Kubeflow 0.5 Kubeflow 0.6 Kubeflow 0.7
1.11 compatible compatible incompatible incompatible
1.12 compatible compatible incompatible incompatible
1.13 compatible compatible incompatible incompatible
1.14 compatible compatible compatible compatible
1.15 incompatible compatible compatible compatible
1.16 incompatible incompatible incompatible incompatible

or check from here:
https://www.kubeflow.org/docs/started/k8s/overview/#minimum-system-requirements
I see the version of your k8s is 1.13.11, which illustrates that you can only choose the Kubeflow version 0.4 and 0.5. Before I upgrade my k8s version > 1,14, I also met the issue the same as yours. So here are two options:

  1. Follow the installation from here:
    https://www.lightbend.com/blog/how-to-deploy-kubeflow-on-lightbend-platform-openshift-installing-kubeflow?_ga=2.218094358.1214307035.1579078664-404386627.1572258766
    The author chose the Kubeflow version v0.4.1 and it works(version could be old and no user isolation supported).
  2. Upgrade your k8s version > 14.0 and try to install Kubeflow > 0.7.0

As the issue: you cannot get access to the central dashboard, that's because the Kubeflow works with istio. The central dashboard cannot be accessed directly with the port-forwarding with the centrol-dashboard pod/svc.
Instead, you should do with this:
kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

@sk1395

This comment has been minimized.

Copy link

@sk1395 sk1395 commented Jan 24, 2020

@superkjp Thank you

I did try port-forwarding:

ubuntu@....:~/mydir$ kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
E0124 06:20:51.577526 27900 portforward.go:224] lost connection to pod

I am trying to do a curl on central dashboard but am getting this error:

Getting cluster ip:
centraldashboard ClusterIP 100.69.194.91 EXTERNAL-IP <none> 80/TCP 25h app.kubernetes.io/component=centraldashboard,app.kubernetes.io/instance=centraldashboard-v0.7.1,app.kubernetes.io/managed-by=kfctl,app.kubernetes.io/name=centraldashboard,app.kubernetes.io/part-of=k
ubuntu@....$ curl 100.69.194.91
curl: (7) Failed to connect to 100.69.194.91 port 80: Connection timed out
//Not able to access from browser using node ip :portnumber [portnumber tried with 8080 and 31380(default ) as well but getting == This site can’t be reachedThe connection was reset.
ERR_CONNECTION_RESET]

Whereas the only thing that works is:
ubuntu@...:~mydir$ kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
ubuntu@...:mydir$ curl http://localhost :8080
<!doctype html>Kubeflow Central Dashboard</title><script src="webcomponentsjs/webcomponents-loader.js"></scrip t><script src="webcomponentsjs/custom-elements-es5-adapter.js"></script>Plea se enable JavaScript to view this website.<script src="vendor.bundle. js" defer="defer"></script><script src="app.bundle.js" defer="defer"></script>
But other than accessing the dashboard ,evrything else works fine I am able to create experiments ,monitor and everything is just fine

@prsrepo

This comment has been minimized.

Copy link

@prsrepo prsrepo commented Jan 27, 2020

I am able to install kubeflow, but while doing port forwarding i am getting this error,

$ kubectl port-forward --address 0.0.0.0 -n istio-system svc/istio-ingressgateway 8080:80
Forwarding from 0.0.0.0:8080 -> 80
Handling connection for 8080
E0127 07:15:33.945920 27086 portforward.go:400] an error occurred forwarding 8080 -> 80: error forwarding port 80 to pod 2fef66785a24af74403192b877c3a857806ebd6fa4b7de89acf880d8656c0fde, uid : exit status 1: 2020/01/27 07:15:33 socat[5157] E connect(5, AF=2 127.0.0.1:80, 16): Connection refused
Handling connection for 8080
E0127 07:15:33.956317 27086 portforward.go:400] an error occurred forwarding 8080 -> 80: error forwarding port 80 to pod 2fef66785a24af74403192b877c3a857806ebd6fa4b7de89acf880d8656c0fde, uid : exit status 1: 2020/01/27 07:15:33 socat[5158] E connect(5, AF=2 127.0.0.1:80, 16): Connection refused
Handling connection for 8080
E0127 07:15:33.965535 27086 portforward.go:400] an error occurred forwarding 8080 -> 80: error forwarding port 80 to pod 2fef66785a24af74403192b877c3a857806ebd6fa4b7de89acf880d8656c0fde, uid : exit status 1: 2020/01/27 07:15:33 socat[5159] E connect(5, AF=2 127.0.0.1:80, 16): Connection refused
Handling connection for 8080

@Anujay-Saraf

This comment has been minimized.

Copy link

@Anujay-Saraf Anujay-Saraf commented Jan 28, 2020

I am getting below error with kfctl which using it minikube version: v0.27.0 :-

INFO[0000] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.1.yaml to /var/folders/s8/58_yq2816tdbbnxf8rh1n4nd50n1bt/T/880895510/tmp.yaml filename="utils/k8utils.go:169"
INFO[0001] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.1.yaml to /var/folders/s8/58_yq2816tdbbnxf8rh1n4nd50n1bt/T/173311459/tmp_app.yaml filename="configconverters/converters.go:71"
INFO[0002] App directory /Users/anusaraf/kfflow/kf-test already exists filename="coordinator/coordinator.go:274"
INFO[0002] Writing KfDef to kfctl_k8s_istio.0.7.1.yaml filename="coordinator/coordinator.go:277"
INFO[0002] No name specified in KfDef.Metadata.Name; defaulting to kf-test based on location of config file: /Users/anusaraf/kfflow/kf-test/kfctl_k8s_istio.0.7.1.yaml. filename="coordinator/coordinator.go:202"
INFO[0002]


Notice anonymous usage reporting enabled using spartakus
To disable it
If you have already deployed it run the following commands:
cd $(pwd)
kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus

For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/


filename="coordinator/coordinator.go:120"
INFO[0002] Creating directory /Users/anusaraf/kfflow/kf-test/.cache filename="kfconfig/types.go:437"
INFO[0002] Fetching https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz to /Users/anusaraf/kfflow/kf-test/.cache/manifests filename="kfconfig/types.go:485"
Error: failed to build kfApp from URI https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.1.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /var/folders/s8/58_yq2816tdbbnxf8rh1n4nd50n1bt/T/getter410360294/archive: EOF
Usage:
kfctl build [flags]

Flags:
-f, --file string Static config file to use. Can be either a local path:
export CONFIG=./kfctl_gcp_iap.yaml
or a URL:
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_existing_arrikto.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.0.yaml
kfctl build -V --file=${CONFIG}
-h, --help help for build
-V, --verbose verbose output default is false

failed to build kfApp from URI https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.1.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /var/folders/s8/58_yq2816tdbbnxf8rh1n4nd50n1bt/T/getter410360294/archive: EOF

@fedeesku1

This comment has been minimized.

Copy link

@fedeesku1 fedeesku1 commented Jan 29, 2020

Latest release should include the fix:
https://github.com/kubeflow/kfctl/releases/tag/v1.0-rc.1

And the issue that caused it:
kubeflow/kfctl#167

I might be missing something here. This fix works for me so I thank you very much!!
However, the documentation for Kubeflow Deployment with kfctl_k8s_istio still links to the old binary https://github.com/kubeflow/kubeflow/releases/tag/v0.7.1 that does not work instead of linking to the one you suggested https://github.com/kubeflow/kfctl/releases/tag/v1.0-rc.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.