Deploy kfctl.sh apply k8s : Service "ambassador" is invalid ?

[yylin1]

I am trying to execute the deployment kubeflow script in my kubernetes GPU Cluster, but an error occurred in the last step.

set up and deploy Kubeflow

${KUBEFLOW_REPO}/scripts/kfctl.sh init ${KFAPP} --platform none
cd ${KFAPP}
${KUBEFLOW_REPO}/scripts/kfctl.sh generate k8s
${KUBEFLOW_REPO}/scripts/kfctl.sh apply k8s

Previously in my Cluster environment, I have previously implemented kubeflow v0.2.2. Is it likely to be affected by the environment?

Problem :

$ {KUBEFLOW_REPO}/scripts/kfctl.sh apply k8s
·
·
kubernetesversion: v1.11.2
path: default
destination:
  server: https://1xx.1xx.xx.xx:8443
  namespace: kubeflow
targets: []
libraries: {}'
+ RESULT=0
+ set -e
+ '[' 0 -eq 0 ']'
+ echo environment default already exists
environment default already exists
+ ks apply default -c ambassador
ERROR handle object: patching object from cluster: merging object with existing state: Service "ambassador" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'

[yylin1]

I try kubeflow v0.2.5 Deploy

Quick start

export KUBEFLOW_VERSION=0.2.5
curl https://raw.githubusercontent.com/kubeflow/kubeflow/v${KUBEFLOW_VERSION}/scripts/deploy.sh | bash

Error message

What's problem in my environment?

+ ks init kubeflow_ks_app
INFO Using context "kubernetes-admin@kubernetes" from kubeconfig file "/Users/yylin/.kube/config"
INFO Creating environment "default" with namespace "yylin", pointing to "version:v1.11.2" cluster at address "https://140.128.18.98:8443"
INFO Generating ksonnet-lib data at path '/Users/yylin/kubeflow/ha-kubeflow/kubeflow_ks_app/lib/ksonnet-lib/v1.11.2'
+ cd /Users/yylin/kubeflow/ha-kubeflow/kubeflow_ks_app
+ ks registry add kubeflow my-kubeflow/kubeflow
ERROR initializing helm HTTP client: normalizing Helm repository URL: "" is an invalid scheme for Helm repository

[jlewi]

Which version of ksonnet are you using?

ERROR initializing helm HTTP client: normalizing Helm repository URL: "" is an invalid scheme for Helm repository

Kubeflow doesn't use Helm so I'm not sure where this is coming from. The only thing I can think of is that this is a ksonnet issue because its trying and failing to setup its support for using helm.

Can you try ksonnet 0.11 if you aren't already using that?

[yylin1]

Hi @jlewi
my environment list :

$ ks version                                                                                 
ksonnet version: 0.12.0-rc1
jsonnet version: v0.10.0
client-go version:

First Proble : master

Deploy I am try KUBEFLOW_TAG=master

ERROR handle object: patching object from cluster: merging object with existing state: Service "ambassador" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'

Second Problem : v0.2

---

I try to use kubeflow Deploy v0.2 script

ERROR initializing helm HTTP client: normalizing Helm repository URL: "" is an invalid scheme for Helm repository

ksonnet version: 0.12.0-rc1

[yylin1]

I try again, Deploy kubeflow v0.2.2 is OK.

export KUBEFLOW_VERSION=0.2.2
curl https://raw.githubusercontent.com/kubeflow/kubeflow/v${KUBEFLOW_VERSION}/scripts/deploy.sh | bash

kubeflow v0.2.5 have Helm repository problem?

[jlewi]

Any particular reason you are using 0.12-rc1? I think there is a 0.12 version.

[yylin1]

@jlewi
It is recommended to use v0.11 or higher in kubeflow = master
try to trun to v0.12 version ?

[jlewi]

Another user is reporting hitting this on slack.
User reports

ks registry list

returns no registries.

So maybe the issue here is that the ks registry add command didn't succeed and that causes problems.

Can you see if ks registry list returns any registries; if not can you try adding one?

[yylin1]

Sorry @jlewi reply late for you,

ks version has not changed v0.12-rc1
I don't know why, try again and Deploy successfully in these days.
My GPU Cluster & minikube try is okay. �

script

export KUBEFLOW_VERSION=0.2.5
curl https://raw.githubusercontent.com/kubeflow/kubeflow/v${KUBEFLOW_VERSION}/scripts/deploy.sh | bash

info.

$ ks registry list
NAME      OVERRIDE PROTOCOL URI
====      ======== ======== ===
incubator          github   github.com/ksonnet/parts/tree/master/incubator
kubeflow           fs       /Users/yylin/kubeflow/ha-kubeflow/kubeflow_repo/kubeflow

Convenient for users script kubeflow master & v0.2 which suggest deploy now ?

[surabhikannan]

still facing this issue in
ks version
ksonnet version: 0.13.0
jsonnet version: v0.11.2
client-go version: kubernetes-1.10.4

kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0", GitTreeState:"clean", BuildDate:"2018-09-27T17:05:32Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"", Minor:"", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2018-01-26T19:04:38Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}

kubeflow version v0.3.0

[nikan1996]

I get the same problem
ks version
ksonnet version: 0.13.0
jsonnet version: v0.11.2
client-go version: kubernetes-1.10.4

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
kubeflow version
v0.3.0

[jomach]

I'm getting the same error and I even don't have ks installed.

[jomach]

user@server1:~/kubeflow$ kfctl generate all -V
INFO[0000] Ksonnet.Generate Name AppDir Platform filename="ksonnet/ksonnet.go:369"
INFO[0000] Creating environment "default" with namespace "default", pointing to "version:v1.14.1" cluster at address "https://127.0.0.1:16443" filename="env/create.go:77"
INFO[0002] Generating ksonnet-lib data at path 'ks_app/lib/ksonnet-lib/v1.14.1' filename="lib/lib.go:148"
INFO[0002] Successfully initialized the app . filename="ksonnet/ksonnet.go:505"
INFO[0002] App add registry kubeflow URI filename="ksonnet/ksonnet.go:621"
Error: couldn't generate KfApp: (kubeflow.error): Code 500 with message: kfApp Generate failed for ksonnet: (kubeflow.error): Code 400 with message: couldn't add registry kubeflow. Error: there was a problem adding registry kubeflow: initializing helm HTTP client: normalizing Helm repository URL: "" is an invalid scheme for Helm repository
Usage:
kfctl generate [all(=default)|k8s|platform] [flags]

Flags:
--email string email if '--platform gcp'
-h, --help help for generate
--hostname string hostname if '--platform gcp'
--ipName string ipName if '--platform gcp'
--mount-local mount-local if '--platform minikube'
-V, --verbose verbose output default is false
--zone string zone if '--platform gcp'

environment:
ubuntu 18.04
snap minikube
KUBEFLOW_TAG=v0.5.1

[solovyevt]

Double that, experiencing the same issue.

` kfctl generate k8s -V
INFO[0000] Ksonnet.Generate Name AppDir Platform filename="ksonnet/ksonnet.go:369"
INFO[0000] Creating environment "default" with namespace "default", pointing to "version:v1.13.5" cluster at address "https://10.46.166.93/k8s/clusters/c-2jxdd" filename="env/create.go:77"
INFO[0003] Generating ksonnet-lib data at path 'ks_app/lib/ksonnet-lib/v1.13.5' filename="lib/lib.go:148"
INFO[0003] Successfully initialized the app . filename="ksonnet/ksonnet.go:505"
INFO[0003] App add registry kubeflow URI filename="ksonnet/ksonnet.go:621"
Error: couldn't generate KfApp: (kubeflow.error): Code 500 with message: kfApp Generate failed for ksonnet: (kubeflow.error): Code 400 with message: couldn't add registry kubeflow. Error: there was a problem adding registry kubeflow: initializing helm HTTP client: normalizing Helm repository URL: "" is an invalid scheme for Helm repository
Usage:
kfctl generate [all(=default)|k8s|platform] [flags]

Flags:
--email string email if '--platform gcp'
-h, --help help for generate
--hostname string hostname if '--platform gcp'
--ipName string ipName if '--platform gcp'
--mount-local mount-local if '--platform minikube'
-V, --verbose verbose output default is false
--zone string zone if '--platform gcp'

couldn't generate KfApp: (kubeflow.error): Code 500 with message: kfApp Generate failed for ksonnet: (kubeflow.error): Code 400 with message: couldn't add registry kubeflow. Error: there was a problem adding registry kubeflow: initializing helm HTTP client: normalizing Helm repository URL: "" is an invalid scheme for Helm repository`

---

UPD: Disregard that, the problem occured when I've tried running kfctl generate k8s -V in an empty folder, seems like the folder wasn't initialized correctly after running kfctl init $KFAPP. Probably that will help someone who's experiencing the same issue.

[yylin1]

@jomach
Check the environment ksonnect version. And clean old kubeflow data, restart deploy kubeflow.

@solovyevt
Try clean and remove kubeflow, restart kubeflow.

I'm try minikube is no problem.