Can not deploy kubeflow on docker-for-kubernetes

[Jeffwan]

I following official documentation https://www.kubeflow.org/docs/started/getting-started/ to deploy v0.3.1 kubeflow on kubernetes on my MacOS. I use kubernetes in docker edge.

ksonnet version: 0.13.0
jsonnet version: v0.11.2
client-go version: kubernetes-1.10.4

➜  ${KUBEFLOW_SRC}/scripts/kfctl.sh apply k8s
+ COMMAND=apply
+ WHAT=k8s
+ ENV_FILE=env.sh
+ SKIP_INIT_PROJECT=false
+ MIN_CLUSTER_VERSION=1.10.7-gke.2
+++ dirname /tmp/kubeflow_src/scripts/kfctl.sh
++ cd /tmp/kubeflow_src/scripts
++ pwd
+ DIR=/tmp/kubeflow_src/scripts
+ source /tmp/kubeflow_src/scripts/util.sh
+ source /tmp/kubeflow_src/scripts/gke/util.sh
++ set -xe
+ source /tmp/kubeflow_src/scripts/util-minikube.sh
++ RED='\033[0;31m'
++ GREEN='\033[0;32m'
++ YELLOW='\033[0;33m'
++ NC='\033[0m'
++ MOUNT_LOCAL=false
+ '[' apply == init ']'
+ source env.sh
++ PLATFORM=none
++ KUBEFLOW_REPO=/tmp/kubeflow_src
++ KUBEFLOW_VERSION=0.3.1
++ KUBEFLOW_KS_DIR=/tmp/kubeflow_src/eks_kf/ks_app
++ KUBEFLOW_DOCKER_REGISTRY=
++ K8S_NAMESPACE=kubeflow
++ KUBEFLOW_CLOUD=null
+ '[' -z apply ']'
+ '[' -z k8s ']'
+ check_install ks
+ which ks
+ check_install kubectl
+ which kubectl
+ source env.sh
++ PLATFORM=none
++ KUBEFLOW_REPO=/tmp/kubeflow_src
++ KUBEFLOW_VERSION=0.3.1
++ KUBEFLOW_KS_DIR=/tmp/kubeflow_src/eks_kf/ks_app
++ KUBEFLOW_DOCKER_REGISTRY=
++ K8S_NAMESPACE=kubeflow
++ KUBEFLOW_CLOUD=null
+ '[' apply == generate ']'
+ '[' apply == apply ']'
+ '[' k8s == platform ']'
+ '[' k8s == all ']'
+ '[' k8s == k8s ']'
+ createNamespace
+ set +e
++ kubectl get namespace kubeflow
+ O='Error from server (NotFound): namespaces "kubeflow" not found'
+ RESULT=1
+ set -e
+ '[' 1 -eq 0 ']'
+ kubectl create namespace kubeflow
namespace "kubeflow" created
+ ksApply
+ pushd /tmp/kubeflow_src/eks_kf/ks_app
/tmp/kubeflow_src/eks_kf/ks_app /tmp/kubeflow_src/eks_kf
+ '[' none == minikube ']'
+ set +e
++ ks env describe default
+ O='level=error msg="environment \"default\" was not found"'
+ RESULT=1
+ set -e
+ '[' 1 -eq 0 ']'
+ ks env add default --namespace kubeflow
INFO Using context "docker-for-desktop" from kubeconfig file "/Users/jiashan/.kube/config"
INFO Creating environment "default" with namespace "kubeflow", pointing to "version:v1.10.3" cluster at address "https://localhost:6443"
+ ks apply default -c ambassador
INFO Applying services kubeflow.ambassador
INFO Creating non-existent services kubeflow.ambassador
INFO Applying services kubeflow.statsd-sink
INFO Creating non-existent services kubeflow.statsd-sink
INFO Applying services kubeflow.ambassador-admin
INFO Creating non-existent services kubeflow.ambassador-admin
INFO Applying roles kubeflow.ambassador
INFO Creating non-existent roles kubeflow.ambassador
INFO Applying serviceaccounts kubeflow.ambassador
INFO Creating non-existent serviceaccounts kubeflow.ambassador
INFO Applying rolebindings kubeflow.ambassador
INFO Creating non-existent rolebindings kubeflow.ambassador
INFO Applying services kubeflow.k8s-dashboard
INFO Creating non-existent services kubeflow.k8s-dashboard
INFO Applying deployments kubeflow.ambassador
INFO Creating non-existent deployments kubeflow.ambassador
+ ks apply default -c jupyterhub
INFO Applying configmaps kubeflow.jupyterhub-config
INFO Creating non-existent configmaps kubeflow.jupyterhub-config
INFO Applying services kubeflow.tf-hub-0
INFO Creating non-existent services kubeflow.tf-hub-0
INFO Applying rolebindings kubeflow.jupyter-role
INFO Creating non-existent rolebindings kubeflow.jupyter-role
INFO Applying roles kubeflow.jupyter-role
INFO Creating non-existent roles kubeflow.jupyter-role
INFO Applying roles kubeflow.jupyter-notebook-role
INFO Creating non-existent roles kubeflow.jupyter-notebook-role
INFO Applying services kubeflow.tf-hub-lb
INFO Creating non-existent services kubeflow.tf-hub-lb
INFO Applying serviceaccounts kubeflow.jupyter-hub
INFO Creating non-existent serviceaccounts kubeflow.jupyter-hub
INFO Applying serviceaccounts kubeflow.jupyter-notebook
INFO Creating non-existent serviceaccounts kubeflow.jupyter-notebook
INFO Applying rolebindings kubeflow.jupyter-notebook-role
INFO Creating non-existent rolebindings kubeflow.jupyter-notebook-role
INFO Applying statefulsets kubeflow.tf-hub
INFO Creating non-existent statefulsets kubeflow.tf-hub
+ ks apply default -c centraldashboard
INFO Applying services kubeflow.centraldashboard
INFO Creating non-existent services kubeflow.centraldashboard
INFO Applying serviceaccounts kubeflow.centraldashboard
INFO Creating non-existent serviceaccounts kubeflow.centraldashboard
INFO Applying clusterroles kubeflow.centraldashboard
INFO Creating non-existent clusterroles kubeflow.centraldashboard
INFO Applying deployments kubeflow.centraldashboard
INFO Creating non-existent deployments kubeflow.centraldashboard
+ ks apply default -c tf-job-operator
INFO Applying customresourcedefinitions tfjobs.kubeflow.org
INFO Creating non-existent customresourcedefinitions tfjobs.kubeflow.org
INFO Applying serviceaccounts kubeflow.tf-job-dashboard
INFO Creating non-existent serviceaccounts kubeflow.tf-job-dashboard
INFO Applying configmaps kubeflow.tf-job-operator-config
INFO Creating non-existent configmaps kubeflow.tf-job-operator-config
INFO Applying serviceaccounts kubeflow.tf-job-operator
INFO Creating non-existent serviceaccounts kubeflow.tf-job-operator
INFO Applying clusterroles tf-job-operator
INFO Creating non-existent clusterroles tf-job-operator
INFO Applying clusterrolebindings tf-job-operator
INFO Creating non-existent clusterrolebindings tf-job-operator
INFO Applying services kubeflow.tf-job-dashboard
INFO Creating non-existent services kubeflow.tf-job-dashboard
INFO Applying clusterroles tf-job-dashboard
INFO Creating non-existent clusterroles tf-job-dashboard
INFO Applying clusterrolebindings tf-job-dashboard
INFO Creating non-existent clusterrolebindings tf-job-dashboard
INFO Applying deployments kubeflow.tf-job-operator-v1alpha2
INFO Creating non-existent deployments kubeflow.tf-job-operator-v1alpha2
INFO Applying deployments kubeflow.tf-job-dashboard
INFO Creating non-existent deployments kubeflow.tf-job-dashboard
+ ks apply default -c argo
ERROR handle object: patching object from cluster: merging object with existing state: Get https://localhost:6443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/workflows.argoproj.io: EOF

After that, get messages

➜  kubectl get pods
Unable to connect to the server: EOF

[Jeffwan]

BTW, I tried v0.2.5 and it's working fine

$ export KUBEFLOW_VERSION=0.2.5
$ curl https://raw.githubusercontent.com/kubeflow/kubeflow/v${KUBEFLOW_VERSION}/scripts/deploy.sh | bash

[jlewi]

➜  kubectl get pods
Unable to connect to the server: EOF

It looks like kubectl is unable to connect to your cluster. It looks like your kubernetes might be unhealthy.

[Jeffwan]

Yes. Once I install 0.3.x cluster become unavailable due to some reason. I can not use kubectl logs to check pods logs. Looks like I need to go to logs for detail reason. Will come back later.

[jlewi]

Thanks @Jeffwan. I recall seeing some issue where Ambassador ended up causing the DNS server to crash. You might want to try taking down Ambassador or some of the other Kubeflow services to see if that fixes the problem.

[Jeffwan]

Double checked it's true. I manually install kubeflow core and kernel components I am using rather than whole package. This can be resolved.