[ question ] when I follow the setup tutorial, I got cannot parse dnsName problem

[gogeof]

label

/label question

what I done

• 1. start a k8s cluster and install calico network plugin
• 2. follow the kubeflow setup tutorial to start kubeflow

# Initialize a ksonnet APP
APP_NAME=my-kubeflow
ks init ${APP_NAME}
cd ${APP_NAME}

# Install Kubeflow components
ks registry add kubeflow github.com/kubeflow/kubeflow/tree/master/kubeflow
ks pkg install kubeflow/core
ks pkg install kubeflow/tf-serving
ks pkg install kubeflow/tf-job

# Create templates for core components
NAMESPACE=kubeflow
kubectl create namespace ${NAMESPACE}
ks generate core kubeflow-core --name=kubeflow-core --namespace=${NAMESPACE}

# Enable collection of anonymous usage metrics
# Skip this step if you don't want to enable collection.
ks param set kubeflow-core reportUsage true
ks param set kubeflow-core usageId $(uuidgen)

# Deploy Kubeflow
ks apply default -c kubeflow-core

what I got:

++ APP_NAME=my-kubeflow
++ ks init my-kubeflow
INFO  Using context 'kubernetes-admin@kubernetes' from the kubeconfig file specified at the environment variable $KUBECONFIG
INFO  Creating environment "default" with namespace "default", pointing to cluster at address "https://192.168.4.107:6443"
INFO  Generating ksonnet-lib data at path '/root/my-kubeflow/my-kubeflow/lib/v1.7.0'
INFO  ksonnet app successfully created! Next, try creating a component with `ks generate`.
++ cd my-kubeflow
++ ks registry add kubeflow github.com/kubeflow/kubeflow/tree/master/kubeflow
++ ks pkg install kubeflow/core
INFO  Retrieved 15 files
++ ks pkg install kubeflow/tf-serving
INFO  Retrieved 4 files
++ ks pkg install kubeflow/tf-job
INFO  Retrieved 5 files
++ NAMESPACE=kubeflow
++ kubectl create namespace kubeflow
Error from server (AlreadyExists): namespaces "kubeflow" already exists
++ ks generate core kubeflow-core --name=kubeflow-core --namespace=kubeflow
INFO  Writing component at '/root/my-kubeflow/my-kubeflow/components/kubeflow-core.jsonnet'
++ ks param set kubeflow-core reportUsage true
INFO  Parameter 'reportUsage' successfully set to '"true"' for component 'kubeflow-core'
+++ uuidgen
++ ks param set kubeflow-core usageId f462151b-75fc-45d5-88b5-d6b9361bacb5
INFO  Parameter 'usageId' successfully set to '"f462151b-75fc-45d5-88b5-d6b9361bacb5"' for component 'kubeflow-core'
++ ks apply default -c kubeflow-core
INFO  Updating configmap kubeflow.jupyterhub-config
ERROR Get https://192.168.4.107:6443/api/v1: tls: failed to parse certificate from server: x509: cannot parse dnsName "kubernetes.default.svc."

what's the problem

ERROR Get https://192.168.4.107:6443/api/v1: tls: failed to parse certificate from server: x509: cannot parse dnsName "kubernetes.default.svc."

what I expect

no any error message

any helper is appreciate!

[pdmack]

Not exactly sure but looks like a kube DNS issue. Could look at the DNS logs for errors.

for ctr in kubedns dnsmasq sidecar; do echo -e "\n$ctr"; kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c $ctr; done

[gogeof]

@pdmack thanks for response.
You're right, kube-dns sidecar failed to start.

# for ctr in kubedns dnsmasq sidecar; do echo -e "\n$ctr"; kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c $ctr; done

kubedns
I0309 00:24:13.043978       1 dns.go:48] version: 1.14.4-2-g5584e04
I0309 00:24:13.051456       1 server.go:70] Using configuration read from directory: /kube-dns-config with period 10s
I0309 00:24:13.051795       1 server.go:113] FLAG: --alsologtostderr="false"
I0309 00:24:13.051824       1 server.go:113] FLAG: --config-dir="/kube-dns-config"
I0309 00:24:13.051834       1 server.go:113] FLAG: --config-map=""
I0309 00:24:13.051840       1 server.go:113] FLAG: --config-map-namespace="kube-system"
I0309 00:24:13.051847       1 server.go:113] FLAG: --config-period="10s"
I0309 00:24:13.051865       1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I0309 00:24:13.051872       1 server.go:113] FLAG: --dns-port="10053"
I0309 00:24:13.051881       1 server.go:113] FLAG: --domain="cluster.local."
I0309 00:24:13.051893       1 server.go:113] FLAG: --federations=""
I0309 00:24:13.051902       1 server.go:113] FLAG: --healthz-port="8081"
I0309 00:24:13.051908       1 server.go:113] FLAG: --initial-sync-timeout="1m0s"
I0309 00:24:13.051972       1 server.go:113] FLAG: --kube-master-url=""
I0309 00:24:13.052002       1 server.go:113] FLAG: --kubecfg-file=""
I0309 00:24:13.052009       1 server.go:113] FLAG: --log-backtrace-at=":0"
I0309 00:24:13.052021       1 server.go:113] FLAG: --log-dir=""
I0309 00:24:13.052028       1 server.go:113] FLAG: --log-flush-frequency="5s"
I0309 00:24:13.052035       1 server.go:113] FLAG: --logtostderr="true"
I0309 00:24:13.052041       1 server.go:113] FLAG: --nameservers=""
I0309 00:24:13.052046       1 server.go:113] FLAG: --stderrthreshold="2"
I0309 00:24:13.052052       1 server.go:113] FLAG: --v="2"
I0309 00:24:13.052208       1 server.go:113] FLAG: --version="false"
I0309 00:24:13.052226       1 server.go:113] FLAG: --vmodule=""
I0309 00:24:13.052614       1 server.go:176] Starting SkyDNS server (0.0.0.0:10053)
I0309 00:24:13.053033       1 server.go:198] Skydns metrics enabled (/metrics:10055)
I0309 00:24:13.053050       1 dns.go:147] Starting endpointsController
I0309 00:24:13.053058       1 dns.go:150] Starting serviceController
I0309 00:24:13.055675       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0309 00:24:13.055827       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0309 00:24:13.562551       1 dns.go:171] Initialized services and endpoints from apiserver
I0309 00:24:13.562573       1 server.go:129] Setting up Healthz Handler (/readiness)
I0309 00:24:13.562584       1 server.go:134] Setting up cache handler (/cache)
I0309 00:24:13.562592       1 server.go:120] Status HTTP port 8081

dnsmasq
I0309 00:24:13.290593       1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}
I0309 00:24:13.291080       1 nanny.go:86] Starting dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]
I0309 00:24:13.788460       1 nanny.go:111] 
W0309 00:24:13.788490       1 nanny.go:112] Got EOF from stdout
I0309 00:24:13.791405       1 nanny.go:108] dnsmasq[10]: started, version 2.78-security-prerelease cachesize 1000
I0309 00:24:13.791592       1 nanny.go:108] dnsmasq[10]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
I0309 00:24:13.791860       1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain ip6.arpa 
I0309 00:24:13.792002       1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa 
I0309 00:24:13.792134       1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain cluster.local 
I0309 00:24:13.792255       1 nanny.go:108] dnsmasq[10]: reading /etc/resolv.conf
I0309 00:24:13.792350       1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain ip6.arpa 
I0309 00:24:13.792444       1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa 
I0309 00:24:13.792456       1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain cluster.local 
I0309 00:24:13.792463       1 nanny.go:108] dnsmasq[10]: using nameserver 8.8.8.8#53
I0309 00:24:13.792469       1 nanny.go:108] dnsmasq[10]: read /etc/hosts - 7 addresses

sidecar
ERROR: logging before flag.Parse: I0309 00:24:13.547701       1 main.go:48] Version v1.14.4-2-g5584e04
ERROR: logging before flag.Parse: I0309 00:24:13.548317       1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})
ERROR: logging before flag.Parse: I0309 00:24:13.548544       1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
ERROR: logging before flag.Parse: I0309 00:24:13.548814       1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
ERROR: logging before flag.Parse: W0309 00:24:13.552913       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:33118->127.0.0.1:53: read: connection refused

[jlewi]

I'm closing this because it looks like the problem was the kube DNS didn't start properly.