Adding a second node fails

[tronar]

Minikube v0.35.0
on Ubuntu 18.04.2 VM with Docker 18.09.3 and driver=none
initial hostname panda, 172.30.0.10
second hostname compute1, 172.30.0.61

Following
https://stackoverflow.com/questions/51687893/how-do-i-get-the-minikube-nodes-in-a-local-cluster/51706547#51706547
I tried to join a second node to the (working) one node cluster by doing a "kubeadm token create --print-join-command" on the original host (VM) and performing the join from a second host (VM).
The join starts ok but then tries to talk to localhost:8443 wich fails:

root@compute1:~# kubeadm join 172.30.0.10:8443 --token izh42n.z5cwj0nrlbykljqt --discovery-token-ca-cert-hash sha256:f239f6044f388af59e4b0063a9aca2006b3bc682fde3edf9ae088768d75104b1 --ignore-preflight-errors=all
[preflight] Running pre-flight checks
        [WARNING Swap]: running with swap on is not supported. Please disable swap
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.3. Latest validated version: 18.06
[discovery] Trying to connect to API Server "172.30.0.10:8443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.30.0.10:8443"
[discovery] Requesting info from "https://172.30.0.10:8443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.30.0.10:8443"
[discovery] Successfully established connection with API Server "172.30.0.10:8443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
unable to fetch the kubeadm-config ConfigMap: failed to get config map: Get https://localhost:8443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: dial tcp [::1]:8443: connect: connection refused

I don't fully understand what is the magic that minikube does on top of Kubernetes kubeadm, so this might be a no-go option, but the join trying to talk to localhost seems like a default not being updated in the cluster config ?
FTR:
root@panda:~# kubectl -n kube-system get cm kubeadm-config -oyaml apiVersion: v1 data: ClusterConfiguration: | apiServer: extraArgs: authorization-mode: Node,RBAC enable-admission-plugins: Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta1 certificatesDir: /var/lib/minikube/certs/ clusterName: kubernetes controlPlaneEndpoint: panda:8443 controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /data/minikube imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: v1.13.4 networking: dnsDomain: cluster.local podSubnet: "" serviceSubnet: 10.96.0.0/12 scheduler: {} ClusterStatus: | apiEndpoints: minikube: advertiseAddress: 172.30.0.10 bindPort: 8443 apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterStatus kind: ConfigMap metadata: creationTimestamp: 2019-03-12T22:29:10Z name: kubeadm-config namespace: kube-system resourceVersion: "167833" selfLink: /api/v1/namespaces/kube-system/configmaps/kubeadm-config uid: 41a89226-4516-11e9-a090-000c29c34fa1

[afbjorklund]

Hmm, this is supposed to work. But probably not tested enough, since the default setup only has one node. Like you say, that "localhost" should probably have been the host name.

kubeadm does:

		// gets the APIEndpoint for the current node from then ClusterStatus in the kubeadm-config ConfigMap
		if err := getAPIEndpoint(configMap.Data, initcfg.NodeRegistration.Name, &initcfg.LocalAPIEndpoint); err != nil {
			return nil, errors.Wrap(err, "failed to getAPIEndpoint")
		}

minikube does:

https://github.com/kubernetes/minikube/blob/master/pkg/minikube/bootstrapper/kubeadm/templates.go#L126

controlPlaneEndpoint: localhost:{{.APIServerPort}}

[tronar]

I am in the process of getting know about what's going on in K8s, at a more or less detailed way. But not there yet. Is there anything I can do to help ?
The cluster config lists the apiEndpoint as having the correct address (172.30.10:8443) and kubeadm is indeed talking to it at first, I do not understand why it switches to localhost then.
As some say in radio, QRV & QAP (ready and standing by)

[afbjorklund]

afaict, the issue is that it is reading a kubeadm config file where we have hardcoded "localhost"

[tronar]

Aha, like that line (126) should be
controlPlaneEndpoint: {{.APIHost}}:{{.APIServerPort}}
or something like that ?

[tronar]

Just FYI, I've temporarily created a DNAT rule to send 127.1:8443 to the master node and it seems to hold.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tstromberg]

Closing as it sounds like a workaround was found.

BTW, this is crazy and I love it. I look forward to proper multi-node support.