Unable to access kubernetes cluster launched on ap-east-1

[qianhaoh]

Kops version: 1.12
Kubectl version: v1.14.1

Command run:
Copy over working image from ap-northeast-1
aws ec2 copy-image --no-encrypted --name k8ns --source-image-id ami-001af84acd04e5ec5 --source-region ap-northeast-1 --region ap-east-1

create cluster using image copied
kops create cluster --master-size=m5.large --master-count=1 --node-count=2 --node-size=c5.large --name=${KOPS_CLUSTER_NAME} --vpc=vpc-xxxxxxx --cloud=aws --zones=ap-east-1a,ap-east-1b,ap-east-1c --image k8ns

last 200 lines from master log
May 16 10:17:26 ip-10-50-32-85 kubelet[2569]: E0516 10:17:26.929662 2569 kubelet.go:2236] node "ip-10-50-32-85.ap-east-1.compute.internal" not found May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.029829 2569 kubelet.go:2236] node "ip-10-50-32-85.ap-east-1.compute.internal" not found May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.129925 2569 kubelet.go:2236] node "ip-10-50-32-85.ap-east-1.compute.internal" not found May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.230024 2569 kubelet.go:2236] node "ip-10-50-32-85.ap-east-1.compute.internal" not found May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.330115 2569 kubelet.go:2236] node "ip-10-50-32-85.ap-east-1.compute.internal" not found May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.430207 2569 kubelet.go:2236] node "ip-10-50-32-85.ap-east-1.compute.internal" not found May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.530299 2569 kubelet.go:2236] node "ip-10-50-32-85.ap-east-1.compute.internal" not found May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.581638 2569 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://127.0.0.1/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.583317 2569 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://127.0.0.1/api/v1/pods?fieldSelector=spec.nodeName%3Dip-10-50-32-85.ap-east-1.compute.internal&limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.584703 2569 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://127.0.0.1/api/v1/nodes?fieldSelector=metadata.name%3Dip-10-50-32-85.ap-east-1.compute.internal&limit=500&resourceVersion=0: dial tcp 127.0.0.1:443: connect: connection refused May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.630442 2569 kubelet.go:2236] node "ip-10-50-32-85.ap-east-1.compute.internal" not found May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.730553 2569 kubelet.go:2236] node "ip-10-50-32-85.ap-east-1.compute.internal" not found May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.830651 2569 kubelet.go:2236] node "ip-10-50-32-85.ap-east-1.compute.internal" not found May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: I0516 10:17:27.852175 2569 kubelet_node_status.go:277] Setting node annotation to enable volume controller attach/detach May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: I0516 10:17:27.852194 2569 kubelet_node_status.go:325] Adding node label from cloud provider: beta.kubernetes.io/instance-type=m5.large May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: I0516 10:17:27.852201 2569 kubelet_node_status.go:336] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=ap-east-1a May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: I0516 10:17:27.852207 2569 kubelet_node_status.go:340] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=ap-east-1 May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: W0516 10:17:27.852221 2569 setters.go:144] replacing cloudprovider-reported hostname of ip-10-50-32-85.ap-east-1.compute.internal with overridden hostname of ip-10-50-32-85.ap-east-1.compute.internal May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: I0516 10:17:27.853781 2569 kubelet_node_status.go:430] Recording NodeHasSufficientDisk event message for node ip-10-50-32-85.ap-east-1.compute.internal May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: I0516 10:17:27.853801 2569 kubelet_node_status.go:430] Recording NodeHasSufficientMemory event message for node ip-10-50-32-85.ap-east-1.compute.internal May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: I0516 10:17:27.853810 2569 kubelet_node_status.go:430] Recording NodeHasNoDiskPressure event message for node ip-10-50-32-85.ap-east-1.compute.internal May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: I0516 10:17:27.853820 2569 kubelet_node_status.go:430] Recording NodeHasSufficientPID event message for node ip-10-50-32-85.ap-east-1.compute.internal May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.854328 2569 pod_workers.go:186] Error syncing pod 2d392d8dcd23094ebb7e69b689f2ec19 ("kube-apiserver-ip-10-50-32-85.ap-east-1.compute.internal_kube-system(2d392d8dcd23094ebb7e69b689f2ec19)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-ip-10-50-32-85.ap-east-1.compute.internal_kube-system(2d392d8dcd23094ebb7e69b689f2ec19)" May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.880584 2569 event.go:212] Unable to write event: 'Patch https://127.0.0.1/api/v1/namespaces/default/events/ip-10-50-32-85.ap-east-1.compute.internal.159f21a54f3196aa: dial tcp 127.0.0.1:443: connect: connection refused' (may retry after sleeping) May 16 10:17:27 ip-10-50-32-85 kubelet[2569]: E0516 10:17:27.930933 2569 kubelet.go:2236] node "ip-10-50-32-85.ap-east-1.compute.internal" not found

error log from /var/log/kube-apiserver.log:
F0516 10:16:35.559237 1 storage_decorator.go:57] Unable to create storage backend: config (&{etcd3 /registry [https://127.0.0.1:4001] /etc/kubernetes/pki/kube-apiserver/etcd-client.key /etc/kubernetes/pki/kube-apiserver/etcd-client.crt /etc/kubernetes/pki/kube-apiserver/etcd-ca.crt true true 1000 0xc420141560 <nil> 5m0s 1m0s}), err (dial tcp 127.0.0.1:4001: connect: connection refused)

[qianhaoh]

Tested with both images available in ap-northeast-1:
k8s-1.12-debian-stretch-amd64-hvm-ebs-2019-05-13
k8s-1.12-debian-stretch-amd64-hvm-ebs-2019-05-14

launching cluster in ap-northeast-1 is fine

[justinsb]

I think this is the same root cause as #6998 - will try to reproduce & fix!

[qianhaoh]

Verified working with v1.12.1 with the new patch. Thank you !