change the default AMI from CoreOS to Amazon Linux 2 LTS #47

pahud · 2019-01-11T02:09:37Z

Background

to make sure the OS is more compatible with other components such as

AWS VPC CNI
AWS ALB Ingress

And eliminate potential complexity of maintainence in the future.

TODO

make sure the latest AMI in cn-north-1 and cn-northwest-1 is compatible with the latest stable kops
make sure latest AWS VPC CNI (1.3) is compatible
Amazon Linux 2 default cluster has inter-pod connectivity issue kubernetes/kops#6341 need to be identified and fixed
CoreDNS as the replacement of kube-dns
make sure AWS ALB Ingress is compatible
update env.config and set Amazon Linux 2 LTS as the default AMI

The text was updated successfully, but these errors were encountered:

pahud · 2019-01-11T02:25:00Z

default cluster creation LGTM

pahud:~/environment/kops-cn (master) $ kops  validate cluster                                                                       
Using cluster from kubectl context: cluster.zhy.k8s.local

Validating cluster cluster.zhy.k8s.local

INSTANCE GROUPS
NAME                    ROLE    MACHINETYPE     MIN     MAX     SUBNETS
master-cn-northwest-1a  Master  t2.medium       1       1       cn-northwest-1a
master-cn-northwest-1b  Master  t2.medium       1       1       cn-northwest-1b
master-cn-northwest-1c  Master  t2.medium       1       1       cn-northwest-1c
nodes                   Node    t2.medium       2       2       cn-northwest-1a,cn-northwest-1b,cn-northwest-1c

NODE STATUS
NAME                                            ROLE    READY
ip-10-0-110-9.cn-northwest-1.compute.internal   master  True
ip-10-0-40-159.cn-northwest-1.compute.internal  master  True
ip-10-0-63-126.cn-northwest-1.compute.internal  node    True
ip-10-0-73-217.cn-northwest-1.compute.internal  master  True
ip-10-0-84-197.cn-northwest-1.compute.internal  node    True

Your cluster cluster.zhy.k8s.local is ready
pahud:~/environment/kops-cn (master) $ kubectl  get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
pahud:~/environment/kops-cn (master) $ kubectl  -n kube-system get po
NAME                                                                     READY     STATUS    RESTARTS   AGE
aws-node-5bbl8                                                           1/1       Running   0          1m
aws-node-dwhp2                                                           1/1       Running   0          1m
aws-node-fdz7k                                                           1/1       Running   0          1m
aws-node-lt6jl                                                           1/1       Running   0          43s
aws-node-r27t2                                                           1/1       Running   0          49s
coredns-6845b984cd-b5vhx                                                 1/1       Running   0          1m
coredns-6845b984cd-swb6c                                                 1/1       Running   0          1m
dns-controller-878d5577d-8hn9h                                           1/1       Running   0          1m
etcd-server-events-ip-10-0-110-9.cn-northwest-1.compute.internal         1/1       Running   0          1m
etcd-server-events-ip-10-0-40-159.cn-northwest-1.compute.internal        1/1       Running   0          37s
etcd-server-events-ip-10-0-73-217.cn-northwest-1.compute.internal        1/1       Running   0          28s
etcd-server-ip-10-0-110-9.cn-northwest-1.compute.internal                1/1       Running   0          1m
etcd-server-ip-10-0-40-159.cn-northwest-1.compute.internal               1/1       Running   0          52s
etcd-server-ip-10-0-73-217.cn-northwest-1.compute.internal               1/1       Running   0          42s
kube-apiserver-ip-10-0-110-9.cn-northwest-1.compute.internal             1/1       Running   1          1m
kube-apiserver-ip-10-0-40-159.cn-northwest-1.compute.internal            1/1       Running   0          54s
kube-apiserver-ip-10-0-73-217.cn-northwest-1.compute.internal            1/1       Running   0          44s
kube-controller-manager-ip-10-0-110-9.cn-northwest-1.compute.internal    1/1       Running   0          59s
kube-controller-manager-ip-10-0-40-159.cn-northwest-1.compute.internal   1/1       Running   0          37s
kube-controller-manager-ip-10-0-73-217.cn-northwest-1.compute.internal   1/1       Running   0          27s
kube-proxy-ip-10-0-110-9.cn-northwest-1.compute.internal                 1/1       Running   0          1m
kube-proxy-ip-10-0-40-159.cn-northwest-1.compute.internal                1/1       Running   0          54s
kube-proxy-ip-10-0-63-126.cn-northwest-1.compute.internal                1/1       Running   0          45s
kube-proxy-ip-10-0-73-217.cn-northwest-1.compute.internal                1/1       Running   0          37s
kube-proxy-ip-10-0-84-197.cn-northwest-1.compute.internal                1/1       Running   0          29s
kube-scheduler-ip-10-0-110-9.cn-northwest-1.compute.internal             1/1       Running   0          1m
kube-scheduler-ip-10-0-40-159.cn-northwest-1.compute.internal            1/1       Running   0          57s
kube-scheduler-ip-10-0-73-217.cn-northwest-1.compute.internal            1/1       Running   0          38s
pahud:~/environment/kops-cn (master) $

pahud · 2019-01-12T04:58:10Z

CoreDNS

pahud:~/environment/kops-cn (master) $ kubectl  -n kube-system describe rs/coredns-6845b984cd | grep Image
    Image:       937788672844.dkr.ecr.cn-north-1.amazonaws.com.cn/coredns:1.2.6

pahud · 2019-01-12T04:58:44Z

ds/aws-node

pahud:~/environment/kops-cn (master) $ kubectl  -n kube-system describe ds/aws-node | grep Image
    Image:      937788672844.dkr.ecr.cn-north-1.amazonaws.com.cn/602401143452.dkr.ecr.us-west-2.amazonaws.com-amazon-k8s-cni:1.3.0
pahud:~/environment/kops-cn (master) $

pahud · 2019-01-13T15:35:34Z

Amazon Linux 2 with ALB Ingress 1.0.1 is good

pahud:~/environment/kops-cn (master) $ kubectl  -n kube-system get po
NAME                                                                      READY     STATUS    RESTARTS   AGE
aws-node-cpl9n                                                            1/1       Running   0          7m
aws-node-d8x2l                                                            1/1       Running   0          7m
aws-node-n4w6c                                                            1/1       Running   0          7m
aws-node-xgp47                                                            1/1       Running   0          7m
aws-node-zzj8k                                                            1/1       Running   0          5m
coredns-6845b984cd-4gd2q                                                  1/1       Running   0          6m
coredns-6845b984cd-lnl9l                                                  1/1       Running   0          6m
dns-controller-878d5577d-267wq                                            1/1       Running   0          8m
etcd-server-events-ip-10-0-109-252.cn-northwest-1.compute.internal        1/1       Running   0          7m
etcd-server-events-ip-10-0-54-194.cn-northwest-1.compute.internal         1/1       Running   0          4m
etcd-server-events-ip-10-0-95-208.cn-northwest-1.compute.internal         1/1       Running   0          7m
etcd-server-ip-10-0-109-252.cn-northwest-1.compute.internal               1/1       Running   0          7m
etcd-server-ip-10-0-54-194.cn-northwest-1.compute.internal                1/1       Running   0          4m
etcd-server-ip-10-0-95-208.cn-northwest-1.compute.internal                1/1       Running   0          7m
kube-apiserver-ip-10-0-109-252.cn-northwest-1.compute.internal            1/1       Running   0          7m
kube-apiserver-ip-10-0-54-194.cn-northwest-1.compute.internal             1/1       Running   0          4m
kube-apiserver-ip-10-0-95-208.cn-northwest-1.compute.internal             1/1       Running   0          7m
kube-controller-manager-ip-10-0-109-252.cn-northwest-1.compute.internal   1/1       Running   0          7m
kube-controller-manager-ip-10-0-54-194.cn-northwest-1.compute.internal    1/1       Running   0          4m
kube-controller-manager-ip-10-0-95-208.cn-northwest-1.compute.internal    1/1       Running   0          7m
kube-proxy-ip-10-0-109-252.cn-northwest-1.compute.internal                1/1       Running   0          7m
kube-proxy-ip-10-0-50-9.cn-northwest-1.compute.internal                   1/1       Running   0          6m
kube-proxy-ip-10-0-54-194.cn-northwest-1.compute.internal                 1/1       Running   0          4m
kube-proxy-ip-10-0-81-194.cn-northwest-1.compute.internal                 1/1       Running   0          6m
kube-proxy-ip-10-0-95-208.cn-northwest-1.compute.internal                 1/1       Running   0          7m
kube-scheduler-ip-10-0-109-252.cn-northwest-1.compute.internal            1/1       Running   0          7m
kube-scheduler-ip-10-0-54-194.cn-northwest-1.compute.internal             1/1       Running   0          4m
kube-scheduler-ip-10-0-95-208.cn-northwest-1.compute.internal             1/1       Running   0          7m
pahud:~/environment/kops-cn (master) $ aws --profile bjs iam put-role-policy --role-name nodes.cluster.zhy.k8s.local --policy-name alb-ingress-extra --policy-document file://iam-policy.json
pahud:~/environment/kops-cn (master) $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.1/docs/examples/rbac-role.yaml
clusterrole.rbac.authorization.k8s.io/alb-ingress-controller created
clusterrolebinding.rbac.authorization.k8s.io/alb-ingress-controller created
serviceaccount/alb-ingress created
pahud:~/environment/kops-cn (master) $ kubectl  apply -f alb-ingress-controller.yaml 
deployment.apps/alb-ingress-controller created
pahud:~/environment/kops-cn (master) $ kubectl -n kube-system logs -f $(kubectl get po -n kube-system | egrep -o alb-ingress[a-zA-Z0-9-]+)
-------------------------------------------------------------------------------
AWS ALB Ingress controller
  Release:    v1.0.1
  Build:      git-ebac62dd
  Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller.git
-------------------------------------------------------------------------------

W0113 15:29:46.996765       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0113 15:29:47.037786       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I0113 15:29:47.037933       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I0113 15:29:47.038054       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0113 15:29:47.038275       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}}
I0113 15:29:47.038441       1 leaderelection.go:185] attempting to acquire leader lease  kube-system/ingress-controller-leader-alb...
I0113 15:29:47.053838       1 leaderelection.go:194] successfully acquired lease kube-system/ingress-controller-leader-alb
I0113 15:29:47.154268       1 :0] kubebuilder/controller "level"=0 "msg"="Starting Controller"  "Controller"="alb-ingress-controller"
I0113 15:29:47.254530       1 :0] kubebuilder/controller "level"=0 "msg"="Starting workers"  "Controller"="alb-ingress-controller" "WorkerCount"=1
^C
pahud:~/environment/kops-cn (master) $ ^C
pahud:~/environment/kops-cn (master) $ ^C
pahud:~/environment/kops-cn (master) $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.1/docs/examples/2048/2048-namespace.yaml

namespace/2048-game created
pahud:~/environment/kops-cn (master) $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.1/docs/examples/2048/2048-deployment.yaml
deployment.extensions/2048-deployment created
pahud:~/environment/kops-cn (master) $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.1/docs/examples/2048/2048-service.yaml
service/service-2048 created
pahud:~/environment/kops-cn (master) $ kubectl  apply -f 2048-ingress.yaml 
ingress.extensions/2048-ingress created
pahud:~/environment/kops-cn (master) $ kubectl -n 2048-game describe ing/2048-ingress
Name:             2048-ingress
Namespace:        2048-game
Address:          63e8a1ac-2048game-2048ingr-6fa0-819625613.cn-northwest-1.elb.amazonaws.com.cn
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     
        /*   service-2048:80 (<none>)
Annotations:
  alb.ingress.kubernetes.io/target-type:             ip
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"alb.ingress.kubernetes.io/scheme":"internet-facing","alb.ingress.kubernetes.io/target-type":"ip","kubernetes.io/ingress.class":"alb"},"labels":{"app":"2048-ingress"},"name":"2048-ingress","namespace":"2048-game"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"service-2048","servicePort":80},"path":"/*"}]}}]}}

  kubernetes.io/ingress.class:       alb
  alb.ingress.kubernetes.io/scheme:  internet-facing
Events:
  Type    Reason  Age   From                    Message
  ----    ------  ----  ----                    -------
  Normal  CREATE  59s   alb-ingress-controller  LoadBalancer 63e8a1ac-2048game-2048ingr-6fa0 created, ARN: arn:aws-cn:elasticloadbalancing:cn-northwest-1:937788672844:loadbalancer/app/63e8a1ac-2048game-2048ingr-6fa0/9ccfa93325ae76a5
  Normal  CREATE  58s   alb-ingress-controller  rule 1 created with conditions [{    Field: "path-pattern",    Values: ["/*"]  }]
pahud:~/environment/kops-cn (master) $

$ curl -s http://63e8a1ac-2048game-2048ingr-6fa0-819625613.cn-northwest-1.elb.amazonaws.com.cn | grep title                                                                                                                        
  <title>2048</title>
      <h1 class="title">2048</h1>

pahud · 2019-01-16T04:31:32Z

Unfortunately, kops with amazonlinux seems to have some inter-pod communication issue
see: kubernetes/kops#6341
We can't switch to amazonlinux2 before we get it sorted.

Cplo · 2019-03-20T03:34:56Z

why not use kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs-2018-08-17 ?

pahud · 2019-03-20T06:37:29Z

@Cplo Kops supports many different linux distributions
https://github.com/kubernetes/kops/blob/master/docs/images.md

To make sure it works best in AWS and eliminate potential compatibility issues and complexity, we plan to stick to Amazon Linux 2 instead, given Amazon EKS is using it's optimized AMI based on Amazon Linux 2 as well. However, we were still having some issue here, so we prefer to stick to CoreOS given it's still the proven working image for Kops in Beijing and Ningxia.

You still can change the AMI to any variant linux images kops supports. Just pass AMI into the Makefile and it will use the AMI ID you specified.

pahud self-assigned this Jan 11, 2019

pahud mentioned this issue Jul 18, 2019

CoreOS AMI 集群正常运行几小时后，node总是进入NotReady状态。 #91

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

change the default AMI from CoreOS to Amazon Linux 2 LTS #47

change the default AMI from CoreOS to Amazon Linux 2 LTS #47

pahud commented Jan 11, 2019 •

edited

Loading

pahud commented Jan 11, 2019 •

edited

Loading

pahud commented Jan 12, 2019

pahud commented Jan 12, 2019

pahud commented Jan 13, 2019

pahud commented Jan 16, 2019

Cplo commented Mar 20, 2019

pahud commented Mar 20, 2019

change the default AMI from CoreOS to Amazon Linux 2 LTS #47

change the default AMI from CoreOS to Amazon Linux 2 LTS #47

Comments

pahud commented Jan 11, 2019 • edited Loading

Background

TODO

pahud commented Jan 11, 2019 • edited Loading

pahud commented Jan 12, 2019

pahud commented Jan 12, 2019

pahud commented Jan 13, 2019

pahud commented Jan 16, 2019

Cplo commented Mar 20, 2019

pahud commented Mar 20, 2019

pahud commented Jan 11, 2019 •

edited

Loading

pahud commented Jan 11, 2019 •

edited

Loading