Termination + start of new master de-registers nodes from AWS ELB (both CLB and NLB)

[amorey]

What happened:

After a master node instance is terminated and a new master node joins the cluster, worker node instances targeted by AWS ELBs are de-registered from the load balancers. In the case of CLBs, instances are removed as listeners and then added back. In the case of NLBs, instances are drained and then re-initialized.

What you expected to happen:

I expected the new master node to join the cluster gracefully without causing the load balancer to de-register instances.

How to reproduce it (as minimally and precisely as possible):

1. Set up a 3-node cluster using kops (e.g. kops create cluster --zones=us-east-1a,us-east-1b)
2. Create a classic load balancer and a network load balancer service:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: echoserver
  labels:
    app: echoserver
spec:
  replicas: 2
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
        - image: k8s.gcr.io/echoserver:1.10
          name: echoserver
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: 4Mi
            limits:
              memory: 20Mi
          imagePullPolicy: Always
---
kind: Service
apiVersion: v1
metadata:
  name: echoserver-nlb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 8080
  selector:
    app: echoserver
---
kind: Service
apiVersion: v1
metadata:
  name: echoserver-clb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "elb"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"                
    service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"                      
    service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"                        
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"                       
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "3"                     
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "5"                                 
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 8080
  selector:
    app: echoserver

3. Terminate the master node instance from the AWS console
4. Watch the CLB/NLB console screens to see the worker node instances get removed/drained when a new master node joins the cluster

Anything else we need to know?:

• I've only tested it on AWS
• Restarting the master node from the command line (shutdown -r now) does not trigger a load balancer drain
• I tried it with a 3-master HA configuration and found the same behavior when all master node instances were terminated simultaneously
• I saw the same behavior with Kubernetes 1.20.X and 1.21.X (installed with kops-1.20.0-beta.1 and kops-1.21.0-alpha.1 respectively)

Environment:

• Kubernetes version (use kubectl version): 1.19.9
• Cloud provider or hardware configuration: AWS
• OS (e.g: cat /etc/os-release): Ubuntu 20.04.2 LTS (Focal Fossa)
• Kernel (e.g. uname -a): Linux ip-172-20-38-84 5.4.0-1039-aws Make it clear that Kubernetes can run anywhere but that the initial scri... #41-Ubuntu SMP Wed Feb 24 23:13:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kops
• Network plugin and version (if this is a network-related bug): kubenet
• Others:

[neolit123]

/area provider/aws
/sig cloud-provider

[amorey]

Are there any updates on this? It would be helpful to know what the triage timeline usually is. As a supplement here are screenshots of the steps to reproduce the problem (side-by-side browser windows are EC2/ELB/TargetGroup consoles):

Step 1: Healthy 3-node cluster

Step 2: Terminate master node instance

Step 3: New master node instance is initialized automatically

Step 4: When new master node joins cluster, instances are de-registered from CLB and NLB

Step 5: Instances are re-initialized in CLB and NLB

Step 6: Healthy 3-node cluster

[cheftako]

/assign @kishorj
/triage accepted

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kishorj]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[amorey]

/remove-lifecycle stale