AWS Security Group rules are removed when adding/removing worker nodes

[zegl]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

When scaling up or down the amount of worker nodes, all rules in the the security group managed by the controller-manager are removed.

Here are the logs from the controller manager when the problem started: It doesn't contain any errors or warnings.

I0522 12:43:25.588545       1 service_controller.go:636] Detected change in list of current cluster nodes. New node set: map[ip-10-1-128-213.eu-west-1.compute.internal:{} ip-10-1-170-254.eu-west-1.compute.internal:{} ip-10-1-149-189.eu-west-1.compute.internal:{} ip-10-1-149-238.eu-west-1.compute.internal:{}]
I0522 12:43:27.168490       1 service_controller.go:644] Successfully updated 15 out of 15 load balancers to direct traffic to the updated set of nodes
I0522 12:43:27.168682       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"ingress-nginx", Name:"ingress-nginx", UID:"b6d3caf6-5290-11e8-a49a-0a96ccd212fe", APIVersion:"v1", ResourceVersion:"133452", FieldPath:""}): type: 'Normal' reason: 'UpdatedLoadBalancer' Updated load balancer with new hosts

I don't know where the number "15" comes from, there's only one LoadBalancer (3 ports, 4 workers, 3 AZs).

Here are the logs from the controller-manager when another node has taken over the leadership:

I0522 12:48:51.268858       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"ingress-nginx", Name:"ingress-nginx", UID:"b6d3caf6-5290-11e8-a49a-0a96ccd212fe", APIVersion:"v1", ResourceVersion:"133452", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0522 12:48:51.282935       1 controller_utils.go:1026] Caches are synced for disruption controller
I0522 12:48:51.282955       1 disruption.go:296] Sending events to api server.
I0522 12:48:52.414735       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"ingress-nginx", Name:"ingress-nginx", UID:"b6d3caf6-5290-11e8-a49a-0a96ccd212fe", APIVersion:"v1", ResourceVersion:"133452", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer

What you expected to happen:

Security group rules should not be removed when the LoadBalancer has not changed.

How to reproduce it (as minimally and precisely as possible):

1. Run Kubernetes worker nodes on AWS,
2. Add a LoadBalancer service that's using AWS NLBs
3. Add/remove a worker nodes.

Anything else we need to know?:

I've only tested this with NLBs, ELBs might not be effected.

The Service has the following annotations:

service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

Restarting the leader controller-manager solves the problem. The new leader will re-add the missing security group rules.

We're running multiple Kubernetes Clusters on the same AWS account.

Environment:

• Kubernetes version (use kubectl version): v1.10.2
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Ubuntu 16.04
• Install tools: None

[zegl]

/sig AWS

[zegl]

I enabled verbose logging and reproduced the problem:

I0522 13:55:42.877392       1 aws_loadbalancer.go:657] Removing rule for client MTU discovery from the network load balancer ([0.0.0.0/0]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:55:42.877432       1 aws_loadbalancer.go:658] Removing rule for client traffic from the network load balancer ([0.0.0.0/0]) to instance (sg-04b4cbf9a3fa15db1)
I0522 13:55:42.877450       1 aws_loadbalancer.go:660] Removing rule for health check traffic from the network load balancer ([0.0.0.0/0]) to instance (sg-04b4cbf9a3fa15db1)
I0522 13:55:42.877472       1 aws_loadbalancer.go:657] Removing rule for client MTU discovery from the network load balancer ([0.0.0.0/0]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:55:42.877488       1 aws_loadbalancer.go:658] Removing rule for client traffic from the network load balancer ([0.0.0.0/0]) to instance (sg-04b4cbf9a3fa15db1)
I0522 13:55:42.877504       1 aws_loadbalancer.go:660] Removing rule for health check traffic from the network load balancer ([0.0.0.0/0]) to instance (sg-04b4cbf9a3fa15db1)
I0522 13:55:42.877524       1 aws_loadbalancer.go:657] Removing rule for client MTU discovery from the network load balancer ([0.0.0.0/0]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:55:42.877540       1 aws_loadbalancer.go:658] Removing rule for client traffic from the network load balancer ([0.0.0.0/0]) to instance (sg-04b4cbf9a3fa15db1)
I0522 13:55:42.877555       1 aws_loadbalancer.go:660] Removing rule for health check traffic from the network load balancer ([0.0.0.0/0]) to instance (sg-04b4cbf9a3fa15db1)
I0522 13:55:42.942559       1 aws.go:2879] Removing security group ingress: sg-04b4cbf9a3fa15db1 [{
  FromPort: 32551,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0",
      Description: "kubernetes.io/rule/nlb/client=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 32551
} {
  FromPort: 31093,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0",
      Description: "kubernetes.io/rule/nlb/client=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 31093
} {
  FromPort: 32030,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0",
      Description: "kubernetes.io/rule/nlb/client=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 32030
}]
I0522 13:55:43.064509       1 aws_loadbalancer.go:660] Removing rule for health check traffic from the network load balancer ([10.1.0.0/16]) to instance (sg-04b4cbf9a3fa15db1)
I0522 13:55:43.064542       1 aws_loadbalancer.go:660] Removing rule for health check traffic from the network load balancer ([10.1.0.0/16]) to instance (sg-04b4cbf9a3fa15db1)
I0522 13:55:43.064585       1 aws_loadbalancer.go:660] Removing rule for health check traffic from the network load balancer ([10.1.0.0/16]) to instance (sg-04b4cbf9a3fa15db1)
I0522 13:55:43.084370       1 aws.go:2879] Removing security group ingress: sg-04b4cbf9a3fa15db1 [{
  FromPort: 31093,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "10.1.0.0/16",
      Description: "kubernetes.io/rule/nlb/health=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 31093
} {
  FromPort: 32030,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "10.1.0.0/16",
      Description: "kubernetes.io/rule/nlb/health=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 32030
} {
  FromPort: 32551,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "10.1.0.0/16",
      Description: "kubernetes.io/rule/nlb/health=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 32551
}]
I0522 13:55:43.164137       1 service_controller.go:326] Not persisting unchanged LoadBalancerStatus for service ingress-nginx/ingress-nginx to registry.
I0522 13:55:43.164469       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"ingress-nginx", Name:"ingress-nginx", UID:"b6d3caf6-5290-11e8-a49a-0a96ccd212fe", APIVersion:"v1", ResourceVersion:"133452", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer

A re-election causes the next kube-controller-manager to re-add the rules:

I0522 13:59:15.782597       1 aws_loadbalancer.go:650] Adding rule for client MTU discovery from the network load balancer ([0.0.0.0/0]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:59:15.782625       1 aws_loadbalancer.go:651] Adding rule for client traffic from the network load balancer ([0.0.0.0/0]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:59:15.782640       1 aws_loadbalancer.go:650] Adding rule for client MTU discovery from the network load balancer ([0.0.0.0/0]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:59:15.782649       1 aws_loadbalancer.go:651] Adding rule for client traffic from the network load balancer ([0.0.0.0/0]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:59:15.782659       1 aws_loadbalancer.go:650] Adding rule for client MTU discovery from the network load balancer ([0.0.0.0/0]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:59:15.782670       1 aws_loadbalancer.go:651] Adding rule for client traffic from the network load balancer ([0.0.0.0/0]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:59:15.801114       1 aws.go:2791] Existing security group ingress: sg-04b4cbf9a3fa15db1 [{
  FromPort: 3,
  IpProtocol: "icmp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0",
      Description: "kubernetes.io/rule/nlb/mtu=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 4
}]
I0522 13:59:15.801170       1 aws.go:2819] Adding security group ingress: sg-04b4cbf9a3fa15db1 [{
  FromPort: 32551,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0",
      Description: "kubernetes.io/rule/nlb/client=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 32551
} {
  FromPort: 32030,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0",
      Description: "kubernetes.io/rule/nlb/client=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 32030
} {
  FromPort: 31093,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0",
      Description: "kubernetes.io/rule/nlb/client=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 31093
}]
I0522 13:59:15.941217       1 aws_loadbalancer.go:653] Adding rule for health check traffic from the network load balancer ([10.1.0.0/16]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:59:15.941249       1 aws_loadbalancer.go:653] Adding rule for health check traffic from the network load balancer ([10.1.0.0/16]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:59:15.941264       1 aws_loadbalancer.go:653] Adding rule for health check traffic from the network load balancer ([10.1.0.0/16]) to instances (sg-04b4cbf9a3fa15db1)
I0522 13:59:15.960322       1 aws.go:2791] Existing security group ingress: sg-04b4cbf9a3fa15db1 [{
  FromPort: 32551,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0",
      Description: "kubernetes.io/rule/nlb/client=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 32551
} {
  FromPort: 31093,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0",
      Description: "kubernetes.io/rule/nlb/client=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 31093
} {
  FromPort: 32030,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0",
      Description: "kubernetes.io/rule/nlb/client=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 32030
} {
  FromPort: 3,
  IpProtocol: "icmp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0",
      Description: "kubernetes.io/rule/nlb/mtu=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 4
}]
I0522 13:59:15.960394       1 aws.go:2819] Adding security group ingress: sg-04b4cbf9a3fa15db1 [{
  FromPort: 32551,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "10.1.0.0/16",
      Description: "kubernetes.io/rule/nlb/health=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 32551
} {
  FromPort: 32030,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "10.1.0.0/16",
      Description: "kubernetes.io/rule/nlb/health=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 32030
} {
  FromPort: 31093,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "10.1.0.0/16",
      Description: "kubernetes.io/rule/nlb/health=ab6d3caf6529011e8a49a0a96ccd212f"
    }],
  ToPort: 31093
}]
I0522 13:59:16.045626       1 service_controller.go:326] Not persisting unchanged LoadBalancerStatus for service ingress-nginx/ingress-nginx to registry.
I0522 13:59:16.045945       1 event.go:218] Event(v1.ObjectReference{Kind:"Service", Namespace:"ingress-nginx", Name:"ingress-nginx", UID:"b6d3caf6-5290-11e8-a49a-0a96ccd212fe", APIVersion:"v1", ResourceVersion:"133452", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer

[zegl]

Honest question: We have one "k8s-managed-lb" shared between all worker nodes. Should we have one managed security group rule per instance instead? I don't know how that would work together with auto scaling groups, but it's worth asking.

[zegl]

/cc @micahhausler

[FrederikNJS]

I'm also testing out Kubernetes 1.10.1 with NLBs for ingress, and I'm seeing exactly the same problem.

[zegl]

@FrederikNS Thanks for letting me know.

I guess we'll have to use ELBs instead until this issue has been resolved.

[FrederikNJS]

It seems that this this might have something to do with using "Private Subnets".

I have 2 clusters running the same version, with the same NLB set up. The only difference between the two clusters is that one has all the worker nodes in public subnets and the other has all nodes in private subnets.

Only the private subnet cluster is experiencing this problem.

I can see from the logs, that right before the ports are removed from the security group, the log outputs Ignoring private subnet for public ELB for all the private worker subnets.

[micahhausler]

Yea this looks like a bug. Thanks for opening this.

[vikasuy]

related: #60825

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[vikasuy]

/remove-lifecycle stale

[kellycampbell]

I think this is fixed by #68422. I tested by replacing a node and watching the security group rules in AWS before/after.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[FrederikNJS]

/remove-lifecycle stale

[FrederikNJS]

I have not been able to test this as kops doesn't support 1.13 yet. And I have not heard anyone report it fixed, so I'd like to keep this open until we have confirmation.

[tuapuikia]

kops 1.11 support k8s 1.11.x. I already tested it on 1.11.7 and it working as expected.

[arielb135]

Does this work on 1.10? was it patched?
what i've experienced - it removes it, and puts it back after ~1 minute - which means, full minute where the service is not active.

in our production cluster it didnt even put it back.

[jochenhebbrecht]

Hi,

Can somebody confirm in which version(s) of K8S this issue is fixed? We are bumping into the same issue and we are on 1.11.6.

Thanks,
Jochen

[kellycampbell]

Please check the changelog files posted on git.

…

On Thu, Mar 14, 2019, 2:04 AM jochenhebbrecht ***@***.***> wrote: Hi, Can somebody confirm in which version(s) of K8S this issue is fixed? We are bumping into the same issue and we are on 1.11.6. Thanks, Jochen — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#64148 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAmNTpF5i9cosKY4eptG4LuP48teeMhAks5vWeZ4gaJpZM4UImi3> .

[jochenhebbrecht]

@kellycampbell thanks! Just located this file: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md and I've noticed it should be fixed in 1.11.6. Unfortunately, that's the version we are on and we're still facing the issue.

From what I can read from the logs, it seems that the leader kubectl-controller-manager pod is stuck. It stopped updating the security group rules.

[kellycampbell]

It's in 1.11.7. Maybe you were confused by the changelog title "Changelog since v1.11.6" ?

…

On Thu, Mar 14, 2019 at 8:38 AM jochenhebbrecht ***@***.***> wrote: @kellycampbell <https://github.com/kellycampbell> thanks! Just located this file: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md and I've noticed it should be fixed in 1.11.6. Unfortunately, that's the version we are on and we're still facing the issue. From what I can read from the logs, it seems that the leader kubectl-controller-manager pod is stuck. It stopped updating the security group rules. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#64148 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAmNTr4aRfY0G2NZ7AZCCOb5pXMzRvlEks5vWkLcgaJpZM4UImi3> .

[jochenhebbrecht]

Yes, sorry, that title confused me.
Ok, we'll try to upgrade our K8S cluster and we'll verify if the issue pops up again or not.

[jaybe78]

I also have a kubernetes cluster in a vpc and I still get the same issue. i'm on kubernetes 1.11.9.
My 2 worker nodes do not pass health check.
@jochenhebbrecht @kellycampbell @arielb135 Have you guys managed to make it work on your side ?

[jochenhebbrecht]

Hi @jaybe78 .

Yes, we managed to make it work by upgrading to 1.11.7. We're currently still on that version and we are no longer bumping into this issue.

Jochen

[M00nF1sh]

@jaybe78 I believe the original issue is already fixed by #68422 (which got back ported to older versions).

Would you help share your serviceSpec and worker node securitygroup settings on AWS to me(@M00nF1sh in k8s slack channel), i can help take a look

[dhanvi]

/sig cloud-provider

[zegl]

I'll close this. The bug has been fixed (and released) on v1.11, v1.12, v1.13, and v1.14.

[1hanymhajna]

Hello,
We just saw the same behavior in v1.18 version,
During node replacement (Scaling up new nodes, and moving old nodes to cordon mode) we notice that the sg has been updated automatically and the relevant ELB rule has been removed.
We are using EKS-1.18, and our load balancer type is ELB, nodes are managed by us.

We can see the event triggered by the workers (in cloudtrail) so it approved it happened automatically from the controller side ( not a human mistake)
But it just happened suddenly, we tried the same flow in other clusters this didn't happen there.
I think ti fix has a specific condition that triggers the bug again or something.

We saw it as well in dev cluster, when we lost some spot nodes it also happened, but again it not happen all the time. so still don't have specific flow how to reproduce it

@zegl

[1hanymhajna]

The fix is just to handle NLB type, doesn't need to make it global? (ELB, ALB) ?