-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Only 5 SGs per ENI allowed #682
Comments
I wasn't aware of this issue... we create a managed securityGroup that will be attached to worker ENIs for every ingress by default.
@bigkraig I think ideally we can use an shared instance securityGroups for all ALB, and add/delete rules inside that instance securityGroup 😄 I'll make an PR for it |
Thanks. That's perfect. You also need < (300 / $sg_eni_limit) rules in ALL of your security groups, which was meant, for me, I couldn't only increase to 8. |
Currently, I am facing the same issue with the same image quay.io/coreos/alb-ingress-controller:1.0-beta.7 |
Also having this issue. In order to increase SGs per interface you need to decrease rules per SG to maintain the limit of 250 rules per interface ( |
Unfortunately SG limitations and dealing with the repercussions of that are a fact of life in AWS. We had the same problems at Ticketmaster and were able to improve the situation by reusing a lot of crafted groups for specific use cases. We can try to find ways of making the controller more adaptable but in the end there isn't anything that can truly solve the problem. |
I ran into this issue when |
@M00nF1sh Anything we can do to help speed up the creation of a PR? :D |
I guess we need to balance the creation of rules between SGs depending on AWS-limits, as these might differ between AWS accounts, but that also proves a bit difficult, if your cluster spans multiple accounts. There doesn't seem to be a simple fix, but having an option to allow merging certain SGs would be nice. |
It would be nice if we could merge the sgs on the instance side since it will always have the same ruleset anyway, allowing all traffic from the ALB to the worker nodes. That way, we'd only have 1 sg per worker's ENI and won't come near the limit. Anyone already on a PR to implement something similar? |
When this issue was originally filed, the SG per ENI limit times the security rule per group limit had to multiply to less than 300. The multiplicative limit is now 1,000 which leaves a bit more headroom for raising the SG per ENI limit. |
Still the limit is only 16
If we can dynamically create new node SG or merge rules into a single SG, then this will mitigate the problems a lot. Seems that @M00nF1sh already has something that's working (thank you:)), any idea when you would release the code? |
@yifan-gu I'm wondering whether we can fallback to a more simple version as below:
Actually i have implemented it in a new branch for ingress group: https://github.com/M00nF1sh/aws-alb-ingress-controller/tree/ingress-group/ |
Hit the same restriction from AWS today as well, and I think we will asking AWS to increase the limit while we think of any workaround using CF script. Really interested to see goes into stable. So we can have a cleaner implementation. |
@M00nF1sh would it be possible to get an estimate of when the fix for this would be released? |
Also hitting this issue. After a little confusion I finally checked the logs and found:
|
@M00nF1sh Sounds like an easier approach, although the limit on the rules in a SG is not too high as well (60 by default, up to 100 [1]), maybe that's enough for most of the use cases. |
@M00nF1sh is there anything we can do to help you get the fix PR that you have developed merged in? |
It appears that no movement is being made on this issue. I'd like to comment that this is also affecting our organization. Perhaps we need to be more vocal about how we're being affected by this limitation? Is there something blocking progress? |
@irlevesque @diclophis For
@irlevesque It can be easily mitigated by manually create a securityGroup, and use |
@M00nF1sh creating each security group manually isn't really workable in our case and this issue is a big blocker for using this nice ingress-controller :( |
This is impacting some projects of mine as well. One issue I see with the |
The annotation The behavior is pretty similar to what Kubernetes AWS cloud provider suggests (Nginx ingress controller uses that library) with the similar annotation The difference is how they manage security groups for an EC2 instance. Generally, speaking, the traffic needs to be delivered to an instance. It goes the following way: Once the load balancer is spun up and the security groups (specified or created) attached to it the controller searches for the security groups attached the network interfaces (ENI) amongst all of the instances in the cluster. Once it found the group(s), it adds the rule so that there's a permission on all network interfaces of all the instance in the cluster allowing traffic from the load balancer to it.
This one tries to be more accurate in specifying rules and only in the case when the annotation Is that the right logic?I'm wondering why alb doesn't have the same logic as AWS cloud provider? It seems to be pretty neat and right. The fact that alb ingress controller in an The workaround.As been said, the workaround for this is to manage security groups externally. In our organization, we used Terraform for spinning up EKS cluster. So the groups for load balancers are going to be managed from there. I had some doubts whether Terraform would conflict with changes made by AWS cloud provider (that adds rules into existing SGs attached to an instance), but as turned out it doesn't. When you apply a plan, it just adds additional rules to its state and doesn't manage those if there's no rule (aws_security_group_rule) matching the name. |
any news on this? |
Hello, While the Thanks! |
March 24-2020. SecurityGroupsPerInterfaceLimitExceeded was also thrown on my case. @M00nF1sh https://aws.amazon.com/premiumsupport/knowledge-center/increase-security-group-rule-limit/ Thanks! |
https://github.com/Alexx-G Edit: |
I don't understand what you mean. The logs are the reason I was able to find this issue:
|
Agreed, Edit, just for curiosity I took a look at cloud provider and found this feature request. Interestingly enough it seems they handle everything on a single SG with no OPTION for multiple SG support. |
@bigkraig How about Node Affinity with Autoscaling? More nodes more ENIs. PODS pertaining to an Ingress that can't get on an ENI can be moved to a new node. |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
I got this error today:
failed association of SecurityGroups due to failed to reconcile managed Instance securityGroup attachment due to SecurityGroupsPerInterfaceLimitExceeded: The maximum number of security groups per interface has been reached.
using
quay.io/coreos/alb-ingress-controller:1.0-beta.7
The text was updated successfully, but these errors were encountered: