Discuss: work around SecurityRuleAddressesOrPortsPerSecurityGroupLimitReached when service floating ip disabled #5919
Description
I'd like to discuss potential solutions for issue #2725 .
The background is we have a large cluster with > 500 nodes and > 30 services. These services have floating ip disabled, so in a NSG rule all backend node IPs are listed as dest addresses. However, NSG has a limit that sum of IPs among all rules must not exceed 4000. In above cluster the value is 500 * 30 = 15000, exceeding the limit a lot.
I don't think NSG has plan to increase such limit. So the only way in my mind is that maybe we can provide an annotation to let user explicitly specify NSG dest addresses(probably a CIDR list), instead of using all backend node IPs.
For example:
"service.beta.kubernetes.io/azure-disable-load-balancer-floating-ip-nsg-dest": "10.1.0.0/16,10.2.0.0/16"
Then we create a NSG rule with destinationAddressPrefix set to 10.1.0.0/16,10.2.0.0/16. It would only count as 2 to the limit.
Thoughts? I'm happy to submit a PR if you're ok with the change.
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Participants
Activity
zarvd commentedon Jun 21, 2024
I am thinking maybe we could set the backend node subnet CIDR as the default destination addresses by default, without requiring additional annotations or settings from the user. Need to double check if it’s feasible.
nilo19 commentedon Jun 24, 2024
Will we lose security if open up all subnet while most of the space may not be used?
nilo19 commentedon Jun 24, 2024
For local services, maybe we can only open nodes where there are endpoints running.
zarvd commentedon Jun 24, 2024
Yes, unfortunately, we would. Had a thought that set CIDR iff nodes * services exceeds the limit; otherwise, set node IP list. Still not sure about the idea.
k8s-triage-robot commentedon Sep 22, 2024
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/remove-lifecycle stale/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
k8s-triage-robot commentedon Oct 22, 2024
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/remove-lifecycle rotten/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
k8s-triage-robot commentedon Nov 21, 2024
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/reopen/remove-lifecycle rottenPlease send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
k8s-ci-robot commentedon Nov 21, 2024
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.