Fixes #49445 by adding additional annotation to define ELB security group (AWS)

[Raffo]

What this PR does / why we need it:
This fixes #49445 by changing the way the SG are applied to AWS ELBs: previously, an ELB would always have a default SG, either a pre-created one or an ad hoc one that always allowed 0.0.0.0/0 even if an annotation with a different security would be specified. This PR changes that by having the "extra" SG be the only one applied when it is specified as this is the way most of the people use AWS anyways: they predefine Security Groups with rules that work with them and they want to be applied.
It would would be probably worth considering changing the name of the annotation as this change of behaviour doesn't really make it "extra".

Which issue(s) this PR fixes :
Fixes #49445

Special notes for your reviewer:
Please advise how to build and test this on AWS to do an integration test, I don't have prior experience with that.

Release note:

Adds `service.beta.kubernetes.io/aws-load-balancer-security-groups` annotation to set the security groups to the AWS ELB to be the only ones specified in the annotation in case this is present (does not add `0.0.0.0/0`). 

/cc @justinsb @chrislovecnm what do you think?

[dims]

/ok-to-test

[justinsb]

Thanks for the PR! So I may be missing something, but I think this is a breaking change - I might previously have been adding a security group to grant access via security-groups (which you can't otherwise do in the API), but I might still want access via a CIDR. This would block the CIDR.

One hacky workaround: you could set the allowed-CIDR to something like 127.0.0.1/32, so that the default rule was still present but you could ignore it. That is very hacky though (and I'm not 100% certain it would work!)

How about another annotation? e.g. service.beta.kubernetes.io/aws-load-balancer-security-groups (without extra)

I think the logic would be that aws-load-balancer-security-groups could replace the primary, as you have today. aws-load-balancer-extra-security-groups should probably still be added just for consistency, though I can't see why you would really use extra unless you wanted to keep the default rules.

Another option is to allow source ranges to specify security groups, or some syntax to disallow all traffic, though I suspect because it's both a field and an annotation (and the field is shared across cloudproviders) that would be harder..

[Raffo]

Hi, thanks for your review!
You are totally right, that's a change that would break the use case you mentioned, so it is probably not a good idea. In the end I saw already in the original comment that the name "extra" in the annotation was not fitting the change. I personally like your suggestion of introducing an additional annotation service.beta.kubernetes.io/aws-load-balancer-security-groups with the behaviour of this PR and keeping the service.beta.kubernetes.io/aws-load-balancer-extra-security-groups as it is right now (even though the name is somehow still a bit confusing).

[Raffo]

I update the PR with the discussed proposal: given that the code is indeed pretty minimal, I'd love to use that to reason if this is really what we are looking for and code makes it easier in this case.

[Raffo]

/test pull-kubernetes-verify

[Raffo]

/test pull-kubernetes-verify

[Raffo]

/test pull-kubernetes-verify

[wrstlrlp13]

Would it not also make sense that if someone used the following under the spec:

loadBalancerSourceRanges:

• xxx.xx.xxx.xx/32
• xxx.xx.xxx.xx/27
• etc.etc.etc/etc

The 0.0.0.0/0 inbound rule would no longer get created?

One would assume that if they're assigning specific load balancer source Ip ranges they don't want everyone to have access...

[Raffo]

@wrstlrlp13 EDIT: that works, the 0.0.0.0/0 rule is only added for ICMP, so the 0.0.0.0/0 would not be allowed if we use the AnnotationLoadBalancerSourceRangesKey annotation.

[Raffo]

/retest

[christopherhein]

@Raffo this seems like a reasonable addition. like @justinsb said the alternative is pretty hacky but what I've used in the past.

Just for documentation sake it might make sense to make it more clear that it's replacing all SGs that would usually be defined. I would say change the last line in the comments to say "Differently from the annotation "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups", this will replace all other security groups assigned to the ELB." at least if I'm reading the code correctly.

👍

[Raffo]

You read it correctly, I'll make the change!

[Raffo]

@justinsb @chrislovecnm can you please take another look?

[tabossert]

👍 Really want this

[Raffo]

Friendly ping /cc @justinsb @zmerlynn @gnufied @jsafrane

[justaugustus]

/lgtm

Can you approve?
/assign @justinsb @zmerlynn @gnufied @jsafrane

[matthias50]

@Raffo Would it be possible to cherry-picked this to 1.12? It seems like a pretty small change and not having this capability causes security issues for us which we'd rather not leave open til 1.13.

[Raffo]

@matthias50 It's technically easy I guess, but it was stated that it is a new feature (it kind of is even though it's fixing something that we can easily say is a bug) and will not be chosen for cherry pick.

[matthias50]

@Raffo, I see the comment from @justinsb. I can see both sides of the story here in that we could look at this as a bug fix or a new feature. According to https://contributor.kubernetes.io/contributors/devel/cherry-picks/, it seems that the branch manager or the patch release team is the one that makes the call as to if this is something we can say is a bug which could be cherry picked or is a new feature.

If it isn't too much trouble, would you mind submitting a PR to cherry pick this and see what the branch managers for 1.12 think (see https://github.com/kubernetes/sig-release/blob/master/releases/release-1.12/release_team.md)? Maybe they agree with @justinb, maybe not. Worst that could happen is that they say no.

[Raffo]

I'd rather ping them here than creating a useless PR that would need to be closed then. I haven't checked the links, would you mind including the relevant people in this conversation?

…

On Mon, Nov 5, 2018, 23:23 Matt Martin ***@***.*** wrote: @Raffo <https://github.com/Raffo>, I see the comment from @justinsb <https://github.com/justinsb>. I can see both sides of the story here in that we could look at this as a bug fix or a new feature. According to https://contributor.kubernetes.io/contributors/devel/cherry-picks/, it seems that the branch manager or the patch release team is the one that makes the call as to if this is something we can say is a bug which could be cherry picked or is a new feature. If it isn't too much trouble, would you mind submitting a PR to cherry pick this and see what the branch managers for 1.12 think (see https://github.com/kubernetes/sig-release/blob/master/releases/release-1.12/release_team.md)? Maybe they agree with @justinb <https://github.com/justinb>, maybe not. Worst that could happen is that they say no. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#62774 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AApv1HJti5rW2Bq2HRoh-0gzSa2DvrNFks5usLp7gaJpZM4TZtge> .

[matthias50]

@dougm, @idealhack, @hoegaarden, @codenrhoden, is this PR one which could be cherry picked back to 1.12?

[ghost]

I understand this is classified as a feature, but given this is kind of a bugfix (depending on how you look at it), is there any chance this could be cherry-picked to 1.11 as well @foxish ? EKS is still on 1.11 and this would solve a lot of problems with security group rule limits for restricted NLBs

[hoegaarden]

@matthias50 @Raffo @cam-cen

For backporting you'd need the patch managers of the releases in question:

• v1.12: @feiskyer
• v1.11: @foxish

As the doc states, they are the authority to bring those changes into v1.1{1,2} or not.

Also you'd need the approval of the original approver (@justinsb in that case) on each of the cherry-pick PRs.
Justin however already stated:

I don't think this should be cherry-picked though - it is new functionality.

... so I'd start with convincing him that those changes need to go into v1.1{1,2} :)

[mqasimsarfraz]

@Raffo Thanks for this feature. I have some question regarding how this should be used. Is loadBalancerSourceRanges a requirement to use this annotation?

Because if loadBalancerSourceRanges and IP address range in security group doesn't match it just purges all the existing rules and adds the default 0.0.0.0/0 also I can see an unattached security groups without any rules created, which is never deleted. Also, I wouldn't expect my existing group to be deleted if ELB is deleted.

[Raffo]

You should be able to use the new annotation without having to specify the ranges, which is exactly the point of this PR. If there is any problem with that, please feel free to submit an issue and quote me on that.

[mqasimsarfraz]

Perfect thanks. Let me open an issue with my findings.

[ac-hibbert]

This doesn't seem to work for me. The new SG I add outside K8S becomes managed by K8S and all the ports are added. I'm looking to have a range for the SG but all the ports that are added to the service are also added to the new SG

[Elias481]

This one should have really been implemented different, it is now very partly and inconvenient.
Proper implementation would be to handle it the same way as cfg.Global.ElbSecurityGroup would handle it (so neighther, then empty, default security groups created nor adjusting the first group in list) - overruling the Global.ElbSecurityGroup.

With current implementation one cannot get rid of the empty default security groups created (which is really unneccesary).
Only advantage in using this feature is in effect, that one can share one managed SG (the first one in list) between services which are known to have the same outside ports and setting it to the same restrictive source-range (which does not help much as long as the other unused groups are created).

[MichelCarroll]

Using service.beta.kubernetes.io/aws-load-balancer-security-groups on our cluster overwrote all the inbound rules off of the first security group in this annotation, which was our shared security group, and brought the platform down. This is definitely not the behaviour I was expecting.

[cazter]

As @ghost mentioned--this is also needed on NLB's which add each allowance to the shared node SG individually instead of creating an SG as ELB's do. This of course causes the shared SG to hit the max more quickly.

[akhmadfld]

Hi
I just comes to this PR, and I think this features is not documented?
I searched https://kubernetes.io/docs/concepts/services-networking/service/ but only found aws-load-balancer-extra-security-groups (the similar one with extra)

And also I want to ask for kubernetes v1.15.9
when I used aws-load-balancer-security-groups, the master instance needs ec2:AuthorizeSecurityGroupIngress and ec2:RevokeSecurityGroupIngress. After I added the missing permissions, the inbound rules were edited to listen 0.0.0.0/0 again

Is this expected? Or maybe I miss something?