AWS EC2 provisioner opening port 2376 on existing security group open to 0.0.0.0/0

[kamileon]

Hi!

Steps
Create cluster using the AWS EC2 provisioner, select an existing security group.

Result
Existing security group is modified - an inbound rule for the TCP 2376, TCP 22 source 0.0.0.0/0 are added!

Expected
Existing security group is not modified.

---

Using the Rancher 2.3.3.

gz#12865
gz#15820

SURE-2505, SURE-2860, SURE-3771

[sowmyav27]

Able to reproduce this on rancher:v2.3.3 and rancher:v2.2.9
Steps:

• Create cluster using the AWS EC2 provisioner, select an existing security group while creating the node template.

Result:

• Existing security group is modified - an inbound rule for the TCP 2376, TCP 22 source 0.0.0.0/0 are added

[sowmyav27]

@kamileon

2376 is docker port, and is required for Rancher to work to be able to talk to the host.

[kamileon]

@sowmyav27 I am aware of the requirements.

I believe when a custom/existing group is used, it should not be modified in any way. The owner takes the responsibility at this point to ensure all the setup is done right as per https://rancher.com/docs/rancher/v2.x/en/installation/references/.

As it stands right now, EC2 provisioner is compromising the security of my setup.

On the side, opening the docker port 2376 to the whole world raises a red flag in my head.

I am looking forward to hear back. Thanks!

[gochist]

Currently docker machine provides an option not to modify the security group(docker/machine#4490). It would be great if rancher provides the option also.

[iamkye]

Having the same issues on our RKE / EC2 clusters. How do we make it stop modifying custom groups on changes? v2.4.5, FYI.

[Israphel]

This issue still exists. When working with Terraform + Rancher it would be ideal if Rancher doesn't mess up the security groups. I tried removing Rancher's permission to add Rules but then everything fails; it won't add machines if it can't add rules, even tho they already exist.

Also, if you assign more than one SG, it will add the same rules everywhere.

[luthermonson]

https://github.com/rancher/machine/blob/master/drivers/amazonec2/amazonec2.go#L1297-L1304 this code probably

[maggieliu]

Can leverage this flag: https://github.com/docker/machine/blob/master/drivers/amazonec2/amazonec2.go#L167. Still need to investigate into the design options (UI requirements?)

[Ellenqs]

I was able to reproduce this on rancher:v2.3.3 and rancher:v2.5.4
Steps:

Create cluster using the AWS EC2 provisioner, select an existing security group while creating the node template. (all open) check if TCP 2376, TCP 22 source 0.0.0.0/0 already exist in the inbound rule, remove them. Then provision the cluster.

Result:

inbound rule for the TCP 2376, TCP 22 source 0.0.0.0/0 are added

[ryansann]

Inbound rules that are added to an existing security group:

Type	Protocol	Port Range	Source
SSH	TCP	22	0.0.0.0/0
Custom TCP	TCP	2376	0.0.0.0/0

Both of these rules are created for Node management purposes.

If we made changes to prevent these rules from being added to security groups, it would result in node provisioning operations failing, unless the existing security group already specified these required rules.

For this reason, instead of removing the logic to add these, we should instead warn users they will be added to their existing security group via the UI.

For EC2 Node Templates, under section 3. Security Groups, when a user specifies Choose one or more existing groups we should show an info/warning box to the user that the aforementioned inbound rules will be added to their selected SG(s). This way, we ensure we are communicating the changes that will be made to the user.

[RP-TSB]

Hello Rancher Dev Team, I would appreciate it if this issue was prioritised instead of removed from every milestone since it affected our security score. Having security groups allowing ports from anywhere is not the best practice.

[Jono-SUSE-Rancher]

Hey @cloudnautique do you want to have a design meeting regarding this specific issue? I know that @ryansann did some research and determined that it was designed to be setup this way, so we don't necessarily want to make any changes without consulting the original intent.

[snasovich]

Design discussion about this issue happened and we are going to stop Rancher from opening ports 22/2376 on user-provided security groups. The fix is likely to move this code under the if-statement below: https://github.com/rancher/machine/blob/168391d2b8861d12cb1df5322e40dafd4cb6b5c4/drivers/amazonec2/amazonec2.go#L1288-L1304

[thedadams]

Root cause

Rancher machine would try to check (in a very unsophisticated way) if ports 22 and 2376 were open on the security group before updating the security group to open them. Because this check was very simple, it led to those ports nearly always being opened.

What was fixed, or what changes have occurred

Now, rancher-machine will only open these ports if the user is using the rancher-node security group.

Areas or cases that should be tested

1. Create (or use) a security group that has all ports open to everywhere. There should be a single inbound rule in the security group that does this. It would be better to not have a bunch of rules opening a bunch of ports to everywhere. Don't name the security group rancher-nodes.
2. Create an EC2 cluster from Rancher using this security group.
3. Previously (before this change) two more rules would be added to this security group: one for port 22 and one for port 2376. After this change, the security group should be unchanged.

What areas could experience regressions?

If a user provides a security that does not allow ports 22 or 2376, then provisioning will fail. This is expected and documentation is being added to cover this new behavior.

Are the repro steps accurate/minimal?

Yes.

[Auston-Ivison-Suse]

Setup For Validation
Rancher: v2-6-head(1d2b746)
Install: Helm
Local k8s: v1.21.10

Validation steps

Test Case 1:

1. Used default security group within ec2.
2. Removed any inbound rules the relate to the ones mentioned in the original issue (22, 2376)
3. In rancher setup provision a cluster.
4. using the above security group when creating the node template.
5. Return to ec2 when provisioning is finished and check the security group to verify no inbound rules were added.

Test Case 2:

1. Create a node template from RKE1 Node template section
2. Go ahead and use the same SG(default) as above, while preemptively checking to make sure no inbound rules that match the original are present. (That will be contained within ec2)
3. Now provision the rke1 cluster with the now created RKE1 Node Template
4. Post the successful provision, return to ec2 and check the inbound rules.

Results
The above test cases passed. The inbound rules were set to the default for kubernetes, but the above rules did not exist.

Setup For Reproduction
Rancher: v2-6-3
Install: Helm
Local k8s: v1.21.8

Steps For Reproduction

1. Provision cluster on any SG
2. Witness within ec2 that the rules mentioned in the original issue have been added.