[Feature Request] Ability to specify additional node security group rules

[lbogdan]

Currently the node security group rules are hardcoded in securitygroup.ts:35, and the only way to alter them is to create the security group ourselves, copy-paste those rules, add additional ones, and send it in nodeSecurityGroup when creating a cluster or node group.

My specific use case for this is allowing SSH to a cluster / node group's nodes.

In the name of convenience, I can think of 3 ways to improve this:

1. Be able to specify additional node security group rules when creating clusters / node groups.
2. Be able to specify a callback that gets the default rules and can alter them (same mechanism as transformations callback in ConfigGroup, for example).
3. Have an allowSSH property in ClusterOptions / ClusterNodeGroupOptions.

[lbogdan]

The proposed API for 1. would be to add

additionalNodeSecurityGroupRules: pulumi.Input<{
  egress?: pulumi.Input<pulumi.Input<{
      cidrBlocks?: pulumi.Input<pulumi.Input<string>[]>;
      description?: pulumi.Input<string>;
      fromPort: pulumi.Input<number>;
      ipv6CidrBlocks?: pulumi.Input<pulumi.Input<string>[]>;
      prefixListIds?: pulumi.Input<pulumi.Input<string>[]>;
      protocol: pulumi.Input<string>;
      securityGroups?: pulumi.Input<pulumi.Input<string>[]>;
      self?: pulumi.Input<boolean>;
      toPort: pulumi.Input<number>;
  }>[]>,
  ingress?: pulumi.Input<pulumi.Input<{
    cidrBlocks?: pulumi.Input<pulumi.Input<string>[]>;
    description?: pulumi.Input<string>;
    fromPort: pulumi.Input<number>;
    ipv6CidrBlocks?: pulumi.Input<pulumi.Input<string>[]>;
    prefixListIds?: pulumi.Input<pulumi.Input<string>[]>;
    protocol: pulumi.Input<string>;
    securityGroups?: pulumi.Input<pulumi.Input<string>[]>;
    self?: pulumi.Input<boolean>;
    toPort: pulumi.Input<number>;
  }>[]>
}>;

rules that will be simply merged with the default rules in createNodeGroupSecurityGroup(https://github.com/pulumi/pulumi-eks/blob/master/nodejs/eks/securitygroup.ts#L35).

Although I'd prefer 2., which is more flexible (for example, you can't replace the default egress default "allow all" with something more specific by using 1.), I would propose something like

nodeSecurityGroupTransform?: (pulumi.Input<aws.ec2.SecurityGroup>) => pulumi.Input<aws.ec2.SecurityGroup>;

which can either mutate the provided security group, or return a new one.

@lukehoban @metral What do you think?

[metral]

I'm thinking option 1 is the best path forward here as it makes expectations clear about the user brings and what the cluster requires.

@lukehoban thoughts?

[d-nishi]

@lbogdan @lukehoban @metral

agree that option 1 is better because a) the alb ingress group implementation relies on the EKS node-security group; and b) nodes allow for multiple security groups;

When implementing this the logic 1 thing to consider is that the ALB/ELB/NLB in k8s will use the default security group on the node with the clustertag. If a new SG with the same clustertag is found then an error will be logged, so we need to factor that in when implementing option 1.

[metral]

Closing in favor of #275