Feature: Restricted SSH access

[cknowles]

As of v0.9.4-rc.3, SSH access is granted to all addresses for controllers, workers, etcd. As an extra layer could we support restrictions on that?

In the case of public controllers but private etcd and workers, generally there may need to be a way to gain SSH access to the private workers by tunnelling through the public controllers. i.e. SSH access allowed from controller SG to worker SG. We could also say that we just support custom bastion and IP address/SG of the bastion has to be provided to allow SSH access to private assets.

[swestcott]

+1 for this.

I feel like there's some overlap with the "glue" security groups feature. If glue SGs where available to the workers, controllers & etcd user would be free to define their security controls as necessary. This would rely on being able to disable the SSH/0.0.0.0 rule.

[mumoshu]

@c-knowles @swestcott How about adding sshAccessCIDR to cluster.yaml?
It will default to 0.0.0.0/32 to keep backward compatibility but can be customized regardless of the subnet in which a worker/controller node is private/public.
Perhaps,0.0.0.0/32 for private worker/controller nodes doesn't hurt? Or do we need a flag to disable the access completely?

An example cluster.yaml would look like:

worker:
  nodePools:
  - name: pool1
    sshAccessCIDR: 0.0.0.0/32
  - name: private1
    sshAccessCIDR: <bastion subnet's CIDR>
    subnets:
    - name: privateSubnet1

controller:
  sshAccessCIDR: <bastion subnet's CIDR>

etcd:
  sshAccessCIDR: <bastion subnet's CIDR>
  subnets:
  - name: privateSubnet1
  ...

[swestcott]

Looks good, could sshAccessCIDR be a list of ranges inside?

[mumoshu]

Yes, with the name `sshAccessCIDRs` 2017年3月24日(金) 18:42 Si Westcott <notifications@github.com>:

…

Looks good, could sshAccessCIDR be a list of ranges inside? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#338 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABV-Zmu2-bKdIXTGC9ViczPTIu2PhVFks5ro4_ngaJpZM4MIHF4> .

[cknowles]

@mumoshu your above idea seems good, I also would be fine with defining it ourselves in the glue SGs etc and then disabling 0.0.0.0/32 somehow like @swestcott mentioned. I assume nothing internal within the cluster needs SSH access for it to function. The only "harm" I can think of if always adding 0.0.0.0/32 to private nodes is those are then potentially accessible via public nodes in the same VPC (there could be many others). I think it's better practice to make these sorts of things non-global and more specific.

[mumoshu]

Thanks @c-knowles!
How about setting sshAccessCIDRs to an empty list or nil to disable 0.0.0.0/32?
cluster.yaml would be like:

worker:
  nodePools:
  - name: pool1
    sshAccessCIDRs:
    securityGroupIds:
    - <your-glue-sg-may-or-may-not-include-your-own-ssh-access-cidrs>

[cknowles]

Assuming we can tell the difference between setting it to empty and it not being there in the YAML parsing, I think that's ok. Not entirely intuitive but I can't think of a better way right now. The only other ideas I had would be if the template just contained the following as the default, it's more verbose but it also makes the default immediately clear.

worker:
  nodePools:
  - name: pool1
    # Remove this if you wish to disable global SSH access
    sshAccessCIDRs: 0.0.0.0/32
#    securityGroupIds:
#    - <your-glue-sg-may-or-may-not-include-your-own-ssh-access-cidrs>

Not setting the sshAccessCIDRs would mean no SSH access but kube-aws would still work as it did out of the box.

[mumoshu]

Thanks @c-knowles, your latter idea seems certainly verbose but clear. Hmm, I'll think of it for a while to see what's the most reasonable now.

[mumoshu]

Hmm...

worker:
  nodePools:
  - name: pool1
    # CIDRs of networks you'd like SSH accesses to be allowed from. Defaults to ["0.0.0.0/32"]. 
    # Explicitly set to an empty array to completely disable it. If you do that, probably you would like to set securityGroupIds to provide this node pool an existing sg with a ssh access allowed from specific ranges.
    #sshAccessCIDRs:
    # - 0.0.0.0/32
    securityGroupIds:
    - <your-glue-sg-may-or-may-not-include-your-own-ssh-access-cidrs>

[cknowles]

Looks fine, I'm a fan of having more of the defaults being uncommented but not sure of your views on that. I was also wondering if we could publish a defaults YAML file in kube-aws somewhere and load defaults from there rather than spread across the code.

[mumoshu]

Any specific reason you'd like it to be a yaml file?
Would it be better to introduce defaults.go which is dedicated to define default values, so that we can utilize the power of go tool-chain like formatting, compile-time errors, etc?

package model

type workernodepool struct {
	Foo string
	Bar string
}

type Worker struct {
	Foo      string
	NodePool workernodepool
}

var Defaults = struct {
	Foo    string
	Worker worker
}{
	Foo: "foo",
	Worker: Worker{
		NodePool: workernodepool{
			"foo",
			"bar",
		},
	},
}

[cknowles]

I thought a YAML would be good to mirror the non-defaults file but I guess defaults for something like worker node pools won't be representable in the same format anyway. A defaults.go would be good.

[jpb]

Is it possible to extend this to limiting external HTTPS traffic as well (current rule) - perhaps by adding apiAccessCIDRs? I need to be able to limit SSH and access to the Kubernetes API to a list of IPs. I suspect if apiAccessCIDRs is introduced, the internal access needs should be added explicitly to the template (so as to not rely on the 0.0.0.0/0 rule) - such as allowing HTTPS traffic from the worker security group(s).

For the specific implementation - sshAccessCIDRs/apiAccessCIDRs (with default ["0.0.0.0/0"]) would satisfy my use-case.

[mumoshu]

@jpb We've added the apiEndpoints setting key for more fine-grained control of API endpoints hence ELBs backing them. I believe it is developed flexible enough to support your request.
Perhaps adding something like apiEndpoints[].loadBalancer.allowedSourceCIDRs which defaults to - 0.0.0.0/32 but is capable of customizations as you like would suffice your request?

Anyway, I'd appreciate it if you could submit an another github issue to address it 🌷

[mumoshu]

I began to prefer sshAllowedSourceCIDRs rather than sshAccessCIDRs for more clarity

[mumoshu]

It turned out that:

• Compared to the ones for etcd and controller, implementing worker.sshAccessAllowedSourceCIDRs is a bit more difficult than I have initially thought
• We didn't have etcd.securityGroupIds and controller.securityGroupIds to attach existing SGs to allow SSH accesses when sshAccessAllowedSourceCIDRs was explicitly set to an empty array

[mumoshu]

I've submitted #551 for this.
@c-knowles @swestcott @jpb Would you mind testing it?

[jpb]

@mumoshu I've tested #551 and it worked as expected 👍

[cmcconnell1]

Cool, have been meaning to look for some SSH hardening and looks like it's been here for awhile.

I've been just rolling my own here with a basic 'finishing' script (post kube-aws up) to configure our SSH access upon cluster deployment. I just run this immediately after cluster deployment to remove world access and allow our allowed subnets to hit our kube nodes--else we see hackers all over the world start their brute force hacking within a few minutes if our clusters are on public nets. That was kinda funny to watch in the kube nodes logs ;-)

With this basic script (for AWS EC2 SG's) it only allows SSH access from within our AWS EC2 and current VPN CIDR ranges. Perhaps it might be useful for others. Disclaimer some pretty gnarly piping here, the tricky part is filtering with jq, but works reliably for me YMMV.

printf "\ngrep kube clusterName from cluster.yaml file setting in the current directory\n"

kube_cluster=$(grep 'clusterName:' ./cluster.yaml | awk -F ": " '{print $2}')

# get our desired kube cluster nodes
# not reliable using text output if the group descriptions are modified, etc
for kube_sg_id in $(aws ec2 describe-security-groups --filters Name=vpc-id,Values=vpc-abc12ca3 | jq -r '.SecurityGroups[] | [.GroupId, .GroupName] | @csv' | grep -i "$kube_cluster" | awk -F ',' '{print $1}' | sed 's/"//g') ; do
    printf "KUBE-SG-ID: $kube_sg_id\n"

    # first we remove the world-access
    aws ec2 revoke-security-group-ingress --group-id ${kube_sg_id} --protocol tcp --port 22 --cidr 0.0.0.0/0

    # now we grant sane default SSH rules for EC2 and VPN admin access
    aws ec2 authorize-security-group-ingress --group-id ${kube_sg_id} --protocol tcp --port 22 --cidr 10.1.0.0/20
    aws ec2 authorize-security-group-ingress --group-id ${kube_sg_id} --protocol tcp --port 22 --cidr 172.31.0.0/24

    printf "\nawless show kube_sg_id: ${kube_sg_id}\n"
    awless show ${kube_sg_id}
done

printf "\nSecurity Group IDs for Kubernetes Cluster: $kube_cluster\n"