Configure AWS cloud provider in Kubernetes when deployed to AWS

[eric-sonchaiwanich]

Request an option, or default behavior, where an AWS cloud provider is configured in Kubernetes when deployed via conjure up. This is a prerequisite for other desirable features available in upstream kubernetes on AWS - enabling dynamic storage provisioning with EBS backed volumes, as well as dynamic ELB provisioning and Route53 integration.

[eric-sonchaiwanich]

This is related to #345, but a more general request. #345 is specific to dynamically provisioning EBS volumes.

[tvansteenburgh]

So I think this involves (per https://kubernetes.io/docs/getting-started-guides/scratch/#cloud-providers):

1. Setting --cloud-provider on the api-server
2. Setting --cloud-config on the api-server (if the cloud requires it)

Should be straightforward to do this via conjure-up since the cloud is known, and the path to the config file could be collected in a configuration step.

On the charm side, we could add config options for cloud-provider and cloud-config.

@eric-sonchaiwanich Do you happen to have an example of a cloud-config file for AWS? I'm having trouble finding an example.

@stokachu @marcoceppi and everyone else, any gotchas to this that I'm not thinking of?

[marcoceppi]

This isn't actually straightforward. You need to have IAM profiles in place, those profiles attached to the right security groups, instances launched a certain way, and some other admin features. Most kube-up scripts mask this complexity, resulting in what seems like an easy turn up. https://medium.com/@samnco/using-aws-elbs-with-the-canonical-distribution-of-kubernetes-9b4d198e2101

[marcoceppi]

That being said, it's totally worthwhile doing this, but it's not cut and dry.

[tvansteenburgh]

@marcoceppi Good info, thanks. It looks like Sam's post documents all the steps necessary to get some minimal aws integration going. Kind of messy, but at least we can hide that inside the conjure-up spell(s).

[marcoceppi]

Yes, conjure is a good start, though until it does day 2 (scaling) adding a unit of k8s worker won't have the right profile data

…

On Tue, Jul 11, 2017, 22:11 Tim Van Steenburgh ***@***.***> wrote: @marcoceppi <https://github.com/marcoceppi> Good info, thanks. It looks like Sam's post documents all the steps necessary to get some minimal aws integration going. Kind of messy, but at least we can hide that inside the conjure-up spell(s). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#346 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAET1axoPLjrMpLvQmTOXJhwGQzWVBSgks5sM9b1gaJpZM4OUbDv> .

[eric-sonchaiwanich]

I don't have an example but recommend looking into how kops configures the cloud provider. I attempted to look myself bu didn't see a configuration file passed as an arg to api-server or any other kubernetes process. What timeframe does this look like this could be addressed?

…

On Jul 11, 2017 5:45 PM, "Marco Ceppi" ***@***.***> wrote: Yes, conjure is a good start, though until it does day 2 (scaling) adding a unit of k8s worker won't have the right profile data On Tue, Jul 11, 2017, 22:11 Tim Van Steenburgh ***@***.***> wrote: > @marcoceppi <https://github.com/marcoceppi> Good info, thanks. It looks > like Sam's post documents all the steps necessary to get some minimal aws > integration going. Kind of messy, but at least we can hide that inside the > conjure-up spell(s). > > — > You are receiving this because you were mentioned. > > > Reply to this email directly, view it on GitHub > <https://github.com/juju-solutions/bundle-canonical- kubernetes/issues/346#issuecomment-314557620>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ AAET1axoPLjrMpLvQmTOXJhwGQzWVBSgks5sM9b1gaJpZM4OUbDv> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#346 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACIel7oo-TD6352eEJuTdzt4KyJfTK8Aks5sM-0CgaJpZM4OUbDv> .

[marcoceppi]

There isn't any configuration other than the parameters being set and the requisite steps being executed on the cloud provider side.

Work is underway to allow passing extra command line arguments to the charms in a declarative fashion, at the same time a spell for conjure-up would have to be updated to do all the pre-install scripts.

There's a reason CDK works against bare metal, vmware, google, azure, rackspace, aws, etc and kops only supports aws. These deep integrations come at a high cost compared to running a stock/vanilla k8s.

I will try to get a POC of a conjure up spell by the end of this week for review.

[tvansteenburgh]

@eric-sonchaiwanich Can't commit to a timeframe just yet. We'll start looking at this in our next development sprint, which starts July 24.

[eric-sonchaiwanich]

@tvansteenburgh @marcoceppi
I seem to have manually enabled dynamic storage provisioning of EBS volumes on CDK - perhaps this is helpful for the implementation.

In summary:

1. Assign AWS IAM roles to Kube masters and workers (this gives master and workers access to AWS credentials, implicit to EC2)
2. Set cloud-provider=aws configuration on apiserver, controller-manager, and kubelet services
3. Configure a storage class provider for aws-ebs

This generally follows the method documented here:
https://medium.com/@samnco/using-aws-elbs-with-the-canonical-distribution-of-kubernetes-9b4d198e2101

Here are the IAM policies used (these could probably be more restrictive)

// cdk-kubernetes-masters-role policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [ "ec2:*" ],
      "Resource": [ "*" ]
    }
  ]
}

// cdk-kubernetes-workers-role policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:AttachVolume",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:DetachVolume",
      "Resource": "*"
    }
  ]
}

Configure cloud provider on master and workers.

# Enable cloud provider on Master
juju ssh 4 "sudo snap set kube-apiserver cloud-provider=aws ; \
  sudo snap set kube-controller-manager cloud-provider=aws"

# Enable cloud provider on Workers
for unit in 5 6 7 ; do
  juju ssh $unit "sudo snap set kubelet cloud-provider=aws"
done

# Reboot
for unit in 4 5 6 7 ; do
  juju ssh $unit "sudo reboot"
done

Create EBS Storage Class provider

cat > sc-aws-ebs.yaml <<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs-1
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
EOF

kubectl create -f sc-aws-ebs.yaml

Create a test claim to validate:

cat > pvc-ebs.yaml <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: my-test-claim-ebs
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: ebs-1
EOF
kubectl create -f pvc-ebs.yaml

Here's how I provisioned the kubernetes cluster (using bundle versions found in the latest stable canonical-kubernetes on jujucharms.com):

$ juju version
2.2.2-xenial-amd64

# Enter AWS service account credentials
juju add-credential aws

# Existing AWS envrionment config
VPC_ID=vpc-xxxx
AWS_REGION=us-east-2
SUBNET_PUBLIC=subnet-xxxx
SUBNET_PRIVATE=subnet-yyyy

juju bootstrap aws/$AWS_REGION k8s-$AWS_REGION \
 --config vpc-id=$VPC_ID \
 --config vpc-id-force=true \
 --to "subnet=$SUBNET_PRIVATE" \
 --bootstrap-constraints "root-disk=128G mem=8G" \
 --credential aws \
 --bootstrap-series xenial \
 --debug

juju add-model --config vpc-id=$VPC_ID cdk-dev

# First, deploy your support applications with:
juju deploy --constraints "instance-type=t2.medium" cs:~containers/etcd-40
juju deploy --constraints "instance-type=t2.medium" cs:~containers/easyrsa-12

# Now enforce your constraints and scale out etcd
juju set-constraints etcd "instance-type=t2.medium"
juju add-unit -n2 etcd

# Now, deploy the Kubernetes Core applications, enforce constraints and scale out:
juju deploy --constraints "cores=2 mem=8G root-disk=32G" cs:~containers/kubernetes-master-35
juju deploy --constraints "instance-type=m4.xlarge" cs:~containers/kubernetes-worker-40
juju deploy cs:~containers/flannel-20
juju set-constraints kubernetes-worker "instance-type=m4.xlarge"
juju add-unit -n2 kubernetes-worker

juju deploy cs:~containers/kubeapi-load-balancer-16

# From the exported bundle
juju add-relation kubernetes-master:kube-api-endpoint kubeapi-load-balancer:apiserver
juju add-relation kubernetes-master:loadbalancer kubeapi-load-balancer:loadbalancer
juju add-relation kubernetes-master:kube-control kubernetes-worker:kube-control
juju add-relation kubernetes-master:certificates easyrsa:client
juju add-relation etcd:certificates easyrsa:client
juju add-relation kubernetes-master:etcd etcd:db
juju add-relation kubernetes-worker:certificates easyrsa:client
juju add-relation kubernetes-worker:kube-api-endpoint kubeapi-load-balancer:website
juju add-relation kubeapi-load-balancer:certificates easyrsa:client
juju add-relation flannel:etcd etcd:db
juju add-relation flannel:cni kubernetes-master:cni
juju add-relation flannel:cni kubernetes-worker:cni


# and expose the master, to connect to the API, and the workers, to get access to the workloads:
juju expose kubernetes-master
juju expose kubernetes-worker

[tvansteenburgh]

@eric-sonchaiwanich This is fantastic, thanks! We'll start looking at this next Monday. Most of the work will happen in conjure-up. I'll link the relevant issues here once we've created them.

cc @stokachu @johnsca

[tvansteenburgh]

cc @battlemidget

[battlemidget]

@tvansteenburgh can you go ahead and assign this to me?

[battlemidget]

@eric-sonchaiwanich Feel free to test this feature out by running conjure-up from our edge channel:

sudo snap refresh conjure-up --classic --edge

[johnsca]

@eric-sonchaiwanich For clarification, when using that edge conjure-up snap to deploy canonical-kubernetes, you should see a third step at the end of the deployment that indicates whether the native integration with AWS is enabled. Of course, the credentials given to conjure-up must have the proper permissions to perform all of the IAM and EC2 operations.

The code in the conjure-up spell is based on Sam's blog post, and can be viewed here. It was tested with the hello-world app and ELB mentioned at the end of the blog, but it should also work for EBS.

[eric-sonchaiwanich]

Great, I will check this out, likely over the next week

…

On Wed, Aug 2, 2017 at 5:42 PM, Cory Johns ***@***.***> wrote: @eric-sonchaiwanich <https://github.com/eric-sonchaiwanich> For clarification, when using that edge conjure-up snap to deploy canonical-kubernetes, you should see a third step at the end of the deployment that indicates whether the native integration with AWS is enabled. Of course, the credentials given to conjure-up must have the proper permissions to perform all of the IAM and EC2 operations. The code in the conjure-up spell is based on Sam's blog post, and can be viewed here <https://github.com/conjure-up/spells/tree/master/canonical-kubernetes/steps/ec2>. It was tested with the hello-world app and ELB mentioned at the end of the blog, but it should also work for EBS. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#346 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACIelxXI92sl_64-tFHlEOAegcwwR9cQks5sUO0rgaJpZM4OUbDv> .

-- Eric Sonchaiwanich Field Engineering | Galactic Fog eric@galacticfog.com (929) 314-2760

[battlemidget]

Thanks @eric-sonchaiwanich

[battlemidget]

@tvansteenburgh @Cynerva @wwwtyro @johnsca

[johnsca]

@pseralat Did you delete your comment? Did you resolve the tagging issue? This function should remove the duplicate tag, so if there's an issue with that we'd appreciate your help debugging it. Could you perhaps provide the describe-security-groups output for the groups with the erroneous tags?

[battlemidget]

FYI: https://docs.ubuntu.com/conjure-up/devel/en/cni/k8s-and-aws

[marcoceppi]

Any reason why these aren't on the upstream docs?

…

On Tue, Aug 15, 2017, 11:18 Adam Stokes ***@***.***> wrote: FYI: https://docs.ubuntu.com/conjure-up/devel/en/cni/k8s-and-aws — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#346 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAET1TPj9YoEinvBflewMKMY758vaH0vks5sYbapgaJpZM4OUbDv> .

[tvansteenburgh]

@marcoceppi No reason, I'll make a card for it.