Skip to content

screwdriver-cd/aws-build-cluster

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

16 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

K8s infrastructure for Screwdriver build cluster on AWS

Introduction

This repository is meant to serve as a quickstart tool to provision necessary EKS cluster and cloud infrastructure resources required by Screwdriver build cluster on AWS. The most trivial platform to deploy Screwdriver build cluster is Kubernetes as the build cluster itself employs a set of executor plugins which speak fluently with K8s under the cover. Also having K8s be the backbone of the container orchestration will abstract away the complexity of running multiple build clusters on AWS.

For the sake of simplicity and ease of use, we recommend administering a K8s cluster with managed Kubernetes offering from the cloud provider, in this case with AWS, instead of self managed K8s cluster.

The second component of this tool is to help deploy a standard Screwdriver build cluster worker on the K8s cluster to communicate to the job queue from the main Screwdriver application, whether it is hosted by you or Screwdriver team.

This tool relies heavily on other open source tools, especially eksctl as IaC for setting up EKS cluster, as well as kubectl for native K8s deployment.

Prerequisite

Tool Dependencies

The followings are the external dependencies required to run this tool:

All of these tools can be installed via Homebrew on Mac OS X.

Getting RabbitMQ credentials for build cluster worker

You should have already reached out to the target Screwdriver cluster admins to obtain the credential for the RabbitMQ running in the cluster. Also to make sure Screwdriver API recognize the new build cluster, one should:

  • update the database build cluster table to register the new build cluster
  • declare and bind corresponding queue on RabbitMQ for the new build cluster

It will be much more straightforward if you actually own the Screwdriver cluster.

Instructions

To get started, first we recommend composing your own configuration file for the EKS cluster. Please refer to examples/config.yaml for structure and definitions.

Second, configure the AWS CLI by running aws configure with your AWS credentials.

Next, to begin the infrastructure provisioning process:

# by default, run.sh will try to find "config.yaml" at CWD if no argument is passed
# -s/--scripts is intended for custom setup
./run.sh path-to-config.yaml [-s/--scripts comma-separated-list-of-scripts]

./run.sh will first prepare the shell environment setup based on given configs, then sequentially execute bash scripts in scripts/ directory in alphanumerical order. You can freely insert your own bash scripts in the order you want by manipulating the naming convention for the scripts.

However, if you just want to execute certain scripts either from the defaults or something of your own, you can optionally do the following:

# execute scripts "20_deployments" and "30_build_cluster_worker" inside scripts/ with configs from ./examples/existing_vpc.yaml
./run.sh ./examples/existing_vpc.yaml -s ./scripts/20_deployments,./scripts/30_build_cluster_worker

And if you already have a cluster.yaml at the root dir, which is compatible with eksctl, it will be used instead of. Otherwise, a generated cluster.yaml will be created in the directory specified by the config GENERATED_DIR in your config file. (see below)

In contrast, if you wish to delete all nodegroups in your cluster,

./run.sh ./examples/config.yaml -d nodegroup

or even the entire cluster,

./run.sh ./examples/config.yaml -d cluster

Considerations for VPC setup

The complexity of the instructions scales along with the number of resources in the infrastructure. There are basically 3 scenarios:

New VPC with new K8s cluster

This is the clean slate scenario. The only required configuration needed for a new VPC setup is VPC CIDR. The VPC CIDR prefix must be between /16 and /24 and not overlap with 172.17.0.0/16 CIDR range because Docker runs in that range. The range should be determined based on the number of IPv4 addresses and workers in the node group.

Example configuration is located in examples/new_vpc.yaml

Existing VPC with new K8s cluster

For existing VPC and subnets, all we need are the resource IDs of the VPC and its subnets. It's required by eksctl to have at least two private subnets in your VPC for the workers. The only need for a public subnet is to host a NAT gateway for the workers in the private subnets to communicate to public internet. Therefore, we highly recommend reviewing your existing VPC to see if it fits or a new one should be created instead.

Example configuration is located in examples/existing_vpc.yaml

Existing VPC with existing K8s cluster

This scenario assumes that you already have an operating K8s cluster with dedicated ndoe group in your cloud infrastructure. Then all you need to do is to run scripts/30_build_cluster_worker with all the depdendent variables assigned or else compose a config file with what you need and pass it as argument to ./run.sh.

Configurations

The way configuration works is that cluster admins will define configs in config.yaml at the root dir, which serves as a footprint of the environment variables consumed throughout the EKS cluster setup as well as the K8s deployment. run.sh dynamically injects environment variables from these configs with names composed from the keys along the path that are only in uppercase. For example,

CLUSTER:
    NAME: my-cluster
    VPC:
        ID: vpc-xxx
        SUBNETS:
            # this becomes a JSON string
            PRIVATE:
                us-east-1a: { id: "subnet-xxx" }
                us-east-1b: { id: "subnet-yyy" }

Three environment variables will be created:

CLUSTER_NAME=my-cluster
CLUSTER_VPC_ID=vpc-xxx
CLUSTER_VPC_SUBNETS_PRIVATE={"us-east-1a":{"id":"subnet-xxx"},"us-east-1b":{"id":"subnet-yyy"}}

Config Definitions

The following table describes all the configurable options one can put in the config.yaml. There is a fully documented sample config.yaml in the /examples directory.

Name Type Description
CLUSTER_NAME * String Name of the EKS cluster
CLUSTER_REGION * String Name of the AWS region of the EKS cluster
CLUSTER_ENV String Annotated environment, e.g. dev, stage, prod
CLUSTER_OWNER String Email of the owner of the EKS cluster
CLUSTER_EKS_VERSION Number K8s version available in EKS, default version is dictated by whatever default in eksctl
CLUSTER_LOGGING_TYPES String[] CloudWatch Logs settings. Options: ["api", "audit", "authenticator", "controllerManager", "scheduler"] or ["*"] for everything
CLUSTER_ENVELOPE_ENCRYPTION_KEY_ARN String KMS key arn for EKS cluster secrets envelope encryption
CLUSTER_WORKER_INSTANCE_TYPE String EC2 instance type for the worker node group
CLUSTER_WORKER_DESIRED_CAPACITY Integer Starting number of worker in node group
CLUSTER_WORKER_MIN_SIZE Integer Minimum number of worker in node group
CLUSTER_WORKER_MAX_SIZE Integer Maximum number of worker in node group
CLUSTER_WORKER_VOLUME_SIZE Integer EBS gp2 volume size in GiB
CLUSTER_WORKER_VOLUME_ENCRYPTED Boolean Flag to enable EBS volume encryption. If false, EKS managed node group will be used instead of self-managed node group
CLUSTER_WORKER_EBS_VOLUME_ENCRYPTION_KEY_ARN String KMS key arn for EC2 EBS volume encryption. Make sure to grant AWSServiceRoleForAutoScaling the use of the key beforehand. Also you need to grant access of the key to the worker node instance's IAM role if you are creating persistent volume using storage class with KMS encryption
CLUSTER_WORKER_UTILIZATION_THRESHOLD Number Worker utilization level below which downscaling will be performed. Default: 0.5
CLUSTER_PV_SC String K8s storage class to use for PV. Default: gp2
CLUSTER_FARGATE_PROFILE_SELECTORS Object[] Array of selectors (namespace & label) for pod to be run in Fargate node. Avoid including "kube-system" as cluster-autoscaler will run in that namespace and require file system
CLUSTER_VPC_ID String VPC resource ID, for existing VPC setup
CLUSTER_VPC_CIDR String VPC CIDR, for both new and existing VPC setup. Must be between /16 and /24
CLUSTER_VPC_PUBLIC_ACCESS Boolean Allow public cluster endpoint access. Default: true
CLUSTER_VPC_PRIVATE_ACCESS Boolean Allow private cluster endpoint access. Default: true
CLUSTER_VPC_SUBNETS_PRIVATE Object Private subnets, requires at least 2 for existing VPC setup. { [AZ name]: { id: "subnet_id", cidr: "subnet_cidr" } }
CLUSTER_VPC_SUBNETS_PUBLIC Object Public subnets for existing VPC setup. { [AZ name]: { id: "subnet_id", cidr: "subnet_cidr" } }
TAG_REPFIX String Prefix for tag preset name
GENERATED_DIR String Path to store generated manifest files. Default: {root_dir}/generated
SD_K8S_NAMESPACE String K8s namespace for build cluster worker and build workers. Default: sd
SD_K8S_LAUNCHER_VERSION String Screwdriver launcher version to be used on the build worker. Default: latest
SD_K8S_BUILD_PREFIX String Prefix included in build worker pod name, label and other meta data. Default: ""
SD_K8S_CPU_MICRO Number Number of vCPU in the annotated screwdriver.cd/cpu: MICRO setting. Default: 0.5
SD_K8S_CPU_LOW Number Number of vCPU in the annotated screwdriver.cd/cpu: LOW setting. Default: 2
SD_K8S_CPU_HIGH Number Number vCPU in the annotated screwdriver.cd/cpu: HIGH setting. Default: 6
SD_K8S_CPU_TURBO Number Number vCPU in the annotated screwdriver.cd/cpu: TURBO setting. Default: 12
SD_K8S_MEMORY_MICRO Integer Number of memory in GiB in the annotated screwdriver.cd/ram: MICRO setting. Default: 1
SD_K8S_MEMORY_LOW Integer Number of memory in GiB in the annotated screwdriver.cd/ram: LOW setting. Default: 2
SD_K8S_MEMORY_HIGH Integer Number of memory in GiB in the annotated screwdriver.cd/ram: HIGH setting. Default: 12
SD_K8S_MEMORY_TURBO Integer Number of memory in GiB in the annotated screwdriver.cd/ram: TURBO setting. Default: 16
SD_K8S_PRIVILEGED Boolean Enable privileged mode for build container
SD_INSTALL_OPTIONAL Boolean Whether or not to install AWS node termination handler, kubernetes dashboard and prometheus
SD_API_HOST * String Hostname for Screwdriver API service
SD_STORE_HOST * String Hostname for Screwdriver Store service
SD_RABBITMQ_HOST * String RabbitMQ Host FQDN
SD_RABBITMQ_PORT * Integer RabbitMQ Host port number
SD_RABBITMQ_VHOST * String RabbitMQ virtual host name
SD_RABBITMQ_EXCHANGE * String RabbitMQ exchange name
SD_RABBITMQ_QUEUE * String RabbitMQ queue name under given virtual host name in the config
SD_RABBITMQ_DLQ * String RabbitMQ Dead Letter Queue name under given virtual host name in the config
SD_RABBITMQ_USERNAME * String RabbitMQ username
SD_RABBITMQ_PASSWORD * String RabbitMQ password

* required config

About

EKS infra for Screwdriver build cluster on AWS

Resources

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages