This repository is meant to serve as a quickstart tool to provision necessary EKS cluster and cloud infrastructure resources required by Screwdriver build cluster on AWS. The most trivial platform to deploy Screwdriver build cluster is Kubernetes as the build cluster itself employs a set of executor plugins which speak fluently with K8s under the cover. Also having K8s be the backbone of the container orchestration will abstract away the complexity of running multiple build clusters on AWS.
For the sake of simplicity and ease of use, we recommend administering a K8s cluster with managed Kubernetes offering from the cloud provider, in this case with AWS, instead of self managed K8s cluster.
The second component of this tool is to help deploy a standard Screwdriver build cluster worker on the K8s cluster to communicate to the job queue from the main Screwdriver application, whether it is hosted by you or Screwdriver team.
This tool relies heavily on other open source tools, especially eksctl as IaC for setting up EKS cluster, as well as kubectl for native K8s deployment.
The followings are the external dependencies required to run this tool:
All of these tools can be installed via Homebrew on Mac OS X.
You should have already reached out to the target Screwdriver cluster admins to obtain the credential for the RabbitMQ running in the cluster. Also to make sure Screwdriver API recognize the new build cluster, one should:
- update the database build cluster table to register the new build cluster
- declare and bind corresponding queue on RabbitMQ for the new build cluster
It will be much more straightforward if you actually own the Screwdriver cluster.
To get started, first we recommend composing your own configuration file for the EKS cluster. Please refer to examples/config.yaml for structure and definitions.
Second, configure the AWS CLI by running aws configure with your AWS credentials.
Next, to begin the infrastructure provisioning process:
# by default, run.sh will try to find "config.yaml" at CWD if no argument is passed
# -s/--scripts is intended for custom setup
./run.sh path-to-config.yaml [-s/--scripts comma-separated-list-of-scripts]./run.sh will first prepare the shell environment setup based on given configs, then sequentially execute bash scripts in scripts/ directory in alphanumerical order. You can freely insert your own bash scripts in the order you want by manipulating the naming convention for the scripts.
However, if you just want to execute certain scripts either from the defaults or something of your own, you can optionally do the following:
# execute scripts "20_deployments" and "30_build_cluster_worker" inside scripts/ with configs from ./examples/existing_vpc.yaml
./run.sh ./examples/existing_vpc.yaml -s ./scripts/20_deployments,./scripts/30_build_cluster_workerAnd if you already have a cluster.yaml at the root dir, which is compatible with eksctl, it will be used instead of. Otherwise, a generated cluster.yaml will be created in the directory specified by the config GENERATED_DIR in your config file. (see below)
In contrast, if you wish to delete all nodegroups in your cluster,
./run.sh ./examples/config.yaml -d nodegroupor even the entire cluster,
./run.sh ./examples/config.yaml -d clusterThe complexity of the instructions scales along with the number of resources in the infrastructure. There are basically 3 scenarios:
- New VPC with new K8s cluster
- Existing VPC with new K8s cluster
- Existing VPC with existing K8s cluster
This is the clean slate scenario. The only required configuration needed for a new VPC setup is VPC CIDR. The VPC CIDR prefix must be between /16 and /24 and not overlap with 172.17.0.0/16 CIDR range because Docker runs in that range. The range should be determined based on the number of IPv4 addresses and workers in the node group.
Example configuration is located in examples/new_vpc.yaml
For existing VPC and subnets, all we need are the resource IDs of the VPC and its subnets. It's required by eksctl to have at least two private subnets in your VPC for the workers. The only need for a public subnet is to host a NAT gateway for the workers in the private subnets to communicate to public internet. Therefore, we highly recommend reviewing your existing VPC to see if it fits or a new one should be created instead.
Example configuration is located in examples/existing_vpc.yaml
This scenario assumes that you already have an operating K8s cluster with dedicated ndoe group in your cloud infrastructure. Then all you need to do is to run scripts/30_build_cluster_worker with all the depdendent variables assigned or else compose a config file with what you need and pass it as argument to ./run.sh.
The way configuration works is that cluster admins will define configs in config.yaml at the root dir, which serves as a footprint of the environment variables consumed throughout the EKS cluster setup as well as the K8s deployment. run.sh dynamically injects environment variables from these configs with names composed from the keys along the path that are only in uppercase. For example,
CLUSTER:
NAME: my-cluster
VPC:
ID: vpc-xxx
SUBNETS:
# this becomes a JSON string
PRIVATE:
us-east-1a: { id: "subnet-xxx" }
us-east-1b: { id: "subnet-yyy" }Three environment variables will be created:
CLUSTER_NAME=my-cluster
CLUSTER_VPC_ID=vpc-xxx
CLUSTER_VPC_SUBNETS_PRIVATE={"us-east-1a":{"id":"subnet-xxx"},"us-east-1b":{"id":"subnet-yyy"}}The following table describes all the configurable options one can put in the config.yaml. There is a fully documented sample config.yaml in the /examples directory.
| Name | Type | Description |
|---|---|---|
| CLUSTER_NAME * | String | Name of the EKS cluster |
| CLUSTER_REGION * | String | Name of the AWS region of the EKS cluster |
| CLUSTER_ENV | String | Annotated environment, e.g. dev, stage, prod |
| CLUSTER_OWNER | String | Email of the owner of the EKS cluster |
| CLUSTER_EKS_VERSION | Number | K8s version available in EKS, default version is dictated by whatever default in eksctl |
| CLUSTER_LOGGING_TYPES | String[] | CloudWatch Logs settings. Options: ["api", "audit", "authenticator", "controllerManager", "scheduler"] or ["*"] for everything |
| CLUSTER_ENVELOPE_ENCRYPTION_KEY_ARN | String | KMS key arn for EKS cluster secrets envelope encryption |
| CLUSTER_WORKER_INSTANCE_TYPE | String | EC2 instance type for the worker node group |
| CLUSTER_WORKER_DESIRED_CAPACITY | Integer | Starting number of worker in node group |
| CLUSTER_WORKER_MIN_SIZE | Integer | Minimum number of worker in node group |
| CLUSTER_WORKER_MAX_SIZE | Integer | Maximum number of worker in node group |
| CLUSTER_WORKER_VOLUME_SIZE | Integer | EBS gp2 volume size in GiB |
| CLUSTER_WORKER_VOLUME_ENCRYPTED | Boolean | Flag to enable EBS volume encryption. If false, EKS managed node group will be used instead of self-managed node group |
| CLUSTER_WORKER_EBS_VOLUME_ENCRYPTION_KEY_ARN | String | KMS key arn for EC2 EBS volume encryption. Make sure to grant AWSServiceRoleForAutoScaling the use of the key beforehand. Also you need to grant access of the key to the worker node instance's IAM role if you are creating persistent volume using storage class with KMS encryption |
| CLUSTER_WORKER_UTILIZATION_THRESHOLD | Number | Worker utilization level below which downscaling will be performed. Default: 0.5 |
| CLUSTER_PV_SC | String | K8s storage class to use for PV. Default: gp2 |
| CLUSTER_FARGATE_PROFILE_SELECTORS | Object[] | Array of selectors (namespace & label) for pod to be run in Fargate node. Avoid including "kube-system" as cluster-autoscaler will run in that namespace and require file system |
| CLUSTER_VPC_ID | String | VPC resource ID, for existing VPC setup |
| CLUSTER_VPC_CIDR | String | VPC CIDR, for both new and existing VPC setup. Must be between /16 and /24 |
| CLUSTER_VPC_PUBLIC_ACCESS | Boolean | Allow public cluster endpoint access. Default: true |
| CLUSTER_VPC_PRIVATE_ACCESS | Boolean | Allow private cluster endpoint access. Default: true |
| CLUSTER_VPC_SUBNETS_PRIVATE | Object | Private subnets, requires at least 2 for existing VPC setup. { [AZ name]: { id: "subnet_id", cidr: "subnet_cidr" } } |
| CLUSTER_VPC_SUBNETS_PUBLIC | Object | Public subnets for existing VPC setup. { [AZ name]: { id: "subnet_id", cidr: "subnet_cidr" } } |
| TAG_REPFIX | String | Prefix for tag preset name |
| GENERATED_DIR | String | Path to store generated manifest files. Default: {root_dir}/generated |
| SD_K8S_NAMESPACE | String | K8s namespace for build cluster worker and build workers. Default: sd |
| SD_K8S_LAUNCHER_VERSION | String | Screwdriver launcher version to be used on the build worker. Default: latest |
| SD_K8S_BUILD_PREFIX | String | Prefix included in build worker pod name, label and other meta data. Default: "" |
| SD_K8S_CPU_MICRO | Number | Number of vCPU in the annotated screwdriver.cd/cpu: MICRO setting. Default: 0.5 |
| SD_K8S_CPU_LOW | Number | Number of vCPU in the annotated screwdriver.cd/cpu: LOW setting. Default: 2 |
| SD_K8S_CPU_HIGH | Number | Number vCPU in the annotated screwdriver.cd/cpu: HIGH setting. Default: 6 |
| SD_K8S_CPU_TURBO | Number | Number vCPU in the annotated screwdriver.cd/cpu: TURBO setting. Default: 12 |
| SD_K8S_MEMORY_MICRO | Integer | Number of memory in GiB in the annotated screwdriver.cd/ram: MICRO setting. Default: 1 |
| SD_K8S_MEMORY_LOW | Integer | Number of memory in GiB in the annotated screwdriver.cd/ram: LOW setting. Default: 2 |
| SD_K8S_MEMORY_HIGH | Integer | Number of memory in GiB in the annotated screwdriver.cd/ram: HIGH setting. Default: 12 |
| SD_K8S_MEMORY_TURBO | Integer | Number of memory in GiB in the annotated screwdriver.cd/ram: TURBO setting. Default: 16 |
| SD_K8S_PRIVILEGED | Boolean | Enable privileged mode for build container |
| SD_INSTALL_OPTIONAL | Boolean | Whether or not to install AWS node termination handler, kubernetes dashboard and prometheus |
| SD_API_HOST * | String | Hostname for Screwdriver API service |
| SD_STORE_HOST * | String | Hostname for Screwdriver Store service |
| SD_RABBITMQ_HOST * | String | RabbitMQ Host FQDN |
| SD_RABBITMQ_PORT * | Integer | RabbitMQ Host port number |
| SD_RABBITMQ_VHOST * | String | RabbitMQ virtual host name |
| SD_RABBITMQ_EXCHANGE * | String | RabbitMQ exchange name |
| SD_RABBITMQ_QUEUE * | String | RabbitMQ queue name under given virtual host name in the config |
| SD_RABBITMQ_DLQ * | String | RabbitMQ Dead Letter Queue name under given virtual host name in the config |
| SD_RABBITMQ_USERNAME * | String | RabbitMQ username |
| SD_RABBITMQ_PASSWORD * | String | RabbitMQ password |
* required config