Infrastructure-as-code script for creating a Managed Kafka Service and VPC for Screwdriver AWS Integration
This repository is meant to serve as an install/uninstall/update script to provision necessary cloud infrastructure resources required for Screwdriver AWS Integration. The following are the resources created by the installation script by default:
- 1 AWS Managed Kafka Cluster
- 3 AWS VPC Endpoint Service (1 for each availability zone of MSK broker endpoint)
- 3 Network Load Balancers (1 for each endpoint service)
- 3 Target Groups (1 for each load balancer)
- 1 Security Group For AWS MSK
- 1 Customer managed KMS Key for the MSK cluster The following resources will be crated with new creation:
- 1 VPC based on the provided cidr block
- Private subnets
- Public subnets
- NAT Gateway
- Internet Gateway
- Route Table
If you opt for installation with an existing vpc, it will skip the vpc infrastructure creation
This script uses open source tool terraform to provision all the resources
The followings are the external dependencies required to run this onboarding script:
All of these tools can be installed via Homebrew on Mac OS X.
Screwdriver API's must be deployed in the same AWS account and region which will connect to the producer service and queue. Also a Amazon Secret Manager secret needs to be created manually (for now) with secret values that will be used in the consumer service.
To get started, update the var file with the required details. Please refer to env.tfvars.json for the variables list. Remove .tmpl extension from file.
Example var file for prod is at example-prod.
Second, configure the AWS CLI by running aws configure with your AWS credentials and select profile for the desired account.
export AWS_PROFILE=<profile_name>
export AWS_REGION=<region_name>
Next, to begin the infrastructure provisioning process:
# by default, setup.sh will try to find "env.tfvars.json"
./setup.sh ./setup.sh will first validate env.tfvars.json for all variables and use default for the ones not found, it will then run terraform init, followed by plan and apply to provision infrastructure.
For step by step installation, you can use the following options
# -i flag will run terraform init and verify backend infrastructure
./setup.sh -i
# -p flag will run terraform plan and create a tf plan
./setup.sh -p
# -a flag will run terraform apply and create the resources
./setup.sh -aYou can also run validation to check for errors before running plan and after running apply by using the -v flag
./setup.sh -vAlternatively, to uninstall all infrastructure
./setup.sh -dThe the number of resources in the infrastructure will be created based on the VPC configuration. There are 2 scenarios
For existing VPC and subnets, all we need are the resource ID of the VPC and the cidr's of the private subnets. If using existing VPC it needs to have both private and public subnets as the resources will be created in private subnets. Also the private subnets should have outbound access to the internet. Therefore, we highly recommend reviewing your existing VPC to see if it fits or a new one should be created instead. Additionally, you can update the other variables like vpc name and consumer function name.
Example configuration for exiting vpc:
aws_region="us-west-2"
tf_backend_bucket="sd-aws-consumer-tf-backend-11111111"
private_subnets = ["10.10.104.0/25", "10.10.104.128/25", "10.10.105.0/25", "10.10.105.128/25"]
vpc_id ="vpc-1234"
msk_cluster_name = "beta-sd-msk"
msk_ebs_vol = 100
msk_instance_type = "kafka.t3.small"
msk_nodes_count = 3
msk_secret_arn = "arn:someExampleSecret"
msk_secret_name = "AmazonMSK_BETA_SD_SECRET"
tags = {PRODUCT:"SCREWDRIVER",ENVIRONMENT:"beta",SERVICE:"sd/producer"}
deploy_env = "beta"In this case a VPC will be created and consumer svc will be provisioned in the new vpc. The required configuration needed for a new VPC setup are the VPC CIDR, the list of private and public subnet cidrs and the availability zones. The VPC CIDR prefix must be between /16 and /24. Additionally, you can update the other variables like vpc name and consumer function name.
Example configuration is new vpc:
aws_region="us-west-2"
tf_backend_bucket="sd-aws-consumer-tf-backend-11111111"
cidr_block = "10.10.104.0/22"
private_subnets = ["10.10.104.0/25", "10.10.104.128/25", "10.10.105.0/25", "10.10.105.128/25"]
public_subnets = ["10.10.106.0/25", "10.10.106.128/25", "10.10.107.0/25", "10.10.107.128/25"]
azs = ["us-west-2a", "us-west-2b", "us-west-2c", "us-west-2d"]
vpc_name ="sd-producer"
msk_cluster_name = "beta-sd-msk"
msk_ebs_vol = 100
msk_instance_type = "kafka.t3.small"
msk_nodes_count = 3
msk_secret_arn = "arn:someExampleSecret"
msk_secret_name = "AmazonMSK_BETA_SD_SECRET"
tags = {PRODUCT:"SCREWDRIVER",ENVIRONMENT:"beta",SERVICE:"sd/producer"}
deploy_env = "beta"The config variables are all part of .tfvars file. These variables will be used in creating the resources.
The following table describes all the configurable variables defined in env.tfvars.json.json
| Name | Type | Description |
|---|---|---|
| aws_region * | String | AWS Region where resources will be provisioned |
| tf_backend_bucket * | String | Terraform backend S3 bucket for storing tf state |
| msk_cluster_name * | String | Screwdriver MSK cluster name |
| msk_ebs_vol * | Integer | EBS volume size for MSK cluster |
| msk_instance_type * | String | Type of ec2 instance for kafka cluster |
| msk_nodes_count * | Integer | Number of nodes for Kafka Cluster |
| msk_secret_name * | String | Name of the MSK secret for authentication |
| vpc_id * | String | User VPC Id |
| private_subnets * | List | List of private subnets |
| public_subnets # | List | List of public subnets |
| cidr_block # | String | CIDR block for the user vpc |
| vpc_name # | String | Name of the user vpc |
| azs # | List | List of availability zones |
| tags * | Map | Map of tags to be used for resource creation |
| deploy_env * | String | The environment prefix where resources will be deployed |
* required config
# required config when creating new vpc
tf_backend_bucket="sd-aws-producer-tf-backend-<accountId>" #replace accountId
msk_ebs_vol=100
msk_instance_type=""kafka.t3.small""
msk_nodes_count=3
msk_secret_name="AmazonMSK_EXAMPLE_SD_SECRET"
vpc_id=null
private_subnets=["10.10.106.0/25", "10.10.106.128/25", "10.10.107.0/25", "10.10.107.128/25"]
cidr_block="10.10.104.0/22"
public_subnets=["10.10.104.0/25", "10.10.104.128/25", "10.10.105.0/25", "10.10.105.128/25"]
azs=["us-west-2a", "us-west-2b", "us-west-2c", "us-west-2d"]
vpc_name="screwdriver-producer"