This repository houses the Terraform infrastructure code for all of the operating environments for fakebusiness services. All of the included resources are listed below.
Under the hood in AWS, fakebusiness leverages the following:
- AWS Elastic Container Service for Web and Worker container hosting.
- AWS Application Load Balancer for managing web connections.
- AWS Relational Database Service for the PostGres db.
- AWS S3 bucket for storing attachments.
- S3 Bucket and Cloudfront for the React application.
- MSK Serverless for Managed Kafka
- Event-driven lambdas for s3 bucket events
- Twingate VPN
Uses Terraform for management of non-ecs resources, like the RDS PostGres database, as well as all of the networking and cluster configuration. Everything else ECS-centric is managed using ECS Deployer. ECS Deployer allows us to host multiple environments and multiple ECS services, as well as some other niceties, in a single config set. ECS Deployer configuration files can be found in the .ecsdeployer/ directory in the fakebusiness Web App repo.
We can also leverage the companion Remote Shell Client, which allows you to create a session in any Operating Environment without disturbing the existing services. This method is great for spelunking and investigating issues in non-local AWS environments, as you have the exact same configs and settings as the actual running application.
The Fake Business Client is hosted inside of a S3+Cloudfront+WAF Holy Trinity. The S3 bucket houses the live build in a directory called current. The site files inside of current directory represent what is served to the client when they hit a naked domain like app.fakebusiness.com. Outside of that, there will be additional directories in the bucket meant to represent past builds, used for comparison in QA efforts. Previous versions and shas are available in a given environment at <sha>.site.dev.fakebusiness.com or <version>.site.staging.fakebusiness.com. In addition to serving the S3 bucket content on the global distribution network, the Cloudfront also houses a lambda function that provides the dynamic routing described above. The source code for this lambda lives in this repo under the /functions directory.
If addition to the standard S3 Cloudfront combo, I implemented a global Web Access Firewall on the Cloudfront distro. This WAF is meant to block international traffic, and provide a strict IP Whitelisting experiencing for pre-production environments. Updates to the WAF should be made with great caution. The Whitelist IP addresses are housed in the variables.tf file, and currently only include the public IP address of the AWS VPN.
The fakebusiness-API uses Terraform for management of non-ecs resources, like the RDS PostGres database, as well as all of the networking and cluster configuration. Everything else ECS-centric is managed using ECS Deployer. ECS Deployer allows us to host multiple environments and multiple ECS services, as well as some other niceties, in a single config set. ECS Deployer configuration files can be found in the .ecsdeployer/ directory in the fakebusiness API repo.
We can also leverage the companion Remote Shell Client, which allows you to create a session in any Operating Environment without disturbing the existing services. This method is great for spelunking and investigating issues in non-local AWS environments, as you have the exact same configs and settings as the actual running application.
Currently, fakebusiness uses AWS Cognito as a User Service. While this service is configured inside of this Terraform file, there are limitations imposed by AWS that limit the effectiveness of Terraform to manage user pools. Please, use caution when editing the configuration for the Cognito, as most changes will require a destorying of some element to recreate.
These operating environments are managed in the Terraform Workspaces.
| Name | App | API |
|---|---|---|
| dev | app.dev.fakebusiness.com | api.dev.fakebusiness.com |
| staging | app.staging.fakebusiness.com | api.staging.fakebusiness.com |
| prod | app.fakebusiness.com | api.fakebusiness.com |
Easy, peesy. Log in at https://fakebusiness.awsapps.com/start and select the account you want to access.
This one requires a little setup. Follow this :
- install the
awscliwithbrew install awscli. - Setup your ~/.aws/config with profiles that look like this
note: you may not have
AdministratorAccess, in which case, make more important friends.
; Fake Business
[profile fakebusiness-production]
sso_start_url = https://fakebusiness.awsapps.com/start
sso_region = us-east-1
sso_account_id = 767398123958
sso_role_name = AdministratorAccess
region = us-east-1
output = json
[profile fakebusiness-staging]
sso_start_url = https://fakebusiness.awsapps.com/start
sso_region = us-east-1
sso_account_id = 730335234660
sso_role_name = AdministratorAccess
region = us-east-1
output = json
[profile fakebusiness-dev]
sso_start_url = https://fakebusiness.awsapps.com/start
sso_region = us-east-1
sso_account_id = 891377140782
sso_role_name = AdministratorAccess
region = us-east-1
output = json- next, going to add a snippet to our
~/.zshrcfile that will make it buttery smooth to login with SSO.
function aws-login {
profile="$1";
if [ -z "$profile" ]; then
echo "Profile is required as first argument.";
return
fi
echo "Using SSO account"
aws sso login --profile "$profile";
eval "$(aws configure export-credentials --profile "$profile" --format env)";
export AWS_PROFILE=$profile;
echo "AWS_PROFILE set to $profile"
}
- Install
kafkawithbrewbrew install kafka wget - Download the IAM MSK Auth magic
cd /opt/homebrew/opt/kafka/libexec/libs
wget https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar
- add the
client.propertiesfile to wherever you are going to run your commands from. I keep it right here. - Connect to the VPN
- Auth to AWS using
aws-login - run kafka commands referencing the config. Here is an example. Notice the bootstrap server is per env :
kafka-topics --bootstrap-server boot-s1wuzf1s.c2.kafka-serverless.us-east-1.amazonaws.com:9098 --command-config client.properties --create --topic files --partitions 2
Now, when you want to log in to an AWS account, use aws-login <profile-name> aka aws-login fakebusiness-dev. Website will ask you to verify and sign-in.
- New branch from
main, ideally named from Github Issue like#20/Do-the-thingor with ownership likeUsername/#20 - Make changes, creating a draft Pull Request for easy collaboration without triggering any Github Deployment Actions.
- When you are ready to party, undraft. Automated workflows including testing, linting and validations will run and complete (hopefully).
- Github Actions will deploy a temporary environment for your Pull Request. Subsequent updates to the Pull Request will be deployed to this infrastructure. That Github Action lives here.
- After reviewing the code changes and the preview URL as needed, approve the code, and merge the Pull Request to
main. - With the merge to
mainbranch, the dev environment is updated. That Github Workflow lives here. - Whatever the reason, you create a new Release in Github with proper semantic versioning, and mark it as a
prerelease. This deploys the staging environment. - QA efforts prove uneventful. Perfect. Sounds like Production o'clock. In Github, edit the release and uncheck the
prereleasebox and Save. - Apply sunscreen, unfold beach chair, and bask in orange glow of a new feature making it all the way to production
This application leverages a few ways to manage configurations and secrets.
- Github Project Secrets - There are three secret values that are only used in the CI pipelines in github. They include AWS Credentials and a Slack Webhook URL. Use Github Secrets for all sensitive values. They are easy to inject into the CI pipelines using the
${{ secrets.WHATEVER_THE_SECRET_KEY_IS }} - SSM - The ECS Task Definitions use SSM to pass the Database URL and S3 application bucket from the Terraform CI process.
All of the default configurations for Terraform live in variables.tf in this repository. Changing these values may impact every environment on apply. In addition to the defaults, you can override specific env vars in the CI process in the various Github workflows. They are prefixed with TF_VAR so they are easy to distinguish.
- name: Terraform Apply
id: apply
env:
TF_WORKSPACE: staging
TF_VAR_db_postgres_version: "15.3"
TF_VAR_db_parameter_group: "default.postgres15"
TF_VAR_db_instance_class: "db.t3.micro"
TF_VAR_slack_webhook_url: ""
run: terraform apply -no-color --auto-approve
continue-on-error: false
Secrets for the client are managed in the CI process, because they are needed at build time in each environment. Here is an example of environment variables at the Job level in Github Actions.
build:
name: Build and Deploy Review Environment
needs: test
if: ${{ ! github.event.pull_request.draft }}
runs-on: ubuntu-latest
concurrency:
group: ${{ github.ref }}-build
cancel-in-progress: true
permissions:
contents: read
pull-requests: write
id-token: write
env:
REACT_APP_ALIX_GRAPHQL_API: "https://dev.api.fakebusiness.com/graphql"
REACT_APP_AWS_REGION: us-east-1
REACT_APP_COGNITO_USERPOOL_ID: us-east-1_ngIaKZuB2
REACT_APP_COGNITO_CLIENT_ID: 3h6ron68mkv94hd8bok2fno93j
CI: false
TSC_COMPILE_ON_ERROR: true
ESLINT_NO_DEV_ERRORS: true
Secrets and environment variables are handled in the ECS Deployer configurations. They leverage both plain-text and SSM values.
environment:
SOMEVARNAME: foobar
APP_BUCKET:
ssm: /fakebusiness/staging/APP_BUCKET
DATABASE_URL:
ssm: /fakebusiness/staging/DATABASE_URL
Assuming you started with this template, here is what is recommended to do next:
- Install
pre-commitwithpre-commit install. - Update the information in the
provider.tf. You will need to add the information about state, as well as default tags. To attain Terraform State bucket configurations, ask a DevOps Engineer, or create your own bucket. - Initiate the Terraform state with
terraform init. - Start adding resources to
main.tf, variables tovariables.tf, Add relevant outputs tooutputs.tfand so forth. - Plan your changes with
terraform plan. - Commit your changes. The
pre-commitprocess will lint your code for you as part of the commit process.
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| availability_zones | List of availability zones in the region | map(string) |
{ |
no |
| aws_accounts | list of aws accounts | list(string) |
[ |
no |
| conduktor_admin_password | Admin password for Conduktor service | string |
n/a | yes |
| datadog_api_key | Datadog API Key | string |
n/a | yes |
| db_instance_class | Size and Class for the RDS Postgres instance | string |
"db.t3.micro" |
no |
| db_password | database password | string |
n/a | yes |
| db_postgres_version | Postgres database version | string |
"16.2" |
no |
| db_storage | Postgres allocated storage | number |
20 |
no |
| google_client_id | Google Client ID for SSO Cognito App | string |
n/a | yes |
| google_client_secret | Google Client Secret for SSO Cognito App | string |
n/a | yes |
| kafka_instance_class | MSK Kafka instance class | string |
"kafka.t3.small" |
no |
| project | Identifier for the project | string |
"fakebusiness" |
no |
| region | AWS Region to deploy resources in | string |
"us-east-1" |
no |
| rudderstack_data_plane_url | Rudderstack Data Plane URL | string |
n/a | yes |
| rudderstack_write_key | Rudderstack Write Key | string |
n/a | yes |
| sentry_auth_token | Sentry Auth Token | string |
n/a | yes |
| slack_webhook | Slack Webhook URL for Cloudwatch Alerts | string |
"https://hooks.slack.com/services/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" |
no |
| twingate_token | Twingate Network Access Token | string |
n/a | yes |
| Name | Description |
|---|---|
| acm_cert_arn | some description |
| api_url | API URL |
| app_url | Web App URL |
| assets_bucket_name | Name of the assets bucket |
| assets_url | Assets URL |
| bootstrap_brokers_iam | TLS connection host:port pairs |
| cognito_client_id | ID of the Cognito Client |
| cognito_user_pool_id | ID of the Cognito User Pool |
| conduktor_url | Conduktor URL |
| database_url | URL to the RDS postgres database |
| document_bucket_arn | ARN of the document bucket |
| document_bucket_name | Name of the document bucket |
| domain | Domain for the project environment |
| email_bucket | SES Email storage bucket ARN |
| email_bucket_name | Name of the email bucket |
| github_oidc_role_arn | Github OIDC Role ARN |
| kafka_arn | ARN of the Serverless Managed Kafka |
| kafka_bootstrap_servers | Bootstrap Servers of the Serverless Managed Kafka |
| nat_gateway_ip | NAT Gateway IP address |
| route53_nameservers | some description |
| route53_zone_id | some description |
| ses_arn | ARN for the SES domain |
| site_bucket_name | Name of the marketing site bucket |
| site_url | Website URL |
| zookeeper_connect_string | Kafka Zookeeper connection string |
| Name | Version |
|---|---|
| archive | 2.4.2 |
| aws | 5.53.0 |
| twingate | 3.0.4 |
| Name | Source | Version |
|---|---|---|
| twingate | ./modules/twingate-connector | n/a |
| Name | Description | Value | Sensitive |
|---|---|---|---|
| acm_cert_arn | some description | "arn:aws:acm:us-east-1:891377140782:certificate/c78aa62c-b775-4672-8c88-c39c84563b12" |
no |
| api_url | API URL | "https://api.dev.fakebusiness.com" |
no |
| app_bucket_name | Name of the web app bucket | "fakebusiness-app-dev" |
no |
| app_url | API URL | "https://app.dev.fakebusiness.com" |
no |
| cognito_client_id | ID of the Cognito Client | "37bev9vimuakocf45d4mrdscus" |
no |
| cognito_user_pool_id | ID of the Cognito User Pool | "us-east-1_Lj117fjcj" |
no |
| database_url | URL to the RDS postgres database | "postgres://fakebusiness:PASSWORD@fakebusiness-dev.cfcess626hnv.us-east-1.rds.amazonaws.com:5432/fakebusiness" |
no |
| document_bucket_arn | ARN of the document bucket | "arn:aws:s3:::fakebusiness-documents-dev" |
no |
| document_bucket_name | Name of the document bucket | "fakebusiness-documents-dev" |
no |
| domain | Domain for the project environment | "dev.fakebusiness.com" |
no |
| github_oidc_role_arn | Github OIDC Role ARN | "arn:aws:iam::891377140782:role/fakebusiness-github-actions" |
no |
| kafka_arn | ARN of the Serverless Managed Kafka | "arn:aws:kafka:us-east-1:891377140782:cluster/fakebusiness-main-cluster/74e12fed-5943-4e44-8077-9518f13623a4-s2" |
no |
| kafka_bootstrap_servers | Bootstrap Servers of the Serverless Managed Kafka | "boot-8a2ogjqa.c2.kafka-serverless.us-east-1.amazonaws.com:9098" |
no |
| nat_gateway_ip | NAT Gateway IP address | "3.211.218.198" |
no |
| route53_nameservers | some description | [ |
no |
| route53_zone_id | some description | "Z090713836UUDSL1B9HD2" |
no |
| secrets_arn | Environment secrets ARN | "arn:aws:secretsmanager:us-east-1:891377140782:secret:environment-6r5ylb" |
no |
| site_bucket_name | Name of the marketing site bucket | "fakebusiness-website-dev" |
no |
| site_url | API URL | "https://site.dev.fakebusiness.com" |
no |
| Name | Description | Value | Sensitive |
|---|---|---|---|
| acm_cert_arn | some description | "arn:aws:acm:us-east-1:730335234660:certificate/483de23f-a3f9-4970-9175-564366cea321" |
no |
| api_url | API URL | "https://api.staging.fakebusiness.com" |
no |
| app_bucket_name | Name of the web app bucket | "fakebusiness-app-staging" |
no |
| app_url | API URL | "https://app.staging.fakebusiness.com" |
no |
| cognito_client_id | ID of the Cognito Client | "7j4ti4pd87o5sbushpo1f3re9h" |
no |
| cognito_user_pool_id | ID of the Cognito User Pool | "us-east-1_h9aygNjAA" |
no |
| database_url | URL to the RDS postgres database | "postgres://fakebusiness:PASSWORD@fakebusiness-staging.c90as0mekgcf.us-east-1.rds.amazonaws.com:5432/fakebusiness" |
no |
| document_bucket_arn | ARN of the document bucket | "arn:aws:s3:::fakebusiness-documents-staging" |
no |
| document_bucket_name | Name of the document bucket | "fakebusiness-documents-staging" |
no |
| domain | Domain for the project environment | "staging.fakebusiness.com" |
no |
| github_oidc_role_arn | Github OIDC Role ARN | "arn:aws:iam::730335234660:role/fakebusiness-github-actions" |
no |
| kafka_arn | ARN of the Serverless Managed Kafka | "arn:aws:kafka:us-east-1:730335234660:cluster/fakebusiness-main-cluster/af976502-e746-46ea-b95c-237dacf29841-s1" |
no |
| kafka_bootstrap_servers | Bootstrap Servers of the Serverless Managed Kafka | "boot-kxkgk87y.c1.kafka-serverless.us-east-1.amazonaws.com:9098" |
no |
| nat_gateway_ip | NAT Gateway IP address | "54.82.22.34" |
no |
| route53_nameservers | some description | [ |
no |
| route53_zone_id | some description | "Z024952234IC03XF20G4E" |
no |
| secrets_arn | Environment secrets ARN | "arn:aws:secretsmanager:us-east-1:730335234660:secret:environment-O22usk" |
no |
| site_bucket_name | Name of the marketing site bucket | "fakebusiness-website-staging" |
no |
| site_url | API URL | "https://site.staging.fakebusiness.com" |
no |
| Name | Description | Value | Sensitive |
|---|---|---|---|
| acm_cert_arn | some description | "arn:aws:acm:us-east-1:767398123958:certificate/35ebd919-d1d6-4e5a-b1ac-a5271e8f7893" |
no |
| api_url | API URL | "https://api.fakebusiness.com" |
no |
| app_bucket_name | Name of the web app bucket | "fakebusiness-app-production" |
no |
| app_url | API URL | "https://app.fakebusiness.com" |
no |
| cognito_client_id | ID of the Cognito Client | "2eadc1pgk4vp5ajamjv81j1jqh" |
no |
| cognito_user_pool_id | ID of the Cognito User Pool | "us-east-1_tOrfZUGZP" |
no |
| database_url | URL to the RDS postgres database | "postgres://fakebusiness:PASSWORD@fakebusiness-production.cpe8q4q6esi0.us-east-1.rds.amazonaws.com:5432/fakebusiness" |
no |
| document_bucket_arn | ARN of the document bucket | "arn:aws:s3:::fakebusiness-documents-production" |
no |
| document_bucket_name | Name of the document bucket | "fakebusiness-documents-production" |
no |
| domain | Domain for the project environment | "fakebusiness.com" |
no |
| github_oidc_role_arn | Github OIDC Role ARN | "arn:aws:iam::767398123958:role/fakebusiness-github-actions" |
no |
| kafka_arn | ARN of the Serverless Managed Kafka | "arn:aws:kafka:us-east-1:767398123958:cluster/fakebusiness-main-cluster/81e9a238-d229-4d44-b609-5933d287c0df-s3" |
no |
| kafka_bootstrap_servers | Bootstrap Servers of the Serverless Managed Kafka | "boot-2zpjjxeb.c3.kafka-serverless.us-east-1.amazonaws.com:9098" |
no |
| nat_gateway_ip | NAT Gateway IP address | "44.209.160.13" |
no |
| route53_nameservers | some description | [ |
no |
| route53_zone_id | some description | "Z06393165SX7J27I8MT8" |
no |
| secrets_arn | Environment secrets ARN | "arn:aws:secretsmanager:us-east-1:767398123958:secret:environment-c60Wuz" |
no |
| site_bucket_name | Name of the marketing site bucket | "fakebusiness-website-production" |
no |
| site_url | API URL | "https://site.fakebusiness.com" |
no |