Johanna

Johanna is a collection of boilerplate Python scripts that can do provisioning/deprovisioning of a simple backend system using AWS.

The backend includes below:

• VPC with two public subnets, two private subnets, routing tables, an internet gateway, a nat gateway and an EIP.
• IAM roles for Elastic Beanstalk
• EC2 key pair (SSH key)
• An Elastic Beanstalk application and an environment for Python Django API server
• An aurora RDS cluster with instances
• An sample SQS
• and more...

You can do provisioning/deprovisioning/reprovisioning of the whole system or partial at once. Especially, the reprovisioning of Django API server means a 'continuous deployement'.

Requirements

- Vagrant 2.2.13+

[For using hbsmith/awslinux2 box] 
- Parallels Desktop 16+ for Mac 
- vagrant-parallels plugin 2.0.1+

How To Play

Using Lili(Vagrant provisioning script) is the simplest way to get a playground.

• Follow Lili README manual
• On Vagrant VM (Ubuntu 16.04)

  1. sudo su

  2. cd /opt/johanna

  3. Execute conf.py to configure your aws environment.

     ./conf.py --email YOUR_EMAIL --keypairname YOUR_AWS_KEYPAIR_NAME --accesskey YOUR_AWS_ACCESSKEY --secretkey YOUR_AWS_SECRETKEY --region AWS_REGION_NAME --az1 AVAILABILITY_ZONE_1 --az2 AVAILABILITY_ZONE_2 --template TEMPLATE_GIT_URL --user DB_USER --pw DB_PASSWORD

     [Example]

     ./conf.py --email ... --keypairname ... --accesskey ... --secretkey ... --region ap-northeast-2 --az1 ap-northeast-2a --az2 ap-northeast-2c --template git@github.com:HardBoiledSmith/kerrigan.git --user db-user --pw db-password

  4. ./run.py

You can use this on web GUI

• raynor is web based GUI for johanna

CLI Options

./run_create_eb.py [OPTIONS] <eb-environment-name>		(ex: './run_create_eb.py sachiel')
./run_terminate_eb.py [OPTIONS] <eb-environment-name>	        (ex: './run_terminate_eb.py sachiel')
./run_create_lambda.py [OPTIONS] <lambda-function-name>		(ex: './run_create_eb.py sachiel_send_email')
./run_terminate_lambda.py [OPTIONS] <lambda-function-name>	(ex: './run_terminate_lambda.py sachiel_send_email')
./run_create_s3.py [OPTIONS] <s3-bucket-name>		        (ex: './run_create_eb.py dv-hbsmith-web')
./run_terminate_s3.py [OPTIONS] <s3-bucket-name>		(ex: './run_terminate_s3.py dv-hbsmith-web')
./run.py [OPTIONS] -- [AWS CLI COMMAND]		                (ex: './run.py -- aws ec2 describe-instances')

• --force or -f Attempt to execute the commend without prompting for phase confirmation.
• --branch or -b Attempt to execute the command with specific git branch.
• --region or -r Attempt to execute the command on specific region.

Script to create cloudfront and route 53

• Execute run_create_cloudfront.py to create cloud front

./run_create_cloudfront.py -b <s3 bucket name> -e <s3 bucket end point> -a <acm-arn> -c <cname> -f

• Execute run_create_cloudfront.py to create route53
• reference HOSTED_ZONE_ID : https://docs.aws.amazon.com/Route53/latest/APIReference/API_AliasTarget.html#Route53-Type-AliasTarget-HostedZoneIdx

./run_create_route53.py -ah Z2FDTNDATAQYW2 -at cloudfront -d <cloudfront domain name> -hn hbsmith.io -n <domain> -r A -f

Script to create route 53 health check

• Full deployment: ./run_create_route53_health_check.py
• Partial deployment: ./run_create_route53_health_check.py <name>
  • ex) ./run_create_route53_health_check.py ramiel
• Partial termination: ./run_terminate_route53_health_check.py <name>
  • ex) ./run_terminate_route53_health_check.py ramiel

Notes

• If you use AWS IAM user credential instead of master account, it must have IAMFullAccess, AWSElasticBeanstalkFullAccess and PowerUserAccess permissions.

  

Vagrant

• set config.json below root of johanna
• move to provisioning folder using johanna $ cd _provisioning
• copy id_rsa : cp ~/.ssh/id_rsa ./_provisioning/configuration/root/.ssh/
• (optional) To use release feature for sentry, must set environment value: SENTRY_AUTH_TOKEN and SENTRY_ORG
  • https://docs.sentry.io/product/releases/suspect-commits/#using-the-cli
• create or run vagrant using johanna/_provisioning $ vagrant up
• (optional) Run BRANCH=<branch name> vagrant up for provisioning with specific git branch.
• connect to vagrant using $ ssh root@dv-johanna.hbsmith.io or $ ssh root@192.168.124.5
• move to johanna folder using $ cd /opt/johanna
• run provisioning script using /opt/johanna $ ./run.py

How to run Lint check (PEP8)

1. Provisioning vagrant
2. Connect http://dv-johanna.hbsmith.io/
3. Go to cd /opt/johanna
4. Run flake8 --config=flake8 .

Command Completion

You can use AWS CLI Command Completion

AWS Codebuild

Before create codebuild projects, you must create these resources manually using AWS Web console:

• Environment Variables (Secure String) at Systems Manager > Parameter Store

  https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-paramstore-securestring.html

• OAuth Connection to GitHub by follow instructions

  For source code in a GitHub repository, the HTTPS clone URL to the repository that contains the source and the buildspec file. You must connect your AWS account to your GitHub account. Use the AWS CodeBuild console to start creating a build project. When you use the console to connect (or reconnect) with GitHub, on the GitHub Authorize application page, for Organization access , choose Request access next to each repository you want to allow AWS CodeBuild to have access to, and then choose Authorize application . (After you have connected to your GitHub account, you do not need to finish creating the build project. You can leave the AWS CodeBuild console.)
  https://docs.aws.amazon.com/cli/latest/reference/codebuild/create-project.html#options

AWS Client VPN

You can create client configuration (.ovpn) for AWS Client VPN

1. Provision VPC, AWS SES into your account
2. Provision Client VPN Endpoint into the VPC
3. SSH into vagrant johanna
4. Go to cd /opt/johanna
5. Run ./run_export_client_vpn_ovpn.py <client vpn name> <region> <email to> <zip password>
6. Check email inbox of <email to>

Troubleshooting - ACM quotas

"You have reached the maximum number of certificates. Delete certificates that are not in use, or contact AWS Support to request an increase."

By default, you can import up to 1000 certificates into ACM, but new AWS accounts might start with a lower limit. If you exceed this limit, request an ACM quota increase with these. You can solve this issue by opening support case.

• AWS Certificate Manager (ACM) > Imported certificates in last 365 days
• AWS Certificate Manager (ACM) > ACM certificates created in last 365 days

How to set up fake email test environment

1. Provision AWS VPC, RDS, Client VPN, Elastic Beanstalk, SQS, AWS SES, SNS, S3 into your account
2. Create vagrant gendo, johanna.
3. SSH into vagrant johanna
4. Go to cd /opt/johanna
5. run that command ./run.py create_vpc
6. run that command ./run.py create_rds
7. run that command ./run_create_ec2_client_vpn.py connect to the self_service address shown in the last line to download the ovpn file.
8. Reset the database data in the created aws rds
9. run that command ./run_create_eb.py sachiel
10. run that command ./run.py create_sqs
11. run that command ./run_create_s3.py <your dev env name>-dv-hbsmith-email-receive
12. run that command ./run_create_s3.py test
13. run that command ./run.py create_sns
14. run that command ./run.py create_ses
15. Go to the company email and click the link to agree to the verify email received from AWS SNS and SES.
16. run that command ./run_create_lambda.py sachiel_fake_email_save
17. run that command ./run_create_lambda.py sachiel_fake_email_delete
18. (OPTIONAL) run that command to deploy ramiel:
    • To all on-premise instances: ./run.py create_codedeploy_ramiel or ./run_create_codedeploy_ramiel.py
    • To specific on-premise instance(s): ./run_create_codedeploy_ramiel.py <comma separated instance hostname(s)>
19. Please do one of the two methods below.
    1. Case 1)
       1. run that command ./run_create_lambda.py gendo_test_fake_mail
       2. Connect to AWS console and run lambda service gendo_test_fake_mail (Seoul Region)
    2. Case 2)
       1. Log in to the mail service and send an email to the email address you want to test (XXXX@hbsmith-email.io).
20. Go to the naoko project and set the http://dv-sachiel.hbsmith.io value in the code at https://github.com/HardBoiledSmith/naoko/blob/4e1c7aedbb8856bf804910a9ddae573310681097/naoko/Utility/SettingsManager.cs#L43 to https:// Change to {your dev env name}-dv-sachiel.hbsmith.io"; and build.
21. run the naoko program. set the URL to -dv-test.hbsmith.io and start recording.
22. Go to the Received Email tab on the test page and follow the test procedure.
23. Follow the guide and enter the email value as referenced below.
    1. If you selected Case1 in Step 18, enter test_gendo@hbsmith-email.io in the Email field.
    2. If you selected Case2 in Step 18, please enter the email address you sent.

Links

• PyCon APAC 2016
• Slideshare