Environmental Outcomes Platform (EOP) Infrastructure

This repository contains code to deploy the EOP infrastructure across all live environments in AWS. All the infrastructure in this repo is managed as code and deployed via merging code changes into the main branch of this repo. As such there should be few times that changes need to be made via click-ops in the AWS console, other than helping with development.

This code is built from the Gruntwork Reference Architecture using services from the Gruntwork AWS Service Catalog. There is supporting documentation in the ./docs folder of this repo, which was initially provided by Gruntworks during the reference architecture setup, which is useful but quite generic, it will be updated over time to be more specific to EOP.

Here's a diagram that shows a rough overview of what the Reference Architecture looks like:

Getting Started

This section is for people new to the EOP team looking to work in or support the AWS infrastructure.

AWS Account

First you need an IAM user in AWS that provides access to. We do this by creating a IAM user in our "security" account which then has access to assume roles in the other accounts.

• Create a PR which adds a user to this file security/_global/account-baseline/users.yml with the appropriate permissions
• Once the PR is merged and deployed. An admin user will need to create a temporary password for you, set to force a new one to be created on next login
• Once you have this:
  1. login to here with the temp password and
  2. Reset the password
  3. Enable MFA

Enabling MFA is required in the Reference Architecture. Without MFA, you will be able to log in, but will not be able to access anything!

You will now be logged into the security account, to access the other accounts in EOP is done by "switching role" to the other accounts with specific access.

Roles which are allowed

• allow-read-only-access-from-other-accounts - Provides read only access to most things
• allow-dev-access-from-other-accounts - Provides access intended for developers, the ability to make changes in the common AWS services
• allow-support-access-from-other-accounts - Provides access that can interact with AWS support
• allow-billing-only-access-from-other-accounts - Provides access for people only interested in billing details
• allow-full-access-from-other-accounts - Provides unlimited access
• allow-auto-deploy-from-other-accounts - Intended for machine users, special role which provides access for our auto deployment tasks limiting the scope of what they can do

Note: Exactly what these roles can do is configurable, and will need to be updated as new AWS services are used.

Shortcuts for read-only access to other accounts

• Shared
• Logs
• Dev
• Stage
• Prod

Note: Each of these will let you choose a color for the particular access, green is good for read only. Most of the time read-only access should suffice.

Local Machine Setup

Now you have access to the AWS account, you may want AWS command line access. This would be for making Terraform changes locally or just avoiding clicking around the UI in favour of command line tools.

• Create Access Keys for your account
• Install aws-vault.
• Add your IAM User credentials:

aws-vault add security

• Add new profiles for each of the accounts to ~/.aws/config. This is a generated config, update the and role accordingly

[default]
region=ap-southeast-2

[profile security]
mfa_serial = arn:aws:iam::063810897000:mfa/<IAM User>

[profile eopdev]
source_profile = security
mfa_serial = arn:aws:iam::063810897000:mfa/<IAM User>
role_arn = arn:aws:iam::657968434173:role/allow-full-access-from-other-accounts

[profile eopprod]
source_profile = security
mfa_serial = arn:aws:iam::063810897000:mfa/<IAM User>
role_arn = arn:aws:iam::422253851608:role/allow-full-access-from-other-accounts

[profile eopstage]
source_profile = security
mfa_serial = arn:aws:iam::063810897000:mfa/<IAM User>
role_arn = arn:aws:iam::564180615104:role/allow-full-access-from-other-accounts

[profile logs]
source_profile = security
mfa_serial = arn:aws:iam::063810897000:mfa/<IAM User>
role_arn = arn:aws:iam::972859489186:role/allow-full-access-from-other-accounts

[profile shared]
source_profile = security
mfa_serial = arn:aws:iam::063810897000:mfa/<IAM User>
role_arn = arn:aws:iam::898449181946:role/allow-full-access-from-other-accounts

• Once configured, you can use AWS vault with Terragrunt, Terraform, the AWS CLI, and anything else that uses the AWS SDK to authenticate. To check if your authentication is working, you can run aws sts caller-identity

aws-vault exec eopdev -- aws sts get-caller-identity

• You can also use aws-vault to log in to the web console for each account:

aws-vault login eopdev --duration 8h -s

SSH Bastion

The config of the systems mean that none of the machines running in AWS are directly accessible from the internet. To be able to access them, there is a Bastion host with SSH that is connected to the VPC and has routes enabled to key services (EC2 instances, databases ... )

The Gruntwork config has automatically hardened the bastion host, and made access available for IAM users in the correct group using their AWS SSH keys.

• Add SSH keys to your account
• usernames are changed to avoid special characters e.g. john.doe@gw.govt.nz => john_doe
• ssh into the appropriate bastion. e.g. for dev

ssh john_doe@bastion.gw-eop-dev.tech

Note: for access to database servers this can be done via a SSH tunnel.

Contributing

Changes to the infrastructure are deployed using the trunk based workflow. That is any change to main will trigger a github actions workflow which will run the changes into the live environment.

Basic outline:

1. pull latest code to your machine
2. create a branch from main for your changes
3. make / commit the changes
   1. Push your changes
   2. this will trigger a Github actions job which will plan what changes will be made when the code is merged
   3. Create a “pull request”
4. Validate what will be changed, discuss and review code
5. "Merge" your branch to the main branch
6. Validate changes are applied successfully

Note: Be careful when changing things that are shared across environments. If you need to change something shared, it will probably need to be factored in such a way that it can be applied to a single environment at a time.

Note: removing resources completely can't be done via the Github Actions process

Learn

Gruntwork Reference Docs

• How to Build an End to End Production-Grade Architecture on AWS: A blog post series that discusses the basic principles behind the Reference Architecture.
• How to use the Gruntwork Infrastructure as Code Library: The Service Catalog is built on top of the Gruntwork Infrastructure as Code Library. Check out this guide to learn what the library is and how to use it.
• Gruntwork Production Deployment Guides: Additional step-by-step guides that show you how to go to production on top of AWS.

Support

If you need help with this repo or anything else related to infrastructure or DevOps, Gruntwork offers Commercial Support via Slack, email, and video conference. If you have questions, feel free to email us at support@gruntwork.io.