Cloud Security Watch
(csw-backend) This repository contains the code for creating an instance of Cloud Security Watch running in an infrastructure created by alphagov/csw-infra
Note - Cloud Security Watch audits can now be run offline
from the command line. If you want to run CSW audits without a web infrastructure please read the CLI ReadMe
Create your virtual env
TODO - switch to venv
If you're running Python 2.7 or 3.7 you will need to install 3.6.5
brew install pyenv pyenv install 3.6.5 virtualenv -p <path to your 3.6.5/bin/python> <virtual env name>
Then you'll need to install the dependencies and dev dependencies locally
cd chalice pip install -r requirements-dev.txt
The requirements file is not stored in the repository root since the requirements need to be packaged by chalice as part of the chalice deploy
Now you can run the unit tests.
python -m unittest discover -v
Run the command above before adding/commiting. If tests fail, don't worry, review the reason for the failures. It might even be that the test is no longer valid, in which case you need to update it or delete it.
End to end testing
If you are running the e2e tests for the first time you probably need to do some installs first.
brew install geckodriver cd /path/to/csw-backend/chalice pip install -r requirements-dev.txt cd ../build reinstall gulp environment.terraform --env=[env] gulp environment.chalice_s3_deploy --env=[env]
For the e2e tests in addition to your env you need to tell it which user to assume. This is the name part of your email address.
gulp environment.e2e --env=[env] --user=[firstname.surname]
Create your AWS environment
Google Cloud Console credentials file.Create a
You will only need to do this in an AWS account where this has not already been set up.
####Create an SSH key. The build script uses ssh to tunnel to the RDS instance and create the database. It also creates a box which can be used for deploying chalice when working remotely with less bandwidth.
The simplest approach is probably to match the env name. The key will be uploaded to AWS so matching the env name avoids conflict and makes it easy to find.
Like id_rsa, the difference between the
private and public key file names should be:
Install gulp-cli globally
sudo npm install -g gulp-cli
Gulp is a task runner used to run various build tasks.
Install npm-reinstall globally
sudo npm install -g npm-reinstall
Does a complete reinstall of npm dependencies not via the cache. Important for picking up github code changes if you're installing a branch rather than a tagged release and want the current commit on that branch.
If you're running in a non-default AWS account where access is assumed via a profile you will need to set up aws-vault so you can assume the profile to run these commands.
If you're running against the AWS account attached to your default credentials and don't use aws-vault you can just run the commands from gulp onwards.
.. becomes ..
aws-vault exec [profile] -- gulp [task]
From here onwards
aws-vault exec [profile] has been left out
but it's assumed that any
gulp task should be preceded
with an appropriate
Building your environment
All the build tasks are run from the build directory
NPM install installs;
- alphagov/govuk-frontend and its dependencies
- gulp and some modules for running buid tasks
- alphagov/csw-infra to terraform the infrastructure
Upload shared parameters
Create shared credentials in parameter store. These are the Google API OAuth credentials and the name of the S3 bucket used to store the terraform states.
These are created once in each AWS account and reused for multiple environments. The assumption is that the live environment will be the only env deployed to the live account. So credentials are shared by test environments but different credentials are used in production.
If you're creating a new environment in an account that is already running an existing environment you can skip this step.
Environments can be test stages or named for individual developers. Names should be short, lower case with no spaces.
aws-vault exec [profile] -- gulp environment.build --env=[env name]
environment.build task is made up of several
Especially when building test or developer accounts you
may run into AWS Service Limits. In most cases these
will be triggered in the
terraform apply task.
If terraform fails the output from the script should show the limit which needs increasing and this can be done through an AWS support request.
You will first be prompted for some settings:
- A 16bit mask for your internal IP ranges.
- 10.x for developers
- 10.10x for test environments
- The name of your ssh key and the path to the public key. This will create an ssh key on AWS and upload your public key.
If you've followed the process above the ssh key name
should match your env name. It asks for the path to the
public part of the key
If you put your private key path in this setting you will need to generate a new ssh key and start again
It will create a
settings.json file for your
Make sure you have a
.ssh/config file in your home
directory. If it's not there just use
touch to create
an empty file.
It then generates some random passwords for RDS and uploads them to AWS parameter store.
Then it creates your terraform tfvars files.
Then it initialises terraform with S3 (For an existing environment it will retrieve the existing environment state file)
terraform apply to build your
infrastructure and saves the terraform
outputs to your
environment.terraform task has sub-tasks
which can all be run independently as necessary
Copies assets from
Then it creates a chalice
chalice deploy --stage=[env]
environment.chalice task has sub-tasks which
can all be run independently as necessary
Finally it creates and bootstraps the database, creating the tables and populating the lookup content and checks.
Loading an existing environment
- Creates your settings file and tfvars files
- Reinitialises terraform from S3
- Reads terraform output into the settings file
- Creates the chalice config
gulp environment.load --env=[env]
Deploy an existing environment
- Compiles SASS and copies assets from
- Runs terraform apply
- Runs chalice deploy
gulp environment.deploy --env=[env]
Alternatively you can run ..
gulp environment.terraform_apply --env=[env]
.. and/or ..
gulp environment.chalice_deploy --env=[env]
Delete and existing environment
- Runs chalice delete
- Runs terraform destroy
gulp environment.cleanup --env=[env]