AWS Uploader

This solution hosts the infrastructure to build a website that allows specific users to upload EXTRACT and MANI files inside a secured S3 bucket and deploy it into dev, pre-prod and production environments. This template is designed to help you start an AWS terraform repository in the same structure across all projects.

The solution deploys:

○ Host S3 Bucket: With enabled static website hosting and the appropriate permissions ○ CloudFront Distribution: Linked to the Host S3 Bucket ○ IAM Roles and Policies ○ HTML Scripts: All html pages that will be hosted inside the S3 Bucket ○ Lambda Script: Create "PreSignedURL" script that will be targeting the Ingest Bucket ○ Ingest S3 Bucket: Creating the ingest bucket where the files will be uploaded and set the appropriate permissions ○ Set up Route 53 DNS: Generates unique URL for each council

cicd

Image needs updating but shows how the flow will work expect all pull requests against main will apply in dev.

Pre Commits

This repository makes use of https://github.com/gruntwork-io/pre-commit to help enforce code quality. Useful guide https://medium.com/slalom-build/pre-commit-hooks-for-terraform-9356ee6db882

Prerequisites

You will need to install the following:

• Pre Commit - brew install pre-commit
• Terraform - brew tap hashicorp/tap, brew install hashicorp/tap/terraform
• TFSec - brew install tfsec
• Terraform Docs - brew install terraform-docs
• Pre Commit Hooks - pip install pre-commit-hooks
• Tflint - brew install tflint

Usage

Once you have met all the prerequisites install pre commit on the reposiotry

pre-commit install

Configuration for the pre commit can be found in .pre-commit-config.yaml

Pre commit is doing:

• terraform-validate - ensuring any terraform code which is being committed is valid
• terraform fmt - formatting any terraform which is being committed.
• terraform-docs - Used to automatically generate documentation for this repo
• Tflint - Finds invalid configuration with AWS Terraform, enforces agreed apon best practices, warns about deprecated syntax
• Tfsec - Checks for misconfigurations in terraform code against security protocols
• Git checks - checks for large files being commited, merge conflicts, end of file fixes, checks for any potential secrets being committed.

Terraform

Only use this method for testing locally, all deployments should go through your CICD pipeline.

Authenticate with AWS

Log into AWS and select the account required. Select Command line and programmatic access and select option 1.

export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="eu-west-2"
export S3_BUCKET_NAME="YOUR_BUCKET_NAME"

Terraform init

terraform init -backend-config=bucket="${S3_BUCKET_NAME}"
-backend-config=key="tfstate/default.tfstate"
-backend-config=dynamodb_table="lockstable"
-backend-config=region="eu-west-2"

Run terraform plan

terraform plan -var-file=env/env.tfvars

Onboarding New Councils

New councils should be onboarded using the dedicated helper tool located in:

scripts/helpers/onboard_councils/

This tool imports council data from an Excel spreadsheet and updates councils.csv automatically.

It helps to:

• Reduce manual editing of councils.csv
• Prevent formatting inconsistencies and typos
• Apply standardised council naming rules
• Produce a clear audit log of changes made

Quick Start

Execute the following where path/to/input is the location of the input file

cd scripts/helpers/onboard_councils
poetry install
poetry run python onboard_councils_from_xlsx.py /path/to/input 

Full instructions

For detailed setup, input file requirements, logging behaviour, and troubleshooting, see:

scripts/helpers/onboard_councils/README.md

Running Behaviour tests

Behaviour tests should be run before raising a Pull Request. Depending on your package manager of choice you can run

yarn install
yarn test

or

npm install
npm test

or, you can make use of the makefile and run

make behaviour-tests

after installing the package dependencies.

Github Automatic reviewers

The template is setup to use codeowners, this is setup within the .github folder within the CODEOWNERS file. By default it is set to the CIA team, to update this change or add another github team.

Dependabot

Dependabot is setup to check dependency versions within terraform, this will automatically open a PR every Monday for any providers which are out of date.

CICD

Concourse uses YAML to create the pipelines, which is works well until you start to create bigger pipelines to support bigger environments. Within the ci folder there are two examples aviator and concourse. Read the README in both sections to work out which is better for your usecase.

Maintenance Mode

The service can be put into maintenance mode to prevent access during system maintenance. A Lambda@Edge function checks the maintenance status and redirects users to a maintenance page.

Authenticate with the AWS account user credentials

Activate with custom message and date:

aws ssm put-parameter --name "/uploader/maintenance-mode" --value '{"enabled":true,"message":"2 pm on Monday 16 December 2025"}' --type String --overwrite

In this case "2 pm on Monday 16 December 2025"" is an example - put whatever message you want in.

Deactivate maintenance mode:

aws ssm put-parameter --name "/uploader/maintenance-mode" --value "false" --type String --overwrite

Changes take effect within 60 seconds due to Lambda@Edge caching. However, you can invalidate the cache instantluy with the below commands.

List cloudfront cache:

aws cloudfront list-distributions

Invalidate the cache:

aws cloudfront create-invalidation --distribution-id <distribution_Id> --paths "/council-tax/*"

Monitoring

The deployed production solution is monitored by uptrends.com. There are two monitors in place

• Connectivity to https://uploader.ingest.aws.onsdigital.uk/council-tax/H00000000-Health-Status.html
• The presence of a fixed string in the page is also tested for - this is currently "Upload the EXTRACT file".

CSS control the uptrends system. Chris U. also has a logon.

Requirements

Name	Version
terraform	>= 1.8.0
terraform	>= 1.8.0
archive	>= 2.7.0
aws	>= 5.94.1
null	>= 3.2.0

Providers

Name	Version
archive	2.7.1
aws	5.99.1
aws.useast	5.99.1
null	3.2.4
terraform	n/a

Modules

Name	Source	Version
ons_upload_bucket	git::https://github.com/ONSdigital/aws-s3-bucket.git	v7.4.0
ons_upload_ingest_bucket	git::https://github.com/ONSdigital/aws-s3-bucket.git	v7.4.0
render_council	./modules/render_council	n/a

Resources

Name	Type
aws_acm_certificate.uploader	resource
aws_acm_certificate_validation.cert	resource
aws_apigatewayv2_api.api	resource
aws_apigatewayv2_integration.get	resource
aws_apigatewayv2_integration.options	resource
aws_apigatewayv2_route.get	resource
aws_apigatewayv2_route.options	resource
aws_apigatewayv2_stage.api	resource
aws_athena_database.access_logs	resource
aws_athena_workgroup.access_logs	resource
aws_cloudfront_distribution.uploader	resource
aws_cloudfront_function.rewrite_default_index_request	resource
aws_cloudfront_origin_access_control.ons_uploader_cloudfront	resource
aws_cloudfront_origin_access_identity.oai	resource
aws_cloudfront_response_headers_policy.custom_security_headers	resource
aws_cloudwatch_log_group.api_gateway_log_group	resource
aws_cloudwatch_log_group.cloudfront_log_group	resource
aws_cloudwatch_log_group.lambda_log_group	resource
aws_iam_policy.PreSignedURL_s3_policy	resource
aws_iam_role.PreSignedURL_role	resource
aws_iam_role.cloudwatch_global	resource
aws_iam_role_policy_attachment.PreSignedURL_s3_policy	resource
aws_iam_role_policy_attachment.lambda_basic	resource
aws_lambda_function.PreSignedURL	resource
aws_lambda_permission.presignedurl_permission	resource
aws_route53_record.cert_validation	resource
aws_route53_record.uploader	resource
aws_s3_bucket.cloudfront_logging_bucket	resource
aws_s3_bucket_acl.cloudfront	resource
aws_s3_bucket_cors_configuration.uploader	resource
aws_s3_bucket_lifecycle_configuration.ingest_lifecycle_policy	resource
aws_s3_bucket_ownership_controls.cloudfront_logging_bucket	resource
aws_s3_bucket_policy.uploader_bucket	resource
aws_s3_bucket_policy.uploader_ingest_bucket	resource
aws_s3_bucket_public_access_block.cloudfront_logging_bucket	resource
aws_s3_bucket_server_side_encryption_configuration.cloudfront_logging_bucket	resource
aws_s3_bucket_website_configuration.ons_upload_configuration	resource
aws_s3_object.council_home_page	resource
aws_s3_object.file_submission	resource
aws_s3_object.home_page	resource
aws_s3_object.result_message	resource
aws_s3_object.success_page	resource
aws_wafv2_web_acl.uploader_waf_cloudfront	resource
null_resource.execute_query	resource
terraform_data.invalidate_cf_caches	resource
archive_file.PreSignedURL	data source
aws_caller_identity.current	data source
aws_canonical_user_id.current	data source
aws_cloudfront_log_delivery_canonical_user_id.cloudfront	data source
aws_iam_policy_document.get_s3_object	data source
aws_iam_policy_document.lambda_role	data source
aws_iam_policy_document.uploader_bucket	data source
aws_iam_policy_document.uploader_ingest_bucket	data source
aws_region.current	data source
aws_route53_zone.domain	data source
aws_sqs_queue.nifi_sqs	data source

Inputs

Name	Description	Type	Default	Required
api_gateway_cloudwatch	name_for_api_gateway_cloudwatch_group	string	"api_gateway_cloudwatch"	no
cloudfront_logging_bucket	Bucket for logging cloudfront distribution	string	n/a	yes
cloudwatch_retention_days	number of days to retain cloudwatch logs	string	365	no
domain_name	Domain name for the DNS within an account to use.	string	n/a	yes
lambda_PreSignedURL_function	lambda name for the PreSignedURL function	string	"PreSignedURL"	no
region	Region in which to create resources	string	"eu-west-2"	no
sqs_notification_id	sqs_notification_id	string	n/a	yes
target_account_id	Target account ID you wish to deploy to	string	n/a	yes
upload_host_bucket_name	Hosting the html for ONS Uploader webapp	string	n/a	yes
upload_ingest_bucket_name	Bucket for ingesting files	string	n/a	yes

Outputs

Name	Description
website_domain	domain name for the cloudfront website