metoffice_ec2

Extract specific parts of the UK Met Office's UKV and MOGREPS-UK numerical weather predictions from AWS, compress, and save to S3 as Zarr. Intended to run on AWS EC2.

Install & test locally

conda env create -f environment.yml 
conda activate metoffice_ec2
pip install -e .
py.test -s

If boto3 is setup to access AWS, then you can run scripts/ec2.py from your local machine to test (although it'll try to pull large amounts of data into & out of S3, so this will get expensive quickly!)

Build & test Docker container locally

Install Docker

docker build .
docker run -e AWS_ACCESS_KEY_ID=<ID> -e AWS_SECRET_ACCESS_KEY=<KEY> -e DEST_BUCKET=<BUCKET> -e SQS_URL=<SQS_URL> <DockerImage>

Install on AWS

A Terraform template is provided in /infrastructure. A Docker Image is also provided publicly.

> More Information about Infrastructure

The following environment variables must be provided:

Name	Type	Description
SQS_URL	String	The URL of the SQS that the Messages are consumed from
DEST_BUCKET	String	S3 bucket that the finished output should be stored in

Manual Setup

Configure AWS permissions

Go to the AWS Identity and Access Management (IAM) console... TODO

Create bucket for storing NWPs

Create a bucket for storing subsetted NWPs. Set the DEST_BUCKET constant in scripts/ec2.py

Configure AWS Simple Queue Service (SQS)

When the Met Office uploads new NWPs to S3, they also send a message to an AWS Simple Notification Service topic. These notifications must be received as soon as they're produced. But our EC2 job isn't kept running 24/7. Our EC2 job is triggered once an hour. So we need a way to capture the SNS notifications when our EC2 job is offline.

A solution is to set up an AWS Simple Queue Service. Set up SQS as per the Met Office's instructions.

Then set the SQS_URL in scripts/ec2.py

Configure EC2 instance

Configure EC2 instance to trigger every hour

Software Development

This code follows the Google Python Style Guide.

The included environment.yml file includes just the packages required to run the script and unit tests.

To do development, please run this command within the metoffice_ec2 conda environment:

conda install black flake8 isort jedi mypy

Format the code with black metoffice_ec2 scripts.

Check coding style with flake8.

Fix import order with isort ..

Run static type checking with mypy metoffice_ec2 scripts.

jedi is for auto-completion (and other things) in IDEs.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Jack Kelly 💻

Flo 💻

Tom White 💻

This project follows the all-contributors specification. Contributions of any kind welcome!