This folder specifies all relevant components to run this project on AWS. All resources are specified in Terraform. By using Terraform anybody can run this script themselves.
Install the terraform cli: https://learn.hashicorp.com/tutorials/terraform/install-cli.
Initialize terraform by running terraform init in this directory.
The configuration options are specified in variables.tf.
There are several components to the system:
Gets the latest NWP data from the UK Met Office and saves the data to S3. This is run on ECS and is triggered by airflow. This is currently part of the nowcasting project. There are currently two different orders.
- The first gets data for the next 12 hours with the following variables:
dlwrf,dswrf,hcc,lcc,mcc,prate,r,sde,si10,t,vis. - The second gets data for the next 42 hours with the following variables:
dswrflccsdetwdir10
More details:
- Code: nwp-consumer
- Terraform and Airflow Dag
- AWS logs: aws/ecs/consumer/nwp and aws/ecs/consumer/nwp-national
Gets the latest satellite data from the EUMETSAT and saves the data to S3. This is run on ECS and is triggered by airflow. This is currently part of the nowcasting project.
- Code: Satip
- Terraform and Airflow Dag
- AWS logs: aws/ecs/consumer/sat
Gets the latest PV data from the Sheffield Solar API and save to the database. This is run on ECS and is triggered by airflow. This is currently part of the nowcasting project.
- Code: PVConsumer
- Terraform and Airflow Dag
- AWS logs: aws/ecs/consumer/pv
Gets GSP solar generation data from PVlive from Sheffield Solar and save to the database. This is run on ECS and is triggered by airflow. This is currently part of the nowcasting project.
- Code: GSPConsumer
- Terraform and Airflow Dag
- AWS logs: aws/ecs/consumer/gsp
Loads NWP and PV data, and then runs the forecast model. The results are saved to the database. This is run on ECS and is triggered by airflow.
- Code: pv-site-production
- Terraform and Airflow Dag
- AWS logs: aws/ecs/pvsite_forecast/
Pvnet 2 is currently our best forecast from 0 to 8 hours. It is a complex CNN model that used Satellite and NWP. First GSP forecasts are made, and then a model is used to forecast the national PV generation.
- Code: PVnet, PVnet national and PVnet App
- Terraform and Airflow Dag
- AWS logs: aws/ecs/forecast_pvnet/
National xg makes forecast from 0 to 36 hours. It is a XGBoost model that used Satellite and NWP data. It prdocues a National forecast with probabilistic forecasts.
- code: uk-pv-national-xg
- Terraform and Airflow Dag
- AWS logs: aws/ecs/forecast_national/
The Forecast blend service reads all of the above forecasts and blends them together apprioately.
- Code: uk-pv-forecast-blend
- Terraform and Airflow Dag (GSP) and Airflow Dag (National)
- AWS logs: aws/ecs/forecast_blend/
The API loads forecasts and true values from the database and present the data in an easy to read way. This is run on Elastic Beantstalk. We use Auth0 to authenticate this API.
The API loads forecasts and true values from the database and present the data in an easy to read way. This is run on Elastic Beantstalk. We use Auth0 to authenticate this API.
- Code: pv-site-api
- Terraform
- AWS logs
- Databases: We have a few postgres database that store the PV and forecast data. These is run on RDS.
- Site databases clean up: Once a day we run a service to remove any data that is more than ~3 days old.
- Airflow is used to trigger the ECS tasks. They use the latest ECS Task definition. This is part of the ocf airflow project.
- OCF Dashboard: This is use for internally looking at the forecasts. Also for managing
sitesandusers.
The forecast is trained offline and the model weights are saved to s3. Python Code