Node.js Microsoft Azure functions responsible for extracting data from the core forecasting engine and importing it into a staging database prior to transformation for reporting and visualisation purposes external to the core forecasting engine. Data extraction is message driven and achieved using a REST API. Data extraction and import activites are performed in a fault tolerant manner to maxmise the amount of data available for reporting and visualisation purposes.
- Message based triggering is used when:
- Importing data for frequently updated locations that are not associated with a core forecasting engine display group.
- Importing data for multiple locations associated with a core forecasting engine fluvial display group.
- Importing data for multiple locations associated with a core forecasting engine coastal display group.
- Replaying data imports following an outage.
- Refreshing the list of fluvial forecast locations.
- Refreshing the list of coastal forecast locations. One message trigger for each of the three coastal location types.
- Refreshing multivariate threshold data.
- Refreshing the set of fluvial locations associated with each core forecasting engine display group.
- Refreshing the set of coastal locations associated with each core forecasting engine display group.
- Refreshing the set of core forecasting engine filters associated with each workflow.
- Refreshing the set of core forecasting engine ignored workflows.
- Optional - Messages containing the primary keys of staging database records holding data extracted from the core forecasting engine can be used to trigger reporting and visualisation activities (see Prerequisites and Non-test related function app settings and environment variables for further details).
- CRON expression based triggering is used to periodically remove stale timeseries data from the staging database.
- Prerequisites
- Installation activities
- Non-test related function app settings and environment variables
- Running the queue based functions
- Testing
- Replaying Messages From Dead Letter Queues After An Outage
If you have an idea you'd like to contribute please log an issue.
All contributions should be submitted via a pull request.
THIS INFORMATION IS LICENSED UNDER THE CONDITIONS OF THE OPEN GOVERNMENT LICENCE found at:
http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3
The following attribution statement MUST be cited in your products and applications when using this information.