DEA Conflux

This is a prototype tool for processing bulk polygon drills.

• License: Apache 2.0
• Contact:sai.ma@ga.gov.au , bex.dunn@ga.gov.au

Installation

Install with pip:

pip install git+https://github.com/GeoscienceAustralia/dea-conflux.git

Or clone the repository and install from the local version.

git clone https://github.com/GeoscienceAustralia/dea-conflux.git
cd dea-conflux
pip install -e .

Usage

Conflux provides a command-line tool dea-conflux for running each step of the polygon drill. Descriptions of the commands are available with dea-conflux --help. Conflux requires a Datacube configuration to work.

To run Conflux on a single scene, give it a scene UUID present in the Datacube, a plugin describing the polygon drill, a place to put output files, and a shapefile defining the polygons:

dea-conflux run-one --uuid SCENE_ID --plugin PLUGIN_PATH -o OUTPUT_PATH -s SHAPEFILE_PATH

Conflux can also read from an AWS SQS queue. Messages must be the UUID of a scene.

dea-conflux run-from-queue --queue QUEUE_NAME --plugin PLUGIN_PATH -o OUTPUT_PATH -s SHAPEFILE_PATH

Note:

The dea-conflux only allows to run on partial mode right now. The polygons which partial overlap with the scene will also be processed.

Plugins

A plugin defines the inputs and outputs of a polygon drill. It is a Python file with the extensions .conflux.py. Examples are provided in the examples/ directory.

Plugins must provide:

• a product name for the drill,
• a version string for the drill,
• the resampling method to use on input rasters,
• the CRS to project the input rasters into,
• the resolution to resample the input rasters into,
• a dictionary of input products and bands,
• a transform function, and
• a summarise function.

Transform function

The transform is run to produce rasters to summarise and should contain operations like masking and band index calculation. Transform will be applied to the whole scene, not to a polygon if you are using a polygon.

Summarise function

The summarise function aggregates a dataset into a number of measurements that summarise a polygon, i.e. the outputs of the drill. Summarise is applied to just the pixels in the polygon.

Pre-commit setup

❯ pip install pre-commit
❯ pre-commit install

Your code will now be formatted and validated before each commit by running pre-commit run -a

The run-from-queue processing pipeline

The dea-conflux has two main processing functions. The run_one designs to do the local test and the run_from_queue design to run big scale processing. The run_from_queue needs a queue system as backend. We are using the AWS SQS queue as example.

The run_one processing pipeline

The run_one feature designs to do the single scene processing. The user have to provide the expected scene UUID in opendatacube.