⤴️ Go back to the Reference System Software repository ⤴️

Processing Sentinel-3

• Processing Sentinel 3
  • Overview
  • Available RS Addons
  • Dependencies
  • Deployment
  • Acroynms

Overview

Within this repository you can find the RS Add-ons the for the Sentinel-3 mission. Each RS-Add-on contains a mission specific workflow that can be deployed on its own and adding the processing preparation as well as the scientific processor that processes the satellite data.

Available RS Addons

The following figure provides an overview about how the different RS Add-ons are chained into each other when all are deployed:

The S3 ACQ wrapper contains the IPFs DDC, L0Pre and L0Post processor as a single step and will process EDRS Sessions into S3 granules. These granules are processed by the L0P processor into S3 L0 products.

The following individual NRT processing chains are available:

• S3 ACQ
• S3 L0P
• S3 OL1
• S3 OL1 RAC
• S3 OL1 SPC
• S3 OL2
• S3 MW1
• S3 SL1
• S3 SL2
• S3 SL2 FRP
• S3 SR1
• S3 SM HY
• S3 SM LI
• S3 SM SI
• S3 PUG

The following individual NTC processing chains are available:

• S3 OL1 NTC
• S3 OL2 NTC
• S3 MW1 NTC
• S3 SL1 NTC
• S3 SL2 NTC
• S3 SL2 FRP NTC
• S3 SM2 HY NTC
• S3 SM2 LI NTC
• S3 SM2 SI NTC
• S3 SR1 NTC
• S3 SY2 NTC
• S3 SY2_AOD NTC
• S3 SY2_VGS NTC
• S3 PUG NTC

The following individual STC processing chains are available:

• S3 MW1 STC
• S3 SM2 HY STC
• S3 SM2 LI STC
• S3 SM2 SI STC
• S3 SR1 STC
• S3 SY2 STC
• S3 SY2_VGS STC
• S3 PUG STC

For more information, please consult https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-3

Dependencies

In order to work properly the RS Add-ons are requiring a few dependencies that needs to be deployed before hands.

Infrastructure Layer

COPRS are using a set of technologies that are utilized by the components. E.g. Kafka, Elasticsearch and MongoDB needs to be available. This infrastructure layer needs to be installed first. For more information on its installation, please consult this repository.

RS Core Components The chain itself will not work without other RS Core components. At least the a Ingestion Chain (for retrieving inputs) and Metadata Extraction (for extracting metadata) needs to be deployed to the cluster. It is also highly recommended to deploy the DLQ as well for a proper error handling.

You can find more information and installation instructions in this repository.

Build

In order to build all the RS add-ons contained in this repository, please execute the following script from inside the repository:

./build_all.sh <target_dir> <version>

The placeholder <target_dir> shall point to the location where you want to put the generated RS add-ons, e.g. /tmp/rs-addons. When the generation had been performed successfully, you can find the ZIPs containing the RS Add-ons at that location.

The placeholder <version> defines the version that shall be added to the RS Add-ons when being generated e.g. 1.2.3.

Be aware that this script will just ensure that the RS Add-ons are build. In order to rebuild the docker images that contains the CFI and the RS add-on servces needed to operate them, please consult this instruction.

Deployment

Each RS Add-on contains a description about the services for the specific processing chain that shall be deployed into the cluster as well as a reference to the docker images that shall be used and a factory default configuration.

In order to deploy a RS Add-on the Ansible scripts from the infrastructure repository can be used. The following Add-Ons are available for deployment:

NRT

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-ACQ_<VERSION>.zip \
    -e stream_name=S3_ACQ

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-L0P_<VERSION>.zip \
    -e stream_name=S3_L0P

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-OL1_<VERSION>.zip \
    -e stream_name=S3_OL1

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-OL1-RAC_<VERSION>.zip \
    -e stream_name=S3_OL1_RAC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-OL1-SPC_<VERSION>.zip \
    -e stream_name=S3_OL1_SPC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-OL2_<VERSION>.zip \
    -e stream_name=S3_OL2

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-MW1_<VERSION>.zip \
    -e stream_name=S3_MW1

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SL1_<VERSION>.zip \
    -e stream_name=S3_SL1

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SL2_<VERSION>.zip \
    -e stream_name=S3_SL2

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SL2-FRP_<VERSION>.zip \
    -e stream_name=S3_SL2_FRP

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SR1_<VERSION>.zip \
    -e stream_name=S3_SR1

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SM2-HY_<VERSION>.zip \
    -e stream_name=S3_SM2_HY

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SM2-LI_<VERSION>.zip \
    -e stream_name=S3_SM2_LI

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SM2-SI_<VERSION>.zip \
    -e stream_name=S3_SM2_SI

NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-OL1-NTC_<VERSION>.zip \
    -e stream_name=S3_OL1_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-OL2-NTC_<VERSION>.zip \
    -e stream_name=S3_OL2_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SL1-NTC_<VERSION>.zip \
    -e stream_name=S3_SL1_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SL2-NTC_<VERSION>.zip \
    -e stream_name=S3_SL2_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SL2-FRP-NTC_<VERSION>.zip \
    -e stream_name=S3_SL2_FRP_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SR1-NTC_<VERSION>.zip \
    -e stream_name=S3_SR1_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SM2-HY-NTC_<VERSION>.zip \
    -e stream_name=S3_SM2_HY_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SM2-LI-NTC_<VERSION>.zip \
    -e stream_name=S3_SM2_LI_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SM2-SI-NTC_<VERSION>.zip \
    -e stream_name=S3_SM2_SI_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SY2-NTC_<VERSION>.zip \
    -e stream_name=S3_SY2_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SY2-VGS-NTC_<VERSION>.zip \
    -e stream_name=S3_SY2_VGS_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SY2-AOD-NTC_<VERSION>.zip \
    -e stream_name=S3_SY2_AOD_NTC

STC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-OL1-STC_<VERSION>.zip \
    -e stream_name=S3_OL1_STC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-OL2-STC_<VERSION>.zip \
    -e stream_name=S3_OL2_STC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SL1-STC_<VERSION>.zip \
    -e stream_name=S3_SL1_STC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SL2-STC_<VERSION>.zip \
    -e stream_name=S3_SL2_STC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SL2-FRP-STC_<VERSION>.zip \
    -e stream_name=S3_SL2_FRP_STC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SR1-STC_<VERSION>.zip \
    -e stream_name=S3_SR1_STC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SM2-HY-STC_<VERSION>.zip \
    -e stream_name=S3_SM2_HY_STC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SM2-LI-STC_<VERSION>.zip \
    -e stream_name=S3_SM2_LI_STC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SM2-SI-STC_<VERSION>.zip \
    -e stream_name=S3_SM2_SI_STC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SY2-STC_<VERSION>.zip \
    -e stream_name=S3_SY2_STC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-SY2-VGS-STC_<VERSION>.zip \
    -e stream_name=S3_SY2_VGS_STC

PUG

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-PUG_<VERSION>.zip \
    -e stream_name=S3_PUG

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-PUG-NTC_<VERSION>.zip \
    -e stream_name=S3_PUG_NTC

ansible-playbook deploy-rs-addon.yaml \
    -i inventory/mycluster/hosts.ini \
    -e rs_addon_location=https://artifactory.coprs.esa-copernicus.eu/artifactory/rs-zip-private/rs-addons/s3/RS_ADDON_S3-PUG-STC_<VERSION>.zip \
    -e stream_name=S3_PUG_STC

For further information on using the deployment script, please consult its manual.

Acroynms

Abbreviation	Definition
COPRS	Copernicus Reference System
CPU	Central Processing Unit
DB	Database
DDC	Direct Data Capture
EDRS	European Data Relay Satellite
IPF	Instrument Processing Facility
MWR	Microwave Radiometer
L0	Level-0
L1	Level-1
L2	Level-2
RAM	Random Access Memory
RS	Reference System
S3	Sentinel-3
SCDF	Spring Cloud Dataflow
SL	Short for SLSTR: Sea and Land Surface Temperature Radiometer
SpEL	Sprint Expression Language
SR	Short for SRAL: SAR Radar Altimeter
SY	Synergy
OBS	Object Storage
OL	Short for OLCI: Ocean and Land Colour Instrument

This project is funded by the EU and ESA.