Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.


Codacy Badge Build Status

The openeo-validation-engine uses the OpenEO-sentinel-reference-jobs as a submodule and fetches the results of these jobs from every specified back-end provider. The results are then validated and a report is generated for each job. These reports are then pushed to the gh-pages branch of this repository.

Due to filesize limitations, comparison images are currently not stored in this process.


We use Python3 in a venv environment. Using a venv is not necessary but recommended. The following commands have to be run sequentially to install the application and its dependencies.

git clone --recursive
cd openeo-result-validation-engine
python3 -m venv ./venv
source venv/bin/activate
pip install -r requirements.txt
cd openeo-python-client
pip install -r requirements.txt
pip install -e .


OpenEO back-end providers can be added in the backendProvider.json. This JSON contains an array of JSON objects that contain the information for a given provider.

  • name: The name of the backend-provider.
  • baseURL: The address of the openEO API endpoint from the backend-provider.
  • credentials: The account information required to authenticate openEO requests.
    • user: The username, this is optional and can be left empty
    • password: The password to authenticate, this is optional and can be left empty
      "name": "WWU_GEE",
      "baseURL": "",
      "credentials": {
        "user": "Example",
        "password": "Example123"

Reference jobs

Reference jobs (jobs that are validated) can be added in the folder openeo-sentinel-reference-jobs. They are currently separated by arbitrary regions to make it a bit easier to distinguish them. Each Job has a folder that should be called Job-N (N is the counter). This folder then has to contain a folder that matches the name specified in the backendProvider.json, e.g. openEO-GEE in our current example. This folder then contains a single process graph.

Validation rules can be configured on a per job basis, in a file called validation-rules.json. There are three possible options:

  • Specifying no rules; default validation rules get selected (stored in the folder rule-configurations)

  • Specifying a name of a ruleset; the validation engine then tries to find a json file with the name name-validation-rules.json

  "ruleset-name": "quick"
  • Specifying validation rules directly inside the json.
  "validation": {
      "rules": {
          "input-data-checks": {
          "output-data-checks": {
              "matching-file-extensions": "png",
              "file-size-check": 0.2
          "color-checks": {
            "threshold-correlation": 0.5,
            "threshold-intersection": 345,
            "threshold-chi-squared": 1500,
            "threshold-hellinger": 0.5,
            "threshold-chi-square-alt": 5000,
            "threshold-kullback-leibler": 5000
          "pixel-checks": {
            "resolution-allow-divergence": 0.5,
            "image-similarity-measures": 0.95,
            "resize-factor": 0.5
          "classification-checks": {
            "matching-boundaries": 0.20,
            "resize-factor": 0.5
          "nan-value-check": {
            "allow-nan": false


Activate the Python venv and start the ValidationEngine.

source venv/bin/activate

Additional CLI arguments

Run python --help to get information on all available CLI commands.

If you want to prevent the validation engine from downloading the same OpenEO jobs again you can pass the argument --offline True. Example: python --offline True

To speed up the validation, a resize factor can be used. This factor reduces the size of the image and thus all computations are quicker. A value of 0.10 results in an image that is 10% of the original size. This of course reduces the quality of the validation.

Example: python --resize 0.10

Mocked examples (jobs stored in the folder mock-examples) can be run with:

python --mock True 

To run only a specific reference job, the job-identifier can be passed as an argument. The structure of the id is as follows: -Job- Examples: Island-Job-2, Netherlands-Job-2

python --refJob Switzerland-Job-1 

Validation against local files

Add a similar json-object as the one below to the backendProvider.json.

    "name": "Local-Python",
    "local": true

In the folder openeo-sentinel-reference-jobs, add a folder with the name of the backend provider, e.g. Local-Python. This folder then should contain a json that stores the path to the locally stored file instead of the process-graph.

  "file": "/path/to/file/file.tiff"

This allows to validate local results against openEO cloud-processed results and it also enables validation of just local results.

Validation "Studies"

The ValidationEngine runs every 24 hours to compute the validation results for two "studies".

Validation reports for the comparison between back-end providers EURAC and WWU/GEE can be found in the branch gh-pages of this repository. It only contains the JSON reports as the raw data from EURAC is to large.

Validation reports for the comparison between the COPERNICUS/S2 and COPERNICUS/S2_SR data set of the WWU/GEE back end can be found in the branch reports-GEE_S2-vs-GEE_SR of this repository. These branch also contains imagery, as we are not able to retrieve native resolution from the WWU/GEE back end and thus are able to store the images on GitHub.

Encrypting Git Token

To publish the validation reports to Github this repository needs a valid GitHub Token.

  1. install the Travis CI Command Line Client by running gem install travis and login with travis login --org
  2. Generate SSH-KEY ssh-keygen -t rsa -b 4096 -C ""
  3. Add to the repository
  4. Encrypt the file with the Travis CLI travis encrypt-file deploy_key --add (This will generate an encryption label)
  5. Adjust encryption label in .travis.yml line 21


Image-based validation of Earth Observation cloud processing service results




No packages published