Skip to content

Latest commit

 

History

History
147 lines (107 loc) · 5.43 KB

README.md

File metadata and controls

147 lines (107 loc) · 5.43 KB
Raw

Tensorlakehouse backend implentation of openEO API

The Tensorlakehouse openEO driver is a backend implementation of the openEO API specification. It allows data scientists to list available collections and processes and to submit synchronous and asynchronous requests for data retrieving and data processing

Table of Contents

User guide

Please read our user-guide section if you're interested to learn how to use openEO

Python Environment

Using a virtual environment for all commands in this guide is strongly recommended. In particular, we recommend python 3.9.16 version

Installation

  1. Go to tensorlakehouse-openeo-driver directory
  2. Install tensorlakehouse-openeo-driver dependencies: pip install -r requirements.txt. Optionally, you can install other dependencies for development purpose: pip install -r dev_requirements.txt
  3. Optional, but strongly suggested: follow the step describe here to setup detect-secrets tool

Running locally using containers

Setting environment varibles:

  • PYTHONPATH for instance, PYTHONPATH=/Users/alice/tensorlakehouse-openeo-driver/
  • STAC_URL URL to the STAC service that you want to connect to (e.g., https://stac-fastapi-sqlalchemy-nasageospatial-dev.cash.sl.cloud9.ibm.com)
  • CREDENTIALS is a set of credentials (encoded in base64) that allows this service to access COS S3 buckets
  • BROKER_URL - URL to the broker, which mediates communication between clients and workers.
  • RESULT_BACKEND - URL to the backend, which is necessary when we want to keep track of the tasks' states or retrieve results from tasks
  • if you want to implement OIDC authentication you need:
    • APPID_ISSUER which is the authorization server url
    • APPID_USERNAME username of the authorization server
    • APPID_PASSWORD password of the authorization server
    • OPENEO_AUTH_CLIENT_ID client ID
    • OPENEO_AUTH_CLIENT_SECRET client secret
  • GEODN_DISCOVERY_USERNAME and GEODN_DISCOVERY_PASSWORD (optional) for basic auth to get GeoDN.Discovery (former PAIRS) metadata

FLASK_APP and FLASK_DEBUG environment variables are useful for debugging:

cd <path-to-parent-dir>/tensorlakehouse-openeo-driver/
export FLASK_APP=tensorlakehouse_openeo_driver.local_app
export FLASK_DEBUG=1
flask run

Building and running container images

Prerequisites:

  • docker or podman-compose installed
  • postgres database with postgis extension
  • redis database

Step 1 Generate credentials

{
    "<my-bucket-name>": {
        "endpoint": "s3.<region>.<hostname>",
        "access_key_id": "<access key>",
        "secret_access_key": "<secret>",
        "region": "<region>"
    },
}

then convert it to base64 by running:

python tensorlakehouse_openeo_driver/util/credentials_manager.py --file <path>

The output should be used to set the CREDENTIALS env variable

Step 2. Set the environment variables and create .env file

# credentials to access cloud object store 
CREDENTIALS=<see step 1>

BROKER_URL=<redis database url>
RESULT_BACKEND=<redis database url>

DASK_SCHEDULER_ADDRESS=http://127.0.0.1:8787

### optional environment variables

PYTHONPATH=/Users/alice/tensorlakehouse-openeo-driver/
# basic credential to proprietary solution
GEODN_DISCOVERY_PASSWORD=<geodn-discovery-password>
GEODN_DISCOVERY_USERNAME=<geodn-discovery-username>
# authorization server
APPID_ISSUER=<authorization server url>
# username and password
APPID_USERNAME=<username>
APPID_PASSWORD=<password>
# client id and secret
OPENEO_AUTH_CLIENT_ID=<client id>
OPENEO_AUTH_CLIENT_SECRET=<client secret>

# default is 9091
TENSORLAKEHOUSE_OPENEO_DRIVER_PORT=9091

Step 3 - Build tensorlakehouse-openeo-driver

Podman is a drop-in replacement for Docker. If you are a Docker user, just replace podman by docker and you will be fine. Go to repository root dir and run:

podman build -t tensorlakehouse-openeo-driver -f Containerfile

Step 4 - Run services using podman-compose

run podman-compose

podman-compose -f podman-compose.yml --env-file /Users/alice/tensorlakehouse-openeo-driver/.env up

Software architecture

Check software architecture diagrams.

Contributing

Check CONTRIBUTING.md.

Getting support

Check SUPPORT.md.