openEO EODC Driver
- openEO version: 1.0.x (currently in development)
- openEO version: 0.4.2 (legacy, see openeo-openshift-driver tag v1.1.2)
This repository contains a fully dockerized implementation of the openEO API (openeo.org), written in Python. The web app implementation is based on Flask, while openEO functionalities are implemented as micro-services with Nameko.
Additionally, three docker-compose files implement a CSW server for data access (
/csw), an Apache Airflow workflow management platform coupled to a Celery cluster for data processing (
/airflow), and UDF Services for the R and Python languages (
/udf). The whole setup can be installed on a laptop to simplify development, but each service can run on independent (set of) machines.
Start the API
In order to start the API web app and its micro-services, a simple docker-compose up is needed. However some environment variables must be set first.
First some environment variable need to be set.
sample.env file and the
/sample-envs folder to to
/envs, respectively. The latter are
included in the
.gitignore by default. Do not change this. Variables in the
.envfile are used in
docker-compose.yml when bringing up the project, while variables in the individual env files in
/envs are available
within each respective container. The following is the list of files to update.
It should be mentioned that most of the env variables are prefixed with
OEO_ those variables are used by
dynaconf - the configuration management tool we use. The env
variables which are not prefixed are used outside of Python and are directly accessed as environment variables without
.env: note that you MUST create manually the folder specified for 'LOG_DIR'
Then also copy
/gateway/gateway/sample_openapi.yaml file to
/gateway/gateway/openapi.yaml and edit
info sections, adding your urls, name and some description.
Bring up web app and services
For local development, you will need a docker network shared across the API, CSW and Airflow setups. Create one like this:
docker network create openeo-v1.0
Leave the name as is or update in the docker-compose_dev.yml files for the API, CSW and Airflow.
From the main folder, run the following command:
docker-compose -f docker-compose.yml -f docker-compose_dev.yml up -d
docker-compose_dev.yml file is identical to
docker-compose.yml, but additionally exposes some ports and assigns the containers to the docker network created above.
Additionally, the bash functions in
dev_openeo_sample can be used (after filling in the relevant fields) to start the services (Nameko) or the gateway (Flask) locally without Docker containers. In this case one can use breakpoints to debug.
Set up all databases with Alembic
A number of databases are set up with Alembic, namely one for users, one for process graphs and one for jobs. Make sure that the API is up and run the following command to initialize all databases:
Add admin user
At least one admin user must exist in the users database, to allow using the API functionalities via the endpoints. This user must be added manually to the database. In order to do so, you need to connecto to the gateway container and run a sequence of commands.
docker exec -it openeo-users-db bash
Once in the container, connect to the database:
psql -U $OEO_DB_USER -p $OEO_DB_PORT -h localhost $OEO_DB_NAME
Before adding user, user profiles must be inserted as well as identity providers (for OpenIDConnect).
The following create two user profiles (
profile_2), with different data access. The specific profile name and fields in the data access depend on the implementation.
insert into profiles (id, name, data_access) values ('pr-19144eb0-ecde-4821-bc8b-714877203c85', 'profile_1', 'basic,pro'); insert into profiles (id, name, data_access) values ('pr-c36177bf-b544-473f-a9ee-56de7cece055', 'profile_2', 'basic');
Identity providers can be added with the following command:
insert into identity_providers (id, id_openeo, issuer_url, scopes, title, description) values ('ip-c462aab2-fdbc-4e56-9aa1-67a437275f5e', 'google', 'https://accounts.google.com', 'openid,email', 'Google', 'Identity Provider supported in this back-end.');
Finally, users can be added to the users database. In order to add a user for Basic auth, one first needs to create a hashed password. Execute the following in a Python console.
from passlib.apps import custom_app_context as pwd_context print(pwd_context.encrypt("my-secure-password"))
Then back on the database command line, run the following replacing
hash-password-goes-here with the output of the previous command (leave it wrapped in single quotes):
insert into users (id, auth_type, role, username, password_hash, profile_id, created_at, updated_at) values ('us-3eb63b58-9a04-4098-84d7-xxxxxxxxxxxx', 'basic', 'admin', 'my-username', 'hash-password-goes-here', 'pr-c36177bf-b544-473f-a9ee-56de7cece055', '2019-12-18 10:45:18.000000', '2019-12-18 10:45:18.000000');
A user for Basic auth with admin rights is now inserted in the database. Note that the
profile_id matches the one of
The following command creates a user with admin rights for OpenIDConnect auth:
insert into users (id, auth_type, role, email, profile_id, identity_provider_id, created_at, updated_at) values ('us-3eb63b58-9a04-4098-84d7-yyyyyyyyyyyy', 'oidc', 'admin', 'firstname.lastname@example.org', 'pr-19144eb0-ecde-4821-bc8b-714877203c85', 'ip-c462aab2-fdbc-4e56-9aa1-67a437275f5e', '2019-12-18 10:45:18.000000', '2019-12-18 10:45:18.000000');
Note that the
identity_provider_id matches the only one created above, and the
profile_id matches the one of
Add collections and processes
Currently, no collection and no process are available yet at the endpoints
sample-auth file to
auth and fill the back-end URL and user credential (user with admin rights). Then run the following to add collections (sourced from the CSW server) and processes to the back-end:
source auth python api_setup.py
Bring down web app and services
In order to bring down the API, run the following docker compose from the main folder:
For development we provide a set of tests including unittests, linting and static type checking. Find more details here.