Special Python Requirements for ETL Pipeline

[Miro-H]

Currently, there is no setup to support (possibly conflicting) package requirements for the ETL pipeline of different projects (or experiment designs).

Especially for the Ansible master, it would be nice if users could specify some virtual environment or requirements.txt file with package dependencies necessary for their ETL pipeline.

[nicolas-kuechler]

For project-specific ETL steps (extractors, transformers, loaders), we could require to have a poetry project in does_config/etl.
(Maybe we should always have one there by default)

In this poetry project, we require to include the module of the provided etl steps (which also gives you Extractor, Transformer, Loader classes to extend for custom steps).

In Ansible where we currently call etl.py we would call the project (or one of multiple) under does_config/etl with its dependencies.

Old call:

  - name: Run ETL pipeline over results files
    delegate_to: localhost
    ansible.builtin.shell:
      cmd: python scripts/etl.py --suite {{ suite }} --id {{ suite_id }}