Ease extension of REANA configuration

[diegodelemos]

Background

Currently, it is only possible to add new configuration through environment variables. For example, the REANA_MAX_CONCURRENT_BATCH_WORKFLOWS is a configuration read by REANA-Commons and used eventually in REANA-Server, this is a simple case which doesn't show too much indirection, it is an environment variable passed to an infrastructure pod.

Problem

When the configuration needs to be passed to a runtime pod, let's say REANA-Job-Controller, the process becomes more tedious. Let's take the example of REANA_STORAGE_BACKEND. Again this is an environment variable read by REANA-Commons, which at some point needs to be used by RJC to decide whether jobs should use shared storage via hostpath or via network file system. So the chain goes like follows:

• (1) One sets the environment variable at Helm template level
• (2) REANA-Commons loads it inside RWC, and passes it to the newly instantiated RJC through environment variable
• (3) The newly spawned RJC loads again through REANA-Commons the configuration and finally uses it to instantiate the job correctly (correct storage backend)

This is a procedure which is:

• Tedious and error-prone; an extra change in an unaffected repository needs to be done (previously, step (2)) to just pass the config over, with a chance of making mistakes in the middle
• Doesn't scale code wise; we just have to add more glue code lines to the code base as the configuration grows, see code in previously mentioned step(2).

Possible solutions

• (1) Central configuration file with all variables: This configuration file is still being loaded by REANA-Commons, so we have only one part in the code where the configuration is loaded, but then all components see all config.

• (2) Split configuration per "groups of components": one configuration file for infrastructure components, another configuration file for runtime pods and a global one (this could explode in complexity if more groups are to be created). A quick and verbose prototype:

REANA_INFRASTRUCTURE_MAX_CONCURRENT_WORKFLOWS = \
    load_config('infrastructure')['MAX_CONCURRENT_WORKFLOWS']
REANA_RUNTIME_COMPUTE_BACKENDS = \
    load_config('runtime')['COMPUTE_BACKENDS']
REANA_STORAGE_BACKEND = load_config_from_file('global')['STORAGE_BACKEND']

• ...

In either (1) or (2), new configmap(s) would be added to the Helm chart and it/they would be paramtrised from values.yaml.