flex-container-orchestrator

flex-container-orchestrator provides the command-line orchestration layer for the FLEXPART-IFS workflow used by the European Weather Cloud (EWC) blueprint.

The current EWC runtime is synchronous and command-line driven. It:

1. selects an IFS forecast cycle from ECMWF MARS, or uses an explicitly requested cycle;
2. retrieves and preprocesses the required IFS fields with flexprep;
3. runs FLEXPART-IFS through Docker Compose; and
4. runs Pyflexplot to generate dispersion plots.

The installed command is flexpart-workflow.

flexpart-workflow
      |
      +-- --latest
      |      `-- check recent IFS cycles in MARS and select the newest
      |          complete cycle for the requested forecast length
      |
      +-- --date YYYYMMDD --time HH
      |      `-- use an explicit IFS cycle
      |
      +-- --step HOURS
      +-- --location SITE
      |
      v
   flexprep
 MARS -> GRIB
      |
      v
FLEXPART-IFS
Docker Compose
      |
      v
  Pyflexplot
Docker Compose

The EWC blueprint installs and configures the runtime dependencies. For a manual or development installation, the orchestrator expects:

• Python >= 3.10 and < 3.13;
• a native ECMWF MARS client with valid user access;
• Docker with the Docker Compose plugin; and
• access to the workflow container images configured in docker-compose.yml.

The Python dependency on flexprep is installed with the orchestrator.

Clone the repository and install it with Poetry:

$ cd flex-container-orchestrator
$ poetry install --with dev

The package exposes the same CLI used on the EWC VM:

$ poetry run flexpart-workflow --help

By default, flexpart-workflow reads:

/etc/flexpart-ifs/workflow.yml

For local development, pass a configuration file explicitly:

$ poetry run flexpart-workflow \
    --config examples/workflow.yml \
    --date 20260812 \
    --time 06 \
    --step 4 \
    --location BEZ

A minimal configuration is provided in examples/workflow.yml. It contains static runtime settings such as paths, the flexprep domain/grid, and plotting configuration. Forecast cycle, forecast duration, and release site are runtime CLI arguments and are intentionally not fixed in the YAML configuration.

main:
  runtime:
    compose_file: ./docker-compose.yml
    run_dir: ./runtime/runs
    plot_dir: ./runtime/plots

  flexprep:
    output_dir: ./runtime/meteorology
    area: [65, -10, 35, 47]
    grid: [0.25, 0.25]
    max_level: 137
    mars_class: od
    expver: "1"

  workflow:
    plotting_preset: opr/ifs-hres-eu/all_pdf
    emission_duration_hours: 1

Use the latest complete IFS cycle available in MARS:

$ flexpart-workflow --latest --step 24 --location BEZ

Use an explicit cycle for a reproducible run:

$ flexpart-workflow \
    --date 20260812 \
    --time 06 \
    --step 24 \
    --location BEZ

--time must be one of 00, 06, 12, or 18 when --date is used.

Before FLEXPART is started, the orchestrator verifies that the required IFS fields are available for the complete requested forecast range. Meteorology is retrieved and preprocessed with flexprep and cached by forecast cycle. FLEXPART is launched only after the expected preprocessed GRIB files are present and non-empty.

The FLEXPART and Pyflexplot components are executed as short-lived Docker Compose services. Runtime data remain on the host through bind-mounted directories.

On an EWC blueprint deployment, the default locations are:

/var/lib/flexpart-ifs/meteorology
/var/lib/flexpart-ifs/runs
/var/lib/flexpart-ifs/plots

The recommended installation path for EWC users is the flexpart-ifs-ewc-blueprint repository. The blueprint installs this orchestrator, the native MARS tooling, Docker, the workflow configuration, and the required container images.

After deployment and user-specific MARS credential setup, a workflow can be started with:

$ flexpart-workflow --latest --step 24 --location BEZ

Run the unit tests with:

$ poetry run python -m pytest

Run static-analysis tools with:

$ poetry run pylint flex_container_orchestrator
$ poetry run mypy flex_container_orchestrator

The repository also contains MeteoSwiss-specific CI/development configuration. When mchbuild is available, the existing local build/test commands can be used:

$ pipx install mchbuild
$ mchbuild local.build local.test

These helpers are not required by the EWC runtime.