Support independent pods for each workflow/job?

[jirikuncar]

• create separate pods for each workflow/job?
• shared volume per workflow
• publish outputs after a job finishes
• make job template configurable (init containers, volumes, ...)

[lukasheinrich]

Hi @jirikuncar,

the individual jobs of a workflow are already one pod each. Currently only with a single ctr, but it's just a matter of exposing init containers. The jobs get a shared volume mounted by default.

We are working on a feature, where the submitted workflow engine instances (i.e. the thing that submits the steps in the right sequence) also gets started as either a) a Job for noninteractive workflows or b) a Deployment for interactive workflows (think: a notebook that steers the workflow)

init containers are something we have on our radar but it's not clear yet how to incorporate them. Generally I personally think the job-ctrl should be a quite thin layer on top of k8s and expose most APIs as long as they are secure, but it does tie us to Kubernetes a bit.

[rokroskar]

@lukasheinrich, can the jobs then use the shared volume mount for sharing results? I thought results of individual jobs get pushed to CephFS.

[lukasheinrich]

yes in the case of a shared filesystem there is one CephFS volume that needs to be shared between the

  +-----[ engine ]
  |
+---+
| V  |----[ job1 ]
+---+
  |
  +[ job2 ]

Later (when jobs 1 and 2 are finished, the engine starts job 3):

  +-----[ engine ]
  |
+---+
| V |
+---+
  |
  +[ job3 ]

the volume lifetime is in principle independent of the pod lifetime (though one could also instruct the engine to copy only a subset of the data to a "results" volume and then delete the workflow volume, in which caase the lifetime of the volume and the engine pod is the same

[diegodelemos]

Workflow pods are now independent and belong to the user who created them (more details about implementation here).