Auto-mount emptyDir volumes for k8sapi/kubelet or runAsNotRoot: true(?) executors

[sachua]

Summary

Will there be support for outputs from base image layer instead of using emptyDir to save params/artifacts when using the kubelet or k8s api executors?

Motivation

I was running Kubeflow Pipelines with the container runtime executor as kubelet, but I kept hitting errors with the pipelines completing their tasks.

I then tried the https://github.com/argoproj/argo/blob/master/examples/artifact-passing.yaml example to try to narrow down the problem and realized that I had to mount an emptyDir volume in order for it to work.

My question is will there be support for outputs from base image layer with k8s api and kubelet executors in the future? Or is it not possible due to the way these executors work?

[alexec]

We use PNS ourselves - so this would be a great one for a community contribution.

[dtaniwaki]

As far as I know, it’s not possible now.
kubernetes/kubectl#454

[Ark-kun]

Argo already adds volumes to capture artifacts for docker executor. Why not for other executors? If Argo tells us to "mount an emptyDir volume", why can't it just auto-mount that volume itself?

[Ark-kun]

Our users need artifacts to work for some non-docker executor.

[sarabala1979]

I agree with @Ark-kun. The controller should add emptydir volume automatically if non-docker executors configured. This is a nice feature and will reduce many support questions.

[Ark-kun]

Related: kubeflow/pipelines#1654 kubeflow/pipelines#561 kubeflow/pipelines#3407

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Ark-kun]

Let's keep this up

[nakfour]

Hey everyone, we use the "k8sapi" executor to run Argo on Openshift. Wanted to see if there is progress on this issue?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alexec]

I've spent some time updating the documentation for the executors and how and when to use [emptyDir]https://argoproj.github.io/argo/empty-dir/).

TL;DR:

• Use the docker executor if you don't care about security and do care about performance. Use the k8sapi executor if you care about security and not about performance. Use PNS for something in between. Don't use kubelet.
• If you use emptyDir you don't get many of the security problems.

It is reasonable for the kubelet and k8sapi executors to automatically mount empty dir volumes for output artifacts.

[alexec]

Argo already adds volumes to capture artifacts for docker executor.

@Ark-kun I'm not sure this is correct?

[Ark-kun]

@Ark-kun I'm not sure this is correct?

AFAIK, Argo tries to use mirrored volumes to make the input artifacts accessible to the main container and capture the output artifacts in the wait container.

See https://github.com/argoproj/argo/blob/4de64c618dea85334c0fa04a4dbc310629335c47/workflow/controller/workflowpod.go#L793 https://github.com/argoproj/argo/blob/4de64c618dea85334c0fa04a4dbc310629335c47/workflow/controller/workflowpod.go#L896
https://github.com/argoproj/argo/blob/e14f4f922ff158b1fa1e0592fc072474e3257bd9/workflow/executor/executor.go#L369

[Ark-kun]

Perhaps we should add logic such that for every output artifact there should be an emptyDir mounted at its parent directory unless there is another volume already mounted underneath.

[alexec]

thank you @Ark-kun

[Ark-kun]

I'm not sure this has been fixed.

[alexec]

Re-opened. This got closed by a PR and should not have been close. Would anyone be interested in investigating how to fix this?

[alexec]

This would be for k8sapi and kubelet executors. I think it may also be needed for whenever runAsNotRoot: true.

[alexec]

argoproj/argoexec:auto-emptydir and argoproj/workflow-controller:auto-emptydir available for anyone who wants to test this.

[alexec]

I'm going to wait for user testing before proceeding.

[alexec]

bump! please let me know if you really want this by testing the controller

[juliusvonkohout]

bump! please let me know if you really want this by testing the controller

I think it is safe to say that we do really want this. Just have a look at the number of upvotes for such a technical issue and the other similar bugreports in kubeflow etc. It might help if you could provide images for testing or even some short instructions for a kubernetes cluster. Then everyone can test his workloads. Otherwise everyone has to dig into how to build the images etc.

[alexec]

I think this issue will be moot with the Emissary Executor.

What is the Emissary Executor?

Why?

• As fast as the fastest executor (docker)
• As secure as the securest executor (k8iapi).
• Every feature of all other executors.

@terrytangyuan @Ark-kun would you be able to test this new executor, please?

You need dev builds of the controller and executor:

• alexcollinsintuit/workflow-controller:emissary
• alexcollinsintuit/argoexec:emissary

[juliusvonkohout]

I have tested it with kubeflow pipelines 1.3

There are only minor issues.

1. op.set_security_context(V1SecurityContext(run_as_user=1000))
   leads to permission errors in /var/argo/templates
   But without that addition all pipelines finished successfully with runasnonroot.

2. Kubeflow pipelines runs with argo 2.7.5 by default. So the artifacts etc where successfully uploaded to minio. Nevertheless it seems that there was a communication error such that i cannot download them from the kubeflow pipelines webinterface anymore, even though they are in minio. So the output location is most likely not properly submitted. This is also the case if i use pns with your special emissary images, so i guess the 2.7.5 interface is just in some way not compatible with 2.12+ .

Since kubeflow pipelines are going to use argo 2.12 in the future, it would be good if the emissary executor is compatible with 2.12.

[alexec]

I don't plan to backport the emissary to v2.12. It's a big and therefore risky change.

[alexec]

I think v2.12 is the right version for Kubeflow. I'd like us to get more involved and help out with Kubeflow more. Can you add me to the PR you're working on please?

[juliusvonkohout]

Well if the interfaces are somewhat compatible its fine. Then i just have to use the upstream argoexec and workflow-controller images for the emissary executor. That worked for 2.7.5 with 2.11 images as far as i remember.

kubeflow/pipelines#4553 is the issue for argo 2.12 in kubeflow
and i think you are already in the conversation because of #4953