Repository navigation
Replies: 4 comments 4 replies
|
Hello @indrajithekanayake, please take a look on the above plan and feel free to suggest any changes. If you agree with everything just reply here that everything looks ok. |
Argo Integration Report
Problem introductionThe problem highlighted here #573 (specifically #135, and #882). In summary, Argo Workflow spawns three containers inside the pod: For that specific problem issue is "execution-model mismatch". The bigger question is execution-model mismatch to what level? We don't really know. That's what we gonna find out at the end of Phase 1 by analyzing Argo and Urunc execution models. Also, we cannot say that if 1. Understanding Argo Workflows execution model (with
|
| Container | Image | Created | Started | Finished | Alive | Exit |
|---|---|---|---|---|---|---|
| init | quay.io/argoproj/argoexec:v4.1.4 |
11:28:43.150 | 11:28:43.197 | 11:28:43.283 | 0.087 s | 0 |
| wait | quay.io/argoproj/argoexec:v4.1.4 |
11:28:47.304 | 11:28:47.431 | 11:33:48.001 | 300.570 s | 0 |
| main | python:alpine3.23 |
11:28:47.700 | 11:28:47.768 | 11:33:47.889 | 300.121 s | 0 |
Interestingly enough, before init exit it copies argoexec executable/binary (the image is basically one Go executable/binary, so same size as above; 166.6MB) into /var-run-argo/argoexec emptyDir for the main container to use.
ex (look at the copied argoexec binary snapshot below):
/var/lib/kubelet/pods/c6903810-78f0-485c-8bf7-f39d6b3e552c/volumes/kubernetes.io~empty-dir/var-run-argo/argoexec 166,569,319 bytesvar-run-argo is not the only emptyDir we are dealing with here. Three emptyDirs' created per pod and destroyed when that pod dies. Its path is keyed by the pod UID. I think it would be better to put in a diagram.
In addition to the emptyDirs' there is projected volume mounted the service-account token to all three containers read-only. Only
waitever uses it (becausewaitrequires the executor Role to communicate the exit code to the API server, and the ServiceAccount credential materials are stored in this projected-volume).
The main actually does the craziest part. Instead of letting runc to launch python /argo/staging/script, main launches it as a child under the argoexec emissary with /var/run/argo/argoexec emissary -- python /argo/staging/script
waitlogged completion 1 ms after emissary logged sub-process exited (so Im assuming it's file watch)
Also, I noticed
stdout42 is only printed in the terminal. Since our workflow is a single step with nothing after, Argo skips capturing thestdout. This was printed in theinitandwaitcontainer logs asincludeScriptOutput=false. test this behavior again when running multi step, DAG workflow. When another pod references to scriptsstdoutresult this must become true
BPF execsnoop was started before the workflow was submitted, so every exec below is caused by this pod/ sandbox. I've used a Python script to reconstruct the spawn sequence below from the /root/argo-lab/captures/execsnoop.log
| TIME | TREE (indent = ancestry) | PID | CONTAINER |
|---|---|---|---|
containerd |
96334 | ||
| 11:28:42 | └─ shim SPAWN |
98658 | sandbox |
| 11:28:42 | └─ shim SERVE ← the pod's shim |
98666 | sandbox |
| 11:28:42 | └─ runc create |
98677 | sandbox |
| 11:28:42 | └─ runc start |
98696 | sandbox |
| 11:28:42 | └─ pause ← ENTRYPOINT |
98690 | sandbox |
| 11:28:43 | └─ runc create |
98703 | init |
| 11:28:43 | └─ runc start |
98723 | init |
| 11:28:43 | └─ argoexec init ← ENTRYPOINT |
98717 | init |
| 11:28:43 | └─ runc kill --all |
98737 | init |
| 11:28:43 | └─ runc delete |
98743 | init |
| 11:28:46 | └─ shim REAP |
98753 | init |
| 11:28:46 | └─ runc delete --force |
98761 | init |
| 11:28:47 | └─ runc create |
98789 | wait |
| 11:28:47 | └─ runc start |
98808 | wait |
| 11:28:47 | └─ argoexec wait ← ENTRYPOINT |
98801 | wait |
| 11:28:47 | └─ runc create |
98822 | main |
| 11:28:47 | └─ runc start |
98842 | main |
| 11:28:47 | └─ argoexec emissary ← ENTRYPOINT |
98836 | main |
| 11:28:47 | └─ python /argo/staging/script |
98858 | main |
| 11:33:47 | └─ runc kill --all |
106595 | main |
| 11:33:47 | └─ runc delete |
106601 | main |
| 11:33:47 | └─ shim REAP |
106608 | main |
| 11:33:47 | └─ runc delete --force |
106616 | main |
| 11:33:48 | └─ runc kill --all |
106621 | wait |
| 11:33:48 | └─ runc delete |
106627 | wait |
| 11:33:49 | └─ shim REAP |
106645 | wait |
| 11:33:49 | └─ runc delete --force |
106653 | wait |
| 11:33:50 | └─ runc kill |
106659 | sandbox |
| 11:33:50 | └─ runc kill --all |
106666 | sandbox |
| 11:33:50 | └─ runc delete |
106672 | sandbox |
| 11:33:50 | └─ shim REAP |
106678 | sandbox |
| 11:33:50 | └─ runc delete --force |
106686 | sandbox |
2. Execution model of urunc (relevant parts for the Argo integration)
More information will be appended over time as my mental image of urunc improves over time.
Below, I tried to put together the urunc execution architecture for the 3 types of guests we have,
| Guest Type | OS / Kernel | Filesystem | Initialization | Application Execution | VMM / Monitor |
|---|---|---|---|---|---|
| 1. Linux Guest | Linux kernel | Initrd, block/devmapper, 9pfs, virtiofs | urunit (when configured) |
urunit launches the application |
QEMU, Firecracker, Cloud Hypervisor |
| 2. FreeBSD Guest | FreeBSD kernel | Block/devmapper, 9pfs | urunit (when configured) |
urunit launches the application |
QEMU, Firecracker |
| 3. Unikernel Guest | Unikernel (OS + application) | Initrd, block/devmapper, shared filesystem (9pfs for supported Unikraft configurations), in-memory filesystem, or no separate rootfs | Unikernel-specific | Integrated application or framework-specific execution | QEMU, Firecracker, Solo5, etc. |
3. Breakdown of the main incompatibilities, along with a compatibility matrix of Argo features against guest capabilities
More information will be appended over time when I come up with new incompatibilities, plus when my understanding of urunc improves over time. Identifying compatibility matrix I will do later given the list can be exhaustive and maybe not so necessary too
Below is a known list of incompatibilities withhold argo intigration with some priority indication (🔴 > 🟡 > 🟢). The green ones are low priority things that break when Im running tests. They dirrectly do not affect argo intigration but worth fixing:
- 🔴 urunc has no
execOCI-runtime subcommand, causing Argo Workflows (and any exec-dependent controller) to hang forever #882 Argo useskubectl execto stop containers. The docs describe this for injected sidecars. Well well well I initially thought this was not a priority, but this turned out to be a high-priority requirement. In the experiment below there were 3execcalls, and v0.8.0 has noexec, so all three attempts failed - 🔴 Passing app arguments on the kernel command line quietly mangles some of them #1067 Need immediate fix in POC stage
- 🟢 The containers urunc hand over to runc are placed under /system.slice/... instead of the pod's slice #1105 Just a correction issue not a blocker. No major impact on Argo compatibility.
- 🟢 Refactor and use libcontianer to setup the execution environment for the monitor process #780 In following experiment
mainexited 1 s so I don't have cgroup captures to prove (Note to me: take a look at this in later experiment). However, I think givenlibcontainer=falseby default, no cgroup is created at all. So QEMU and virtiofsd stay in whatever cgroup their parent was in ( shim's cgroup undersystem.slice). The problem is VM's memory (guest RAM, 1024 MiB in our config, plus QEMU and virtiofsd) isn't counted as the pod's usage. It's charged to the shim's/containerd's cgroup instead.
My first instinct on the above was to test with libcontainer=true and I did. With that libcontianer library created cgroup for the VM but again in the wrong place. It landed in /system.slice/kubepods-pod<uid>.slice:cri-containerd:<id>, not inside kubepods.slice/…. It redirects us again to #1105 and solving that would fix this too. Also, it led to another discovery in pkg/unikontainers/libcontainer_cgo.go:67, urunc replaces the whole resources section with a new one that only says "allow all devices". That throws away the memory and CPU limits, and VM now has memory.max=max and cpu.max=max. So I guess worth opening another issue. Note: before opening an issue wrote small patch but it led to OOM, so it seems a deliberate design choice. But maybe from inspiring from kata we can introduce overhead I added my findings in #127 as a comment, given this will be something maintainers work on.
|
Below is the ArgoCon Europe draft. Currently, @cmainas and I have edit access. If anyone else requires access, please send a request. CFP deadline: Sunday, 18 October 2026, at 11:59pm Central European Time (UTC+2) https://docs.google.com/document/d/1TGLaSP5rMxJQBxVSQHzL1fPkX7qyilYvnUOF_veuIwg/edit?usp=sharing |
Design document describing the integration
1. POC for #135 hang (urunc v0.8.0 Linux virtiofs guest)
In urunc v0.8.0, the Linux guest can boot in 3 ways basically,
As we discussed at the starting point, I'm focusing on shared fs because in block or initrd, urunc shares no host directories with the guest, which will make POC bit complicated. For logging I'm using the same logging stack I used before, with some changes to
So the final apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: experiment-
spec:
entrypoint: demo
podSpecPatch: '{"runtimeClassName":"urunc","enableServiceLinks":false}'
templates:
- name: demo
script:
image: docker.io/urunc-argo/python-linux-qemu:135
imagePullPolicy: Never
command: [python]
source: |
import sys, time
print("EXPERIMENT-BEGIN", file=sys.stderr, flush=True)
time.sleep(300)
print("42")Three exit codes were coming from
Overall in urunc v0.8.0 Linux virtiofs guest #135 hang does not happen. It occurs in older versions plus initrd and block file system only. There are 3 new hangs in shared fs. Observing the process tree helps us understand what's exactly wrong.
There could be more things to work on but above two has to sorted first IMO... So currently im working on it 2. POC for #135 hang (urunc v0.8.0 Unikernel guest with initrd-based rootfs)
3. POC for #135 hang (urunc v0.8.0 Unikernel guest with Block-based rootfs)
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Mentorship plan: Integration of urunc's sandbox execution with Argo
This discussion outlines the proposed plan for the CNCF mentorship project "Integration of urunc's sandbox execution with Argo"”.
The plan is structured into three phases. Each phase has clear goals and specific outcomes, along with suggested tasks and sub-tasks to help guide the work. The listed sub-tasks are meant as guidance and reference points, they are not strict requirements. The exact order of tasks within each phase can be adjusted as long as the main outcomes are achieved.
Phase 1
The goal of Phase 1 is to build the necessary background on both sides of the problem: the architecture and execution flow of Argo, and the execution model of urunc. This phase is expected to last up to 3 weeks (07/09/2026 – 27/09/2026). During this time, the focus will be on understanding how Argo turns a workflow into pods and how those pods coordinate, understanding which of these assumptions urunc breaks and scoping all the necessary fixes.
Tentative tasks and sub-tasks
initcontainer, thewaitsidecar and themaincontainer, and the role of theargoexecbinary in each of themOutcome
Deadline: Completed no later than 27/09/2026 (AoE).
Description: A report with:
Phase 2
The goal of Phase 2 is to design the integration and implement a PoC with the necessary patches in urunc. This phase is expected to last up to 5 weeks (28/09/2026 – 01/11/2026). During this time, the focus will shift from exploration to a concrete design, agreed with the maintainers, and to the implementation of the core parts of the integration: the lifecycle and signalling between the sandbox and the rest of the workflow pod.
Tentative tasks and sub-tasks
waitsidecar: what signals the completion of a step, where the exit status is published, and which ordering guarantees are required.Outcomes
Deadline: Completed no later than 01/11/2026 (AoE).
Description: The following items:
Phase 3
The goal of Phase 3 is to complete the integration, finalize the changes, create the necessary documentation and perform some basic evaluation. This phase is expected to last up to 4 weeks (02/11/2026 – 27/11/2026). During this time, the focus will be on the necessary testing, tutorials and evaluation comparing the normal vs "urunc's" execution model.
Tentative tasks and sub-tasks
Outcomes
Deadline: Completed no later than 27/11/2026 (AoE).
Description: The following items:
All reactions