k8s upgrade resulted in executor error from gVisor

[rconjaerts]

Hi!

A colleague upgraded our k8s cluster/nodes in GCP before leaving on holiday and since he did this our argo jobs aren't running anymore. I tried upgrading argo-workflows and argo-events, but that didn't do anything, and I can't seem to revert the k8s cluster/nodes to the old version in GCP.

I'm pretty sure I have to change something in k8s, to make it work for argo, and hope you might know what my next steps should be.

The error I'm getting when running a job is the following:

starting container: starting sub-container [/var/run/argo/argoexec emissary --loglevel info --log-format text -- node --unhandled-rejections=strict --max-old-space-size=384 index.js]: failed to setupFS: mounting submounts: mount shared mount "var-run-argo" to "/var/run/argo": creating mount point "/var/run/argo": failed to create parent directory of mountpoint "/var/run/argo": not a directory

If there's any other information I can provide I will, not sure what is relevant in this case.

Thanks!
Rob

[rconjaerts]

For future reference, we were using gvisor in our k8s node pool, and since upgrading to version 1.29 this gave issues for argo because in our workflow we also had:
podSpecPatch: '{"runtimeClassName": "gvisor"}'

This has been removed, and we created a new node pool without using gvisor.