125 Error Pertaining to Storage Shutdown

[andygodish]

Question

I'm running some self-hosted github runners on an OpenShift cluster based on the configurations in this repo. More specifically, using Dockerfiles cloned from the base and buildah Containerfiles.

A couple of existing app workflows that are now targeting these new self-hosted runners are failing during the build/push steps when trying to execute buildah commands in the runners with the following found in the logs:

chown /home/runner/.local/share/containers/storage/vfs: operation not permitted
time="2022-10-25T16:19:20Z" level=warning msg="failed to shutdown storage: \"chown /home/runner/.local/share/containers/storage/vfs: operation not permitted\""
time="2022-10-25T16:19:20Z" level=error msg="exit status 125"
Error: Error: buildah exited with code 12

I deconstructed one of these workflows, and this seems to crop up when workflow steps are trying to use the public redhat github actions. Specifically, these two:

uses: redhat-actions/buildah-build@v2
---
uses: redhat-actions/push-to-registry@v2

Troubleshooting

I can exec into the runner pod after encountering this 125 error and see that all buildah commands cannot be run. For example, buildah ps will result in a failure and the following output:

chown /home/runner/.local/share/containers/storage/vfs: operation not permitted
WARN[0000] failed to shutdown storage: "chown /home/runner/.local/share/containers/storage/vfs: operation not permitted

I can fix the issue by force deleting that .local directory or by performing a rollout restart of my self-hosted runner since that .local directory only exists following your first build.

I'm guessing that the public action that is being pulled in is somehow interfering with the configurations set in the buildah container image. Maybe it is overriding the vfs configuration being set here. Looking to get a better understanding of the configurations used by buildah when running in an OpenShift pod.

Thanks!