Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman: workaround race during container creation #1659

Merged
merged 1 commit into from
Jun 16, 2021
Merged

podman: workaround race during container creation #1659

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
32 changes: 30 additions & 2 deletions heartbeat/podman
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,8 +358,18 @@ run_new_container()
local rc

ocf_log info "running container $CONTAINER for the first time"
ocf_run podman run $opts $image $cmd
out=$(podman run $opts $image $cmd 2>&1)
rc=$?

if [ -n "$out" ]; then
out="$(echo "$out" | tr -s ' \t\r\n' ' ')"
if [ $rc -eq 0 ]; then
ocf_log info "$out"
else
ocf_log err "$out"
fi
fi

if [ $rc -eq 125 ]; then
# If an internal podman error occurred, it might be because
# the internal storage layer still references an old container
Expand All @@ -370,6 +380,24 @@ run_new_container()
ocf_run podman rm --storage $CONTAINER
ocf_run podman run $opts $image $cmd
rc=$?
elif [ $rc -eq 127 ]; then
# rhbz#1972209: podman 3.0.x seems to be hit by a race
# where the cgroup is not yet set up properly when the OCI
# runtime configures the container. If that happens, recreate
# the container as long as we get the same error code or
# until start timeout preempts us.
while [ $rc -eq 127 ] && (echo "$out" | grep -q "cgroup.*scope not found") ; do
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we have some limit here, to prevent endless loop? It shouldn't happen, but some more security wouldn't hurt.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pacemaker itself preempts this infinite loop as soon as the start operation timeout is reached. This is usually the idiom we use in the resource agents to loop while we're allowed.

ocf_log warn "Internal podman error while assigning cgroup. Retrying."
# Arbitrary sleep to prevent consuming all CPU while looping
sleep 1
podman rm -f "$CONTAINER"
out=$(podman run $opts $image $cmd 2>&1)
rc=$?
done
# Log the created container ID if it succeeded
if [ $rc -eq 0 ]; then
ocf_log info "$out"
fi
fi

return $rc
Expand Down Expand Up @@ -422,7 +450,7 @@ podman_start()
fi

if [ $rc -ne 0 ]; then
ocf_exit_reason "podman failed to launch container"
ocf_exit_reason "podman failed to launch container (rc: $rc)"
return $OCF_ERR_GENERIC
fi

Expand Down