Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: remove stream copy infinite loop #9367

Conversation

danmihai1
Copy link
Contributor

@danmihai1 danmihai1 commented Mar 28, 2024

This reverts commit 1c5693b.

Avoid apparent infinite loop when ReadStreamRequest is blocked by policy - for some of the pods.

When running the k8s-limit-range.bats test with Policy enabled, the Shim + VMM never get terminated on my cluster. Not sure why the sandbox clean-up works better for other tests, but the k8s-limit-range test pod gets stuck in an infinite loop:

stdout io stream copy error happens: error = %wrpc error: code =
PermissionDenied desc = \"ReadStreamRequest is blocked by policy

...

policy check: ReadStreamRequest

...

stdout io stream copy error happens: error = %wrpc error: code =
PermissionDenied desc = \"ReadStreamRequest is blocked by policy

...

policy check: ReadStreamRequest

...

Fixes: #9380

@katacontainersbot katacontainersbot added the size/small Small and simple task label Mar 28, 2024
@danmihai1 danmihai1 added do-not-merge PR has problems or depends on another wip Work in Progress (PR incomplete - needs more work or rework) no-backport-needed ok-to-test and removed ok-to-test labels Mar 28, 2024
@danmihai1 danmihai1 force-pushed the danmihai1/infinite-io-stream-copy-loop branch from 494baf5 to a8955ef Compare March 28, 2024 22:34
@danmihai1 danmihai1 removed do-not-merge PR has problems or depends on another wip Work in Progress (PR incomplete - needs more work or rework) labels Mar 28, 2024
@danmihai1 danmihai1 requested a review from fidencio March 28, 2024 22:35
This reverts commit 1c5693b.

Avoid apparent infinite loop when ReadStreamRequest is blocked by
policy - for some of the pods.

When running the k8s-limit-range.bats test with Policy enabled,
the Shim + VMM never get terminated on my cluster. Not sure why
the sandbox clean-up works better for other tests, but the
k8s-limit-range test pod gets stuck in an infinite loop:

stdout io stream copy error happens: error = %wrpc error: code =
PermissionDenied desc = \"ReadStreamRequest is blocked by policy

...

policy check: ReadStreamRequest

...

stdout io stream copy error happens: error = %wrpc error: code =
PermissionDenied desc = \"ReadStreamRequest is blocked by policy

...

policy check: ReadStreamRequest

...

Fixes: kata-containers#9380

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
@danmihai1 danmihai1 force-pushed the danmihai1/infinite-io-stream-copy-loop branch from a8955ef to 600f926 Compare March 28, 2024 22:43
@danmihai1 danmihai1 changed the title runtime: break from stream copy loop runtime: remove stream copy infinite loop Mar 28, 2024
Copy link
Member

@fidencio fidencio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks @danmihai1!

Copy link
Member

@ChengyuZhu6 ChengyuZhu6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks @danmihai1 !

@danmihai1
Copy link
Contributor Author

/test

@danmihai1 danmihai1 merged commit 3a7dbcf into kata-containers:main Mar 29, 2024
313 of 385 checks passed
@danmihai1 danmihai1 deleted the danmihai1/infinite-io-stream-copy-loop branch March 29, 2024 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ok-to-test size/small Small and simple task
Projects
None yet
Development

Successfully merging this pull request may close these issues.

runtime: occasional Shim + VMM leak while running Kata CI tests for k8s
4 participants