Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Errors from cgroup destroy are swallowed. Log error at warning level. #84599

Merged
merged 1 commit into from May 17, 2020
Merged

Errors from cgroup destroy are swallowed. Log error at warning level. #84599

merged 1 commit into from May 17, 2020

Conversation

mrobson
Copy link
Contributor

@mrobson mrobson commented Oct 31, 2019

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
Housekeeping sync loop finds an orphaned pod and tries to clean it up. The removal of the kubepod slice may fail. This leads to the docker scope timing out and entering a failed state. The returned error(s) from runc are swallowed making debugging difficult.

Which issue(s) this PR fixes:
Fixes #84595

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 31, 2019
@k8s-ci-robot
Copy link
Contributor

Hi @mrobson. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 31, 2019
@mrobson
Copy link
Contributor Author

mrobson commented Oct 31, 2019

/assign @vishh

@tedyu
Copy link
Contributor

tedyu commented Oct 31, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 31, 2019
mattjmcnaughton
mattjmcnaughton reviewed Nov 1, 2019
View reviewed changes
Copy link
Contributor

@mattjmcnaughton mattjmcnaughton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/ok-to-test

Thanks for your work here!

pkg/kubelet/cm/pod_container_manager_linux.go Show resolved Hide resolved
@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 1, 2019
@mrobson
Copy link
Contributor Author

mrobson commented Nov 1, 2019

/retest

@mattjmcnaughton
Copy link
Contributor

mattjmcnaughton commented Nov 6, 2019 via email

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 7, 2019
@mrobson
Copy link
Contributor Author

mrobson commented Nov 7, 2019

@mattjmcnaughton @tedyu updates pushed with warnings for both. PTAL thanks!

@tedyu
Copy link
Contributor

tedyu commented Nov 7, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 7, 2019
mattjmcnaughton
mattjmcnaughton approved these changes Nov 7, 2019
View reviewed changes
Copy link
Contributor

@mattjmcnaughton mattjmcnaughton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@mrobson thank you for your patience during the discussion :)
@tedyu thank you for your insights, as always :)

@mrobson
Copy link
Contributor Author

mrobson commented May 13, 2020

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 13, 2020
@mrobson mrobson changed the title Errors from cgroup destroy are swallowed. Log error at debug level. Errors from cgroup destroy are swallowed. Log error at warning level. May 13, 2020
@mrobson
Copy link
Contributor Author

mrobson commented May 13, 2020

/assign @tallclair hi Tim - this kind of stalled out, can you PTAL? - this is a follow up to #83625

@derekwaynecarr
Copy link
Member

/approve

just noted for future: we call go pcm.Destroy() so the returned error is never logged.

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, mrobson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 13, 2020
@mrobson
Copy link
Contributor Author

mrobson commented May 13, 2020

/test pull-kubernetes-files-remake

@mrobson
Copy link
Contributor Author

mrobson commented May 13, 2020

/test pull-kubernetes-conformance-kind-ga-only-parallel

1 similar comment
@mrobson
Copy link
Contributor Author

mrobson commented May 13, 2020

/test pull-kubernetes-conformance-kind-ga-only-parallel

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

1 similar comment
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@mrobson
Copy link
Contributor Author

mrobson commented May 14, 2020

/retest

1 similar comment
@mrobson
Copy link
Contributor Author

mrobson commented May 14, 2020

/retest

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

4 similar comments
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@mrobson
Copy link
Contributor Author

mrobson commented May 15, 2020

/test pull-kubernetes-e2e-kind

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

7 similar comments
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit f011430 into kubernetes:master May 17, 2020
@k8s-ci-robot k8s-ci-robot added this to the v1.19 milestone May 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tedyu tedyu tedyu left review comments

@mattjmcnaughton mattjmcnaughton mattjmcnaughton approved these changes

@feiskyer feiskyer Awaiting requested review from feiskyer

Assignees

@tedyu tedyu

@ConnorDoyle ConnorDoyle

@mattjmcnaughton mattjmcnaughton

@vishh vishh

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note-none Denotes a PR that doesn't merit a release note. sig/node Categorizes an issue or PR as relevant to SIG Node. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.19
Development

Successfully merging this pull request may close these issues.

Errors returned from pod_container_manager_linux Destroy to remove a pods cgroup are swallowed
9 participants
@mrobson @k8s-ci-robot @tedyu @mattjmcnaughton @dims @fejta-bot @derekwaynecarr @ConnorDoyle @vishh