Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check FileInfo against nil during walk of container dir path #86277

Merged
merged 1 commit into from Dec 17, 2019

Conversation

@tedyu
Copy link
Contributor

tedyu commented Dec 15, 2019

What type of PR is this?
/kind bug

What this PR does / why we need it:
When checking kubelet log under https://gcsweb.k8s.io/gcs/kubernetes-jenkins/pr-logs/pull/86271/pull-kubernetes-e2e-gce/1205989922050150403/artifacts/e2e-41a80d25d0-674b9-minion-group-vrmw/ , I saw the following:

I1214 23:54:03.413009    1331 operation_generator.go:713] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7525fe3b-e6fa-4d02-87df-2a08ff0c9f50-default-token-rssv4" (OuterVolumeSpecName: "default-token-rssv4") pod "7525fe3b-e6fa-4d02-87df-2a08ff0c9f50" (UID: "7525fe3b-e6fa-4d02-87df-2a08ff0c9f50"). InnerVolumeSpecName "default-token-rssv4". PluginName "kubernetes.io/secret", VolumeGidValue ""
E1214 23:54:03.413115    1331 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/nfs/8e4f2439-71a4-4102-a049-ba0bb42378ce-pvc-e2e1b57c-fdc0-46d9-a47d-d7ca4555e1c7\" (\"8e4f2439-71a4-4102-a049-ba0bb42378ce\")" failed. No retries permitted until 2019-12-14 23:54:03.912383421 +0000 UTC m=+1573.953851145 (durationBeforeRetry 500ms). Error: "recovered from panic \"runtime error: invalid memory address or nil pointer dereference\". (err=<nil>) Call stack:
goroutine 118595 [running]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.RecoverFromPanic(0xc002b0c310)
\tstaging/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:158 +0xb5
panic(0x3a2ad60, 0x6d31df0)
\tGOROOT/src/runtime/panic.go:679 +0x1b2
k8s.io/kubernetes/pkg/volume/util/subpath.doCleanSubPaths.func1(0xc002930f00, 0x9f, 0x0, 0x0, 0x497d8a0, 0xc000b34150, 0x0, 0x0)
\tpkg/volume/util/subpath/subpath_linux.go:243 +0xbd
path/filepath.walk(0xc002930be0, 0x9d, 0x4a37a00, 0xc001f2a270, 0xc000ab3ba8, 0x0, 0x9d)
\tGOROOT/src/path/filepath/path.go:378 +0x20c

This PR adds check against incoming error before dereferencing FileInfo

Which issue(s) this PR fixes:
Fixes #

Fixed a panic in the kubelet cleaning up pod volumes

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Dec 15, 2019

/test pull-kubernetes-e2e-gce

1 similar comment
@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Dec 15, 2019

/test pull-kubernetes-e2e-gce

@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Dec 15, 2019

/priority backlog

@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Dec 16, 2019

/cc @saad-ali

@k8s-ci-robot k8s-ci-robot requested a review from saad-ali Dec 16, 2019
@yutedz yutedz force-pushed the yutedz:subpath-walk-nil branch from 527a9e4 to 891755a Dec 16, 2019
@tedyu tedyu changed the title Check FileInfo against nil during walk of container dir path Check incoming error during walk of container dir path Dec 16, 2019
@yutedz yutedz force-pushed the yutedz:subpath-walk-nil branch from 891755a to 4975216 Dec 16, 2019
@liggitt liggitt added this to the v1.17 milestone Dec 16, 2019
@yutedz yutedz force-pushed the yutedz:subpath-walk-nil branch from 4975216 to 3693cf7 Dec 16, 2019
@tedyu tedyu changed the title Check incoming error during walk of container dir path Check FileInfo against nil during walk of container dir path Dec 16, 2019
@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Dec 16, 2019

/test pull-kubernetes-e2e-gce

@yutedz yutedz force-pushed the yutedz:subpath-walk-nil branch from 3693cf7 to 62d7772 Dec 17, 2019
@msau42

This comment has been minimized.

Copy link
Member

msau42 commented Dec 17, 2019

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label Dec 17, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Dec 17, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: msau42, tedyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Dec 17, 2019

thanks. go ahead and cherrypick to release-1.17 as well to get CI running there.

@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Dec 17, 2019

/test pull-kubernetes-e2e-gce-storage-slow

@k8s-ci-robot k8s-ci-robot merged commit a57a8f3 into kubernetes:master Dec 17, 2019
18 checks passed
18 checks passed
cla/linuxfoundation yutedz authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-csi-serial Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-storage-slow Job succeeded.
Details
pull-kubernetes-e2e-gce-storage-snapshot Job succeeded.
Details
pull-kubernetes-e2e-kind Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
@k8s-ci-robot k8s-ci-robot modified the milestones: v1.17, v1.18 Dec 17, 2019
k8s-ci-robot added a commit that referenced this pull request Dec 25, 2019
…-upstream-release-1.17

[v1.17.1] cherry pick of #86277: Check FileInfo against nil during walk of container dir path
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.