New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pods stay in Pending for a long time #70835

Open
liqlin2015 opened this Issue Nov 9, 2018 · 1 comment

Comments

Projects
None yet
2 participants
@liqlin2015

liqlin2015 commented Nov 9, 2018

What happened:

Pods stay in Pending status for a long time.

Here is my pod

# kubectl -n services get pods -o wide
NAME                                        READY     STATUS    RESTARTS   AGE       IP        NODE
redis-65dcfbfccc-dtk8x                      0/1       Pending   0          19h       <none>    9.193.199.43

And from kubelet log

Nov  8 11:20:11 inmbzicpsanw08 hyperkube[1599]: I1108 11:20:11.726517    1599 config.go:405] Receiving a new pod "redis-65dcfbfccc-dtk8x_services(2826d997-e31a-11e8-b5cb-005056bf37ac)"
Nov  9 06:25:32 inmbzicpsanw08 hyperkube[1599]: I1109 06:25:32.408191    1599 kubelet.go:1861] SyncLoop (ADD, "api"): "redis-65dcfbfccc-dtk8x_services(2826d997-e31a-11e8-b5cb-005056bf37ac)"
Nov  9 06:25:32 inmbzicpsanw08 hyperkube[1599]: I1109 06:25:32.493868    1599 kubelet_pods.go:1375] Generating status for "redis-65dcfbfccc-dtk8x_services(2826d997-e31a-11e8-b5cb-005056bf37ac)"
Nov  9 06:25:32 inmbzicpsanw08 hyperkube[1599]: I1109 06:25:32.504139    1599 volume_manager.go:347] Waiting for volumes to attach and mount for pod "redis-65dcfbfccc-dtk8x_services(2826d997-e31a-11e8-b5cb-005056bf37ac)"
Nov  9 06:25:34 inmbzicpsanw08 hyperkube[1599]: I1109 06:25:34.306831    1599 volume_manager.go:380] All volumes are attached and mounted for pod "redis-65dcfbfccc-dtk8x_services(2826d997-e31a-11e8-b5cb-005056bf37ac)"
Nov  9 06:25:35 inmbzicpsanw08 hyperkube[1599]: I1109 06:25:35.476785    1599 kuberuntime_manager.go:385] No sandbox for pod "redis-65dcfbfccc-dtk8x_services(2826d997-e31a-11e8-b5cb-005056bf37ac)" can be found. Need to start a new one
Nov  9 06:25:35 inmbzicpsanw08 hyperkube[1599]: I1109 06:25:35.476823    1599 kuberuntime_manager.go:570] computePodActions got {KillPod:true CreateSandbox:true SandboxID: Attempt:0 NextInitContainerToStart:nil ContainersToStart:[0] ContainersToKill:map[]} for pod "redis-65dcfbfccc-dtk8x_services(2826d997-e31a-11e8-b5cb-005056bf37ac)"
Nov  9 06:25:35 inmbzicpsanw08 hyperkube[1599]: I1109 06:25:35.476890    1599 kuberuntime_manager.go:579] SyncPod received new pod "redis-65dcfbfccc-dtk8x_services(2826d997-e31a-11e8-b5cb-005056bf37ac)", will create a sandbox for it

As you seen from kubelet log, kubelet get the pods at Nov 8 11:20:11, but start the sync loop at Nov 9 06:25:32. Why takes so long for kubelet to take action? Sounds like the configCh https://github.com/kubernetes/kubernetes/blob/v1.10.0/pkg/kubelet/kubelet.go#L1846 is hanging at that time, kubelet main sync up thread did not detect events from that channel.

What you expected to happen:

After scheduled Pods should be created by kubelet immediately.

How to reproduce it (as minimally and precisely as possible):

Not sure.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): 1.10.0
  • Cloud provider or hardware configuration: None
  • OS (e.g. from /etc/os-release): Ubuntu 16.04
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

/kind bug

@liqlin2015

This comment has been minimized.

liqlin2015 commented Nov 9, 2018

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node and removed needs-sig labels Nov 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment