New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

heapster Failed create pod sandbox. #55672

Closed
KeithTt opened this Issue Nov 14, 2017 · 2 comments

Comments

Projects
None yet
3 participants
@KeithTt

KeithTt commented Nov 14, 2017

kube version: 18.2
docker version: 17.06.2-ce
OS version: debian stretch

I scale the replicas of heapster from 1 to 3, COZ there is only one node's metrics can be displayed on the dashboard, and the other two pods pending:

# kubectl get pod --all-namespaces
NAMESPACE     NAME                                    READY     STATUS              RESTARTS   AGE
kube-system   contiv-api-proxy-d29m7                  1/1       Running             1          1h
kube-system   contiv-etcd-5rwb6                       1/1       Running             0          1h
kube-system   contiv-etcd-q549f                       1/1       Running             0          1h
kube-system   contiv-netmaster-xj65z                  1/1       Running             0          1h
kube-system   contiv-netplugin-4wjfh                  1/1       Running             0          1h
kube-system   contiv-netplugin-bc75r                  1/1       Running             0          1h
kube-system   contiv-netplugin-r4cpz                  1/1       Running             0          1h
kube-system   heapster-59ff54b574-7q8pn               0/1       ContainerCreating   0          29m
kube-system   heapster-59ff54b574-jd4x8               1/1       Running             0          1d
kube-system   heapster-59ff54b574-sh9zr               0/1       ContainerCreating   0          29m
kube-system   kube-apiserver-uy05-13                  1/1       Running             0          1d
kube-system   kube-apiserver-uy08-07                  1/1       Running             0          1d
kube-system   kube-controller-manager-uy05-13         1/1       Running             0          1d
kube-system   kube-controller-manager-uy08-07         1/1       Running             0          1d
kube-system   kube-dns-545bc4bfd4-b9njt               3/3       Running             0          1d
kube-system   kube-proxy-bbssx                        1/1       Running             0          1d
kube-system   kube-proxy-xl5m6                        1/1       Running             0          1d
kube-system   kube-proxy-zzkvq                        1/1       Running             0          1d
kube-system   kube-scheduler-uy05-13                  1/1       Running             0          1d
kube-system   kube-scheduler-uy08-07                  1/1       Running             0          1d
kube-system   kubernetes-dashboard-57f7b78668-jcsq5   1/1       Running             0          23h
Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               28m                  default-scheduler  Successfully assigned heapster-59ff54b574-sh9zr to uy08-07
  Normal   SuccessfulMountVolume   28m                  kubelet, uy08-07   MountVolume.SetUp succeeded for volume "heapster-token-pwx75"
  Warning  FailedSync              28m (x8 over 28m)    kubelet, uy08-07   Error syncing pod
  Normal   SandboxChanged          18m (x354 over 28m)  kubelet, uy08-07   Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  3m (x874 over 28m)   kubelet, uy08-07   Failed create pod sandbox.
# kubectl get po heapster-59ff54b574-7q8pn -n kube-system -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/created-by: |
      {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"heapster-59ff54b574","uid":"46cb0998-c828-11e7-9459-34e6d7899e5d","apiVersion":"extensions","resourceVersion":"187948"}}
  creationTimestamp: 2017-11-14T07:14:52Z
  generateName: heapster-59ff54b574-
  labels:
    k8s-app: heapster
    pod-template-hash: "1599106130"
    task: monitoring
  name: heapster-59ff54b574-7q8pn
  namespace: kube-system
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: heapster-59ff54b574
    uid: 46cb0998-c828-11e7-9459-34e6d7899e5d
  resourceVersion: "187960"
  selfLink: /api/v1/namespaces/kube-system/pods/heapster-59ff54b574-7q8pn
  uid: 822621d9-c90b-11e7-9459-34e6d7899e5d
spec:
  containers:
  - command:
    - /heapster
    - --source=kubernetes:https://kubernetes.default
    - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
    image: harbor.oupeng.com/heapster/heapster-amd64:v1.4.3
    imagePullPolicy: IfNotPresent
    name: heapster
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: heapster-token-pwx75
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: uy08-08
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: heapster
  serviceAccountName: heapster
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.alpha.kubernetes.io/notReady
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.alpha.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: heapster-token-pwx75
    secret:
      defaultMode: 420
      secretName: heapster-token-pwx75
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2017-11-14T07:14:52Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2017-11-14T07:14:52Z
    message: 'containers with unready status: [heapster]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2017-11-14T07:14:52Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: harbor.oupeng.com/heapster/heapster-amd64:v1.4.3
    imageID: ""
    lastState: {}
    name: heapster
    ready: false
    restartCount: 0
    state:
      waiting:
        reason: ContainerCreating
  hostIP: 192.168.5.105
  phase: Pending
  qosClass: BestEffort
  startTime: 2017-11-14T07:14:52Z

one more thing, I change the network plugin from calico to contiv, does it have anthing to do with the problem?

@KeithTt

This comment has been minimized.

Show comment
Hide comment
@KeithTt

KeithTt Nov 14, 2017

/sig scalability

KeithTt commented Nov 14, 2017

/sig scalability

@KeithTt

This comment has been minimized.

Show comment
Hide comment
@KeithTt

KeithTt Nov 19, 2017

A configuration error...close it.

KeithTt commented Nov 19, 2017

A configuration error...close it.

@KeithTt KeithTt closed this Nov 19, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment