Pod main containers finished but init container is still waiting for resource #41580

Open
zhan849 opened this Issue Feb 16, 2017 · 0 comments

Projects

None yet

1 participant

@zhan849
zhan849 commented Feb 16, 2017

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):
No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):
"kubernetes pod main container finished but init container is pending"

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3", GitCommit:"4957b090e9a4f6a68b4a40375408fdc74a212260", GitTreeState:"clean", BuildDate:"2016-10-16T06:36:33Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.3-26+09efe457e9341d", GitCommit:"09efe457e9341d2552b3089cdd465bfd33bf78b4", GitTreeState:"clean", BuildDate:"2017-02-01T05:38:53Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: aws m3.2xlarge instance
  • OS (e.g. from /etc/os-release): debian 8
  • Kernel (e.g. uname -a): 4.41
  • Install tools: kube-up
  • Others:

What happened:
We have a pod which has state inconsistency:
main containers are finished but one of the init containers is pending

$ kubectl describe pod saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020-gji21
Name:		saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020-gji21
Namespace:	axuser
Node:		ip-10-144-6-124.us-west-2.compute.internal/10.144.6.124
Start Time:	Thu, 16 Feb 2017 05:03:12 +0000
Labels:		app=saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020
		controller-uid=bb30999d-f404-11e6-aece-02eb5a7ee587
		job-name=saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020
		name=saas-unit-test-base
		role=user
		root_workflow_id=7e25ca1e-f404-11e6-8e68-0a58c0a88020
		service_instance_id=7e25d304-f404-11e6-8e72-0a58c0a88020
		tier=devops
Status:		Pending
IP:
Controllers:	Job/saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020
Init Containers:
  axpull:
    Container ID:	docker://33cebb2248dbc41021f157d64cf13a5ff083985876c1bb16fbc068cee72a6924
    Image:		get.applatix.io/axdb-dev:v1.9
    Image ID:		docker://sha256:214698d2b1a50c7dcf7ce532dfffa1be3ae07395b0db961a7b314632d4762b37
    Port:
    Command:
      /bin/true
    Requests:
      cpu:		10m
      memory:		100Mi
    State:		Terminated
      Reason:		Completed
      Exit Code:	0
      Started:		Thu, 16 Feb 2017 04:59:47 +0000
      Finished:		Thu, 16 Feb 2017 04:59:47 +0000
    Ready:		True
    Restart Count:	0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7fowl (ro)
    Environment Variables:	<none>
  axinit:
    Container ID:
    Image:		get.applatix.io/axsys/artifacts:1.6.1-d638331
    Image ID:
    Port:
    Args:
      pre
    Requests:
      cpu:		10m
      memory:		400Mi
    State:		Waiting
      Reason:		PodInitializing
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /ax-artifacts-scratch from artifacts-scratch (rw)
      /ax-artifacts-scratch/in/0 from ax-art-0 (rw)
      /copyto from static-bins (rw)
      /etc/axspec from annotations (rw)
      /var/run/docker.sock from docker-socket-file (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7fowl (ro)
    Environment Variables:
      AX_CUSTOMER_IMAGE_NAME:	get.applatix.io/axdb-dev:v1.9
      AX_POD_NAME:		saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020-gji21 (v1:metadata.name)
      AX_POD_IP:		 (v1:status.podIP)
      AX_ARTIFACTS_SCRATCH:	/ax-artifacts-scratch
      AX_JOB_NAME:		saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020
Containers:
  axsidekickwait:
    Container ID:	docker://d349d4c081192c1ed8fa3b1342a86c156dfde34a5dd4ea1db3c64b413bed494f
    Image:		get.applatix.io/axsys/artifacts:1.6.1-d638331
    Image ID:		docker://sha256:2175e5038cf5f888925a56562aadd0928f2ee26caedd3d23f10359d84f01c015
    Port:
    Args:
      post
    Requests:
      cpu:		10m
      memory:		200Mi
    State:		Terminated
      Reason:		Error
      Exit Code:	137
      Started:		Thu, 16 Feb 2017 05:00:01 +0000
      Finished:		Thu, 16 Feb 2017 05:03:40 +0000
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /ax-artifacts-scratch from artifacts-scratch (rw)
      /copyto from static-bins (rw)
      /etc/axspec from annotations (rw)
      /logs from containerlogs (rw)
      /var/run/docker.sock from docker-socket-file (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7fowl (ro)
    Environment Variables:
      AX_CUSTOMER_ID:		8be85aec-c7de-11e6-a65f-0234d974d1bf
      AX_ARTIFACTS_SCRATCH:	/ax-artifacts-scratch
      AX_MAIN_CONTAINER_NAME:	saas-unit-test-base
      AX_JOB_NAME:		saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020
      LOGMOUNT_PATH:		/logs
      AX_POD_NAME:		saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020-gji21 (v1:metadata.name)
      AX_CUSTOMER_IMAGE_NAME:	saas-unit-test-base
      AX_REGION:		us-west-2
      AX_POD_IP:		 (v1:status.podIP)
  saas-unit-test-base:
    Container ID:	docker://1abb504c00690a677e1ecb4e726b522b1b889771d03e7174ba4ed9f710fa045f
    Image:		get.applatix.io/axdb-dev:v1.9
    Image ID:		docker://sha256:214698d2b1a50c7dcf7ce532dfffa1be3ae07395b0db961a7b314632d4762b37
    Port:
    Command:
      /ax-artifacts-scratch/executor.sh
    Limits:
      memory:	1536Mi
    Requests:
      cpu:		1
      memory:		1536Mi
    State:		Terminated
      Reason:		Completed
      Exit Code:	0
      Started:		Thu, 16 Feb 2017 05:00:02 +0000
      Finished:		Thu, 16 Feb 2017 05:02:09 +0000
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /ax-artifacts-scratch from artifacts-scratch (rw)
      /ax-execu-host from static-bins (rw)
      /src from ax-art-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7fowl (ro)
    Environment Variables:
      AX_CONTAINER_NAME:		saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020
      AX_ROOT_SERVICE_INSTANCE_ID:	7e25ca1e-f404-11e6-8e68-0a58c0a88020
      AX_SERVICE_INSTANCE_ID:		7e25d304-f404-11e6-8e72-0a58c0a88020
Conditions:
  Type		Status
  Initialized 	False
  Ready 	False
  PodScheduled 	True
Volumes:
  artifacts-scratch:
    Type:	EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  static-bins:
    Type:	EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  docker-socket-file:
    Type:	HostPath (bare host directory volume)
    Path:	/var/run/docker.sock
  ax-art-0:
    Type:	EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  annotations:
    Type:	DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  containerlogs:
    Type:	HostPath (bare host directory volume)
    Path:	/mnt/ephemeral/docker/containers
  default-token-7fowl:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-7fowl
QoS Class:	Burstable
Tolerations:	<none>
No events.

What you expected to happen:
main container should not even start if init container is not yet started

How to reproduce it (as minimally and precisely as possible):
this happens in our stress test, haven't figured out how to stably reproduce

Anything else we need to know:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment