Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deleting deployment does not delete pods associated with it #32985

Closed
DImuthuUpe opened this issue Sep 18, 2016 · 5 comments
Closed

Deleting deployment does not delete pods associated with it #32985

DImuthuUpe opened this issue Sep 18, 2016 · 5 comments
Labels
area/app-lifecycle area/workload-api/deployment sig/apps Categorizes an issue or PR as relevant to SIG Apps.

Comments

@DImuthuUpe
Copy link

DImuthuUpe commented Sep 18, 2016 •
edited
Loading

Kubernetes version (use kubectl version): v1.2.6

Environment:

  • Cloud provider or hardware configuration: Virtual Machine with 4GB physical memory and 2 cpu cores
  • OS (e.g. from /etc/os-release): Ubuntu 14.04
  • Kernel (e.g. uname -a): Linux ubuntu-m1 3.13.0-95-generic When the apiserver panics, log a stacktrace. #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: Manual (self created ansible scripts)
  • Others:

What happened:
I have created a deployment with 3 pods and tried to delete the deployment using REST API. Deployment object and associated replica sets were deleted but when I see 'kubectl get pods' output, I could see that there are 3 pods remaining which were created from above deployment.

Deployment object name : ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad
Remaining pod names:
ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196-6zn50
ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196-ms3fz
ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196-nctq2

I checked kube-scheduler log and what I noticed is that scheduler is creating a new replica set (ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196) while it was deleting other replica sets for the deployment. Ultimately it deleted above replica set but it missed to delete pods created from the replica set.

I0914 15:08:11.436331     804 event.go:211] Event(api.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad", UID:"f39f3c6b-7a5e-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"712429", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-553526050 to 2
E0914 15:08:11.455148     804 deployment_controller.go:395] Error syncing deployment default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad: deployments.extensions "ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad" cannot be updated: the object has been modified; please apply your changes to the latest version and try again
I0914 15:08:11.476871     804 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-553526050", UID:"f3a02ff8-7a5e-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"712430", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-553526050-aqc9n
I0914 15:08:11.479662     804 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-553526050", UID:"f3a02ff8-7a5e-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"712430", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-553526050-e27zc
E0914 15:08:16.937478     804 deployment_controller.go:395] Error syncing deployment default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad: deployments.extensions "ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad" cannot be updated: the object has been modified; please apply your changes to the latest version and try again
I0914 15:09:23.006072     804 event.go:211] Event(api.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad", UID:"f39f3c6b-7a5e-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"712473", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-553526050 to 0
E0914 15:09:23.014360     804 deployment_controller.go:395] Error syncing deployment default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad: Deployment.extensions "ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad" is invalid: status.unavailableReplicas: Invalid value: -2: must be greater than or equal to 0
I0914 15:09:23.021228     804 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-553526050", UID:"f3a02ff8-7a5e-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"712474", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-553526050-e27zc
I0914 15:09:23.034585     804 event.go:211] Event(api.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad", UID:"f39f3c6b-7a5e-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"712473", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381 to 3
I0914 15:09:23.041750     804 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-553526050", UID:"f3a02ff8-7a5e-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"712474", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-553526050-aqc9n
I0914 15:09:23.107364     804 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381", UID:"1e4c0bb4-7a5f-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"712476", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381-slc9b
I0914 15:09:23.113622     804 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381", UID:"1e4c0bb4-7a5f-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"712476", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381-jlem5
I0914 15:09:23.114482     804 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381", UID:"1e4c0bb4-7a5f-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"712476", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381-3ggyl
E0914 15:09:23.135217     804 deployment_controller.go:395] Error syncing deployment default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad: deployments.extensions "ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad" cannot be updated: the object has been modified; please apply your changes to the latest version and try again
E0914 20:55:48.407387     804 deployment_controller.go:395] Error syncing deployment default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad: deployments.extensions "ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad" cannot be updated: the object has been modified; please apply your changes to the latest version and try again
I0915 09:40:51.039394     815 deployment_controller.go:412] Waiting for replica set / pod controller to sync, requeuing deployment default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad
I0915 10:19:48.663471     815 event.go:211] Event(api.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad", UID:"f39f3c6b-7a5e-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"716056", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381 to 0
I0915 10:19:48.680659     815 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381", UID:"1e4c0bb4-7a5f-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"716057", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381-3ggyl
I0915 10:19:48.712661     815 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381", UID:"1e4c0bb4-7a5f-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"716057", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381-jlem5
I0915 10:19:48.712801     815 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381", UID:"1e4c0bb4-7a5f-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"716057", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381-slc9b
E0915 10:19:48.714738     815 deployment_controller.go:395] Error syncing deployment default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad: Deployment.extensions "ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad" is invalid: status.unavailableReplicas: Invalid value: -2: must be greater than or equal to 0
E0915 10:19:48.770944     815 deployment_controller.go:395] Error syncing deployment default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad: deployments.extensions "ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad" cannot be updated: the object has been modified; please apply your changes to the latest version and try again
I0915 10:20:00.398393     815 event.go:211] Event(api.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad", UID:"f39f3c6b-7a5e-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"716078", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196 to 3
E0915 10:20:00.424923     815 deployment_controller.go:395] Error syncing deployment default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad: deployments.extensions "ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad" cannot be updated: the object has been modified; please apply your changes to the latest version and try again
I0915 10:20:00.689887     815 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196", UID:"dbc727bb-7aff-11e6-b713-08002787645f", APIVersion:"extensions", ResourceVersion:"716079", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196-6zn50
I0915 10:20:00.692263     815 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196", UID:"dbc727bb-7aff-11e6-b713-08002787645f", APIVersion:"extensions", ResourceVersion:"716079", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196-ms3fz
I0915 10:20:00.701085     815 event.go:211] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196", UID:"dbc727bb-7aff-11e6-b713-08002787645f", APIVersion:"extensions", ResourceVersion:"716079", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196-nctq2
I0915 10:20:57.856225     815 event.go:211] Event(api.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad", UID:"f39f3c6b-7a5e-11e6-89e8-08002787645f", APIVersion:"extensions", ResourceVersion:"716116", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196 to 3
E0915 10:20:57.864905     815 deployment_controller.go:395] Error syncing deployment default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad: deployments.extensions "ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad" cannot be updated: the object has been modified; please apply your changes to the latest version and try again
I0915 10:20:58.628055     815 replica_set.go:511] ReplicaSet has been deleted default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-1103507196
I0915 10:20:58.709122     815 replica_set.go:511] ReplicaSet has been deleted default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-2395606419
I0915 10:20:58.792619     815 replica_set.go:511] ReplicaSet has been deleted default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-3659083381
I0915 10:20:58.866851     815 replica_set.go:511] ReplicaSet has been deleted default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad-553526050
I0915 10:20:58.881177     815 deployment_controller.go:424] Deployment has been deleted default/ultraesb-4bb6588f-db05-47b9-af6a-6717baa03dad

What you expected to happen: All the pods created from deployment should be deleted once the deployment is deleted.

How to reproduce it (as minimally and precisely as possible): This is a very rare case. 99% times when I delete a deployment, it works perfectly. Above issue occurred when I tried to delete multiple deployments in sequence (almost in parallel).

Anything else do we need to know:
@janetkuo
Copy link
Member

cc @kubernetes/deployment

@0xmichalis
Copy link
Contributor

You tried to delete the deployment using the REST API directly or kubectl? Probably the latter. This should be fixed for 1.3.

@DImuthuUpe
Copy link
Author

We tried to delete the deployment using the java client https://github.com/fabric8io/kubernetes-client. So basically it is though the REST API

@0xmichalis
Copy link
Contributor

I am not familiar with the fabric client and how are they implementing the deployment reaper - unfortunately it is a client-specific thing atm, deleting a deployment via the API cannot cascade currently.

See #12143 and #26120

@janetkuo
Copy link
Member

This looks like a bug in the java client you're using

Specifically, the deployment reaper invokes its RS reaper to delete the RSes. But before that, those RSes should have scaled themselves to 0. See code here https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/DeploymentOperationsImpl.java#L169

@DImuthuUpe Would you file a bug against their repo? Thanks.

@tobegit3hub tobegit3hub mentioned this issue Oct 24, 2016
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/app-lifecycle area/workload-api/deployment sig/apps Categorizes an issue or PR as relevant to SIG Apps.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@DImuthuUpe @0xmichalis @janetkuo @k8s-github-robot