Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible Bug: deleted deployment but related pod didn't delete #83672

Open
zhangxingdeppon opened this issue Oct 9, 2019 · 10 comments

Comments

@zhangxingdeppon
Copy link

commented Oct 9, 2019

What happened:
hi,buddies,I did face a problem,it‘s very important to me,in our product environment,sometimes the following situation occurs:deleted deployment but related pod didn't delete
What you expected to happen:
I expected to delete deployment ,and delete pod at the same time,in addition. I want to know the reason ,tks
How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
[root@D21117U10-62-66 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:31:33Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:

56C 256G
  • OS (e.g: cat /etc/os-release):

[root@D21117U10-62-66 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 (Maipo)
  • Kernel (e.g. uname -a):

[root@D21117U10-62-66 ~]# uname -a
Linux D21117U10-62-66 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:

[root@D21117U10-62-66 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:36:44Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
  • Network plugin and version (if this is a network-related bug):
  • Others:

The details of problem:
firstly,the “aipaas-model serving” namesapce‘s pod:


[root@D21117U10-62-66 data]# kubectl get pod -n aipaas-modelserving
NAME                                                     READY   STATUS    RESTARTS   AGE
bankcard-ptvp6llg-serving-v1-6f96c7f69d-p8x7h            1/2     Running   6          5h24m
bkcardocr-6hxk3uvs-serving-v1-7fd6db565-v2cv4            2/2     Running   0          42m
callmedad-05ziytkh-serving-v1-689cbd7965-bt9rr           1/2     Running   0          27d
commonocr-t2od87nz-serving-v1-5d88d64dc7-4w8r5           2/2     Running   0          102m
fqzcommon-7d6wx5hg-serving-fqzdkm1-5f77489958-dl7m6      2/2     Running   0          23m
fraud4lsjr-eiwccpf1-serving-ls01190927-5db479dd4-9rjgc   2/2     Running   0          12d
fraud4lsjr-eiwccpf1-serving-ls01190927-5db479dd4-zgz5d   2/2     Running   0          12d
idcardocr-xixazq9p-serving-v1-776756fbf5-7wmrr           2/2     Running   0          109m
onlinet-supcz8ud-serving-tonnx-79f6f77c69-lkfgp          1/2     Running   0          29h
pmml-ssam02yp-serving-v1-59f85bd884-dms49                2/2     Running   0          29h
yxpop-114z1r2e-serving-v1-55654cb695-bm6l6               1/2     Running   0          29h

the “aipaas-model serving” namesapce‘s deployment:



[root@D21117U10-62-66 data]# kubectl get deploy -n aipaas-modelserving
NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
bkcardocr-6hxk3uvs-serving-v1            1/1     1            1           42m
commonocr-t2od87nz-serving-v1            1/1     1            1           103m
fqzcommon-7d6wx5hg-serving-fqzdkm1       1/1     1            1           23m
fraud4lsjr-eiwccpf1-serving-ls01190927   2/2     2            2           12d
idcardocr-xixazq9p-serving-v1            1/1     1            1           109m
pmml-ssam02yp-serving-v1                 1/1     1            1           29h

I did not found the related replicaset ,the “aipaas-model serving” namesapce‘s rs:


[root@D21117U10-62-66 ~]# kubectl get rs -n aipaas-modelserving 
NAME                                               DESIRED   CURRENT   READY   AGE
bkcardocr-6hxk3uvs-serving-v1-7fd6db565            1         1         1       3h52m
commonocr-t2od87nz-serving-v1-5d88d64dc7           1         1         1       4h53m
fqzcommon-7d6wx5hg-serving-fqzdkm1-5f77489958      1         1         1       3h33m
fraud4lsjr-eiwccpf1-serving-ls01190927-5db479dd4   2         2         2       12d
idcardocr-xixazq9p-serving-v1-776756fbf5           1         1         1       4h59m
pmml-ssam02yp-serving-v1-59f85bd884                1         1         1       32h

through the above details you will see some pods that did not belong to any deployment ,
for example “bankcard-ptvp6llg-serving-v1-6f96c7f69d-p8x7h ”、”callmedad-05ziytkh-serving-v1-689cbd7965-bt9rr “、"onlinet-supcz8ud-serving-tonnx-79f6f77c69-lkfgp"、"yxpop-114z1r2e-serving-v1-55654cb695-bm6l6".

secondly,I have check the logs from the controller manager related to pod "callmedad-05ziytkh-serving-v1-689cbd7965-bt9rr".
controller manager leader log:


[root@D21117U10-62-66 ~]# kubectl logs -f kube-controller-manager-d21112u04-61-143 -n kube-system  |grep callmedad-05ziytkh-serving
I0912 05:59:22.598437       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"aipaas-modelserving", Name:"callmedad-05ziytkh-serving-v1", UID:"77b66f16-d522-11e9-8085-9cb654b8ed1e", APIVersion:"apps/v1", ResourceVersion:"54756699", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set callmedad-05ziytkh-serving-v1-689cbd7965 to 1
I0912 05:59:22.624464       1 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"aipaas-modelserving", Name:"callmedad-05ziytkh-serving-v1-689cbd7965", UID:"77bc8ae7-d522-11e9-b5bb-0894ef05c4c0", APIVersion:"apps/v1", ResourceVersion:"54756700", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: callmedad-05ziytkh-serving-v1-689cbd7965-bt9rr
I0926 06:09:06.396864       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"aipaas-modelserving", Name:"callmedad-05ziytkh-serving-v1", UID:"77b66f16-d522-11e9-8085-9cb654b8ed1e", APIVersion:"apps/v1", ResourceVersion:"59256284", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set callmedad-05ziytkh-serving-v1-689cbd7965 to 0
E0926 06:09:06.401170       1 replica_set.go:450] Sync "aipaas-modelserving/callmedad-05ziytkh-serving-v1-689cbd7965" failed with Operation cannot be fulfilled on replicasets.apps "callmedad-05ziytkh-serving-v1-689cbd7965": StorageError: invalid object, Code: 4, Key: /registry/replicasets/aipaas-modelserving/callmedad-05ziytkh-serving-v1-689cbd7965, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 77bc8ae7-d522-11e9-b5bb-0894ef05c4c0, UID in object meta:

I guess that information :
"Sync "aipaas-modelserving/callmedad-05ziytkh-serving-v1-689cbd7965" failed with Operation cannot be fulfilled on replicasets.apps "callmedad-05ziytkh-serving-v1-689cbd7965": StorageError: invalid object, Code: 4, Key: /registry/replicasets/aipaas-modelserving/callmedad-05ziytkh-serving-v1-689cbd7965, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 77bc8ae7-d522-11e9-b5bb-0894ef05c4c0, UID in object meta:" maybe is key to resolve this problem . but I have no idea about it ,can you give me some idea?
In addition,another pod "bankcard-ptvp6llg-serving-v1" that also didn't have related deployment, it's diffirent with “callmedad-05ziytkh-serving-v1” ,the log just scaled down replica set to 0 but didn't delete it and no error messages.
controller manager leader log:


[root@D21117U10-62-66 ~]# cat /data/controller-manager-143.log |grep bankcard-ptvp6llg-serving-v1
I1009 02:57:23.960880       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"aipaas-modelserving", Name:"bankcard-ptvp6llg-serving-v1", UID:"84ddfc2f-ea40-11e9-8085-9cb654b8ed1e", APIVersion:"apps/v1", ResourceVersion:"63392023", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set bankcard-ptvp6llg-serving-v1-6f96c7f69d to 1
I1009 02:57:23.989499       1 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"aipaas-modelserving", Name:"bankcard-ptvp6llg-serving-v1-6f96c7f69d", UID:"84df1d76-ea40-11e9-b5bb-0894ef05c4c0", APIVersion:"apps/v1", ResourceVersion:"63392024", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: bankcard-ptvp6llg-serving-v1-6f96c7f69d-p8x7h
I1009 03:01:11.785394       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"aipaas-modelserving", Name:"bankcard-ptvp6llg-serving-v1", UID:"84ddfc2f-ea40-11e9-8085-9cb654b8ed1e", APIVersion:"apps/v1", ResourceVersion:"63392921", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set bankcard-ptvp6llg-serving-v1-6f96c7f69d to 0
can you give me some idea ? I want to resolve it,thanks.
@zhangxingdeppon

This comment has been minimized.

Copy link
Author

commented Oct 9, 2019

@kubernetes/sig-apps-bugs

@k8s-ci-robot k8s-ci-robot added sig/apps and removed needs-sig labels Oct 9, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Oct 9, 2019

@zhangxingdeppon: Reiterating the mentions to trigger a notification:
@kubernetes/sig-apps-bugs

In response to this:

@kubernetes/sig-apps-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@calmkart

This comment has been minimized.

Copy link
Contributor

commented Oct 14, 2019

@zhangxingdeppon
please give deployment yaml here?
callmedad & bankcard

@zhangxingdeppon

This comment has been minimized.

Copy link
Author

commented Oct 14, 2019

@zhangxingdeppon
please give deployment yaml here?
callmedad & bankcard

@calmkart okay,because the related deployment resource has deleted ,just pod resource we can see,but i can give you another deployment yaml that is similar to the related deployment


[root@D21117U10-62-66 ~]# kubectl edit deployment bkcardocr-8ejw7khl-serving-v1  -n aipaas-modelserving

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2019-10-14T06:28:26Z"
  generation: 1
  labels:
    app: bkcardocr-8ejw7khl-serving-v1
  name: bkcardocr-8ejw7khl-serving-v1
  namespace: aipaas-modelserving
  resourceVersion: "65054671"
  selfLink: /apis/extensions/v1beta1/namespaces/aipaas-modelserving/deployments/bkcardocr-8ejw7khl-serving-v1
  uid: d43fb9c1-ee4b-11e9-8085-9cb654b8ed1e
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: bkcardocr-8ejw7khl-serving
      version: v1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: bkcardocr-8ejw7khl-serving
        version: v1
    spec:
      containers:
      - command:
        - python3
        - run.py
        - --handle_file=bank_card_service.py
        - --handle_class=modelService
        - --url=http://aipaas-serving-model-service.aipaas:10014/serving/download?path=/model/data/ailab/MachineLearning/1910091109557840001/v1
        image: 172.29.62.9:8080/library/model-serving:v1.0.2
        imagePullPolicy: Always
        name: bkcardocr-8ejw7khl-serving-v1
        ports:
        - containerPort: 8888
          name: jupyter
          protocol: TCP
        resources:
          limits:
            cpu: "4"
            memory: 8Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: "1"
            memory: 2Gi
            nvidia.com/gpu: "1"
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        node: worker
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

this is pod yaml:


[root@D21117U10-62-66 ~]# kubectl get  pod bankcard-ptvp6llg-serving-v1-6f96c7f69d-p8x7h  -n aipaas-modelserving  -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    sidecar.istio.io/status: '{"version":"761ebc5a63976754715f22fcf548f05270fb4b8db07324894aebdb31fa81d960","initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs"],"imagePullSecrets":null}'
  creationTimestamp: "2019-10-09T02:57:23Z"
  generateName: bankcard-ptvp6llg-serving-v1-6f96c7f69d-
  labels:
    app: bankcard-ptvp6llg-serving
    pod-template-hash: 6f96c7f69d
    version: v1
  name: bankcard-ptvp6llg-serving-v1-6f96c7f69d-p8x7h
  namespace: aipaas-modelserving
  resourceVersion: "63397133"
  selfLink: /api/v1/namespaces/aipaas-modelserving/pods/bankcard-ptvp6llg-serving-v1-6f96c7f69d-p8x7h
  uid: 84e3625b-ea40-11e9-b5bb-0894ef05c4c0
spec:
  containers:
  - command:
    - python3
    - run.py
    - --handle_file=bank_card_service.py
    - --handle_class=modelService
    - --url=http://aipaas-serving-model-service.aipaas:10014/serving/download?path=/model/data/ailab/zhangxiao/1910081026442350000/v1
    env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: none
    image: 172.29.62.9:8080/library/model-serving:v1.0.0
    imagePullPolicy: Always
    name: bankcard-ptvp6llg-serving-v1
    ports:
    - containerPort: 8888
      name: jupyter
      protocol: TCP
    resources:
      limits:
        cpu: "1"
        memory: 4Gi
      requests:
        cpu: 250m
        memory: 1Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-qvqrc
      readOnly: true
  - args:
    - proxy
    - sidecar
    - --domain
    - $(POD_NAMESPACE).svc.cluster.local
    - --configPath
    - /etc/istio/proxy
    - --binaryPath
    - /usr/local/bin/envoy
    - --serviceCluster
    - bankcard-ptvp6llg-serving.$(POD_NAMESPACE)
    - --drainDuration
    - 45s
    - --parentShutdownDuration
    - 1m0s
    - --discoveryAddress
    - istio-pilot.istio-system:15010
    - --zipkinAddress
    - zipkin.istio-system:9411
    - --dnsRefreshRate
    - 300s
    - --connectTimeout
    - 10s
    - --proxyAdminPort
    - "15000"
    - --concurrency
    - "2"
    - --controlPlaneAuthPolicy
    - NONE
    - --statusPort
    - "15020"
    - --applicationPorts
    - "8888"
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: ISTIO_META_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: ISTIO_META_CONFIG_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: ISTIO_META_INTERCEPTION_MODE
      value: REDIRECT
    - name: ISTIO_META_INCLUDE_INBOUND_PORTS
      value: "8888"
    - name: ISTIO_METAJSON_LABELS
      value: |
        {"app":"bankcard-ptvp6llg-serving","pod-template-hash":"6f96c7f69d","version":"v1"}
    image: 172.29.62.9:8080/istio/istio/proxyv2:1.2.2
    imagePullPolicy: IfNotPresent
    name: istio-proxy
    ports:
    - containerPort: 15090
      name: http-envoy-prom
      protocol: TCP
    readinessProbe:
      failureThreshold: 30
      httpGet:
        path: /healthz/ready
        port: 15020
        scheme: HTTP
      initialDelaySeconds: 1
      periodSeconds: 2
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 128Mi
    securityContext:
      procMount: Default
      readOnlyRootFilesystem: true
      runAsUser: 1337
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/certs/
      name: istio-certs
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-qvqrc
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - args:
    - -p
    - "15001"
    - -u
    - "1337"
    - -m
    - REDIRECT
    - -i
    - '*'
    - -x
    - ""
    - -b
    - "8888"
    - -d
    - "15020"
    image: 172.29.62.9:8080/istio/istio/proxy_init:1.2.2
    imagePullPolicy: IfNotPresent
    name: istio-init
    resources:
      limits:
        cpu: 100m
        memory: 50Mi
      requests:
        cpu: 10m
        memory: 10Mi
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      procMount: Default
      runAsNonRoot: false
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  nodeName: d21016u04-62-101
  nodeSelector:
    node: worker
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-qvqrc
    secret:
      defaultMode: 420
      secretName: default-token-qvqrc
  - emptyDir:
      medium: Memory
    name: istio-envoy
  - name: istio-certs
    secret:
      defaultMode: 420
      optional: true
      secretName: istio.default

maybe you feel confused,because we used service mesh "Istio",so the yaml has a sidecar pod ,If you want other information,I am very pleasure to provide it,Tks

@calmkart

This comment has been minimized.

Copy link
Contributor

commented Oct 14, 2019


[root@D21117U10-62-66 ~]# kubectl logs -f kube-controller-manager-d21112u04-61-143 -n kube-system  |grep callmedad-05ziytkh-serving
I0912 05:59:22.598437       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"aipaas-modelserving", Name:"callmedad-05ziytkh-serving-v1", UID:"77b66f16-d522-11e9-8085-9cb654b8ed1e", APIVersion:"apps/v1", ResourceVersion:"54756699", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set callmedad-05ziytkh-serving-v1-689cbd7965 to 1
I0912 05:59:22.624464       1 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"aipaas-modelserving", Name:"callmedad-05ziytkh-serving-v1-689cbd7965", UID:"77bc8ae7-d522-11e9-b5bb-0894ef05c4c0", APIVersion:"apps/v1", ResourceVersion:"54756700", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: callmedad-05ziytkh-serving-v1-689cbd7965-bt9rr
I0926 06:09:06.396864       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"aipaas-modelserving", Name:"callmedad-05ziytkh-serving-v1", UID:"77b66f16-d522-11e9-8085-9cb654b8ed1e", APIVersion:"apps/v1", ResourceVersion:"59256284", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set callmedad-05ziytkh-serving-v1-689cbd7965 to 0
E0926 06:09:06.401170       1 replica_set.go:450] Sync "aipaas-modelserving/callmedad-05ziytkh-serving-v1-689cbd7965" failed with Operation cannot be fulfilled on replicasets.apps "callmedad-05ziytkh-serving-v1-689cbd7965": StorageError: invalid object, Code: 4, Key: /registry/replicasets/aipaas-modelserving/callmedad-05ziytkh-serving-v1-689cbd7965, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 77bc8ae7-d522-11e9-b5bb-0894ef05c4c0, UID in object meta:

paste the logs near 06:09:06?

@zhangxingdeppon

This comment has been minimized.

Copy link
Author

commented Oct 15, 2019

callmedad-05ziytkh-serving
@calmkart
okay,i upload ten minutes log,you can see it has some warn log about autoscaling/metric,maybe could ignore the information,beacause we don't need to configure metric-server to autoscaling,tks.
exceptionPod-relatedLog-ten-minute.log

@mrrandrade

This comment has been minimized.

Copy link

commented Oct 17, 2019

Hey there!

I'm having the same trouble here (kubernetes 1.15.3), but on a very specific situation.

  • If I operate on objects with kubectl, a deployment delete triggers a cascade replicaset delete as expected.

But we don't operate on our cluster using kubectl; we use a custom solution that is kind of outdated.

  • If I operate using this custom tools, a deployment delete, instead of triggering the cascade delete, triggers a cascade patch where the Controller removes the ownerReferences section under metadata, but doesn't delete the rs object.

This is the apiserver log in a custom format showing what he does:

# Our application sends the delete:
"2019-10-17T18:21:24.164603Z|custom-user:apps/v1/deployments/labatf-eu-ejb:delete:null"
"2019-10-17T18:21:24.171643Z|custom-user:apps/v1/deployments/labatf-eu-ejb:delete:200"

# The controller triggers its operations, that includes a patch that removes its ownerreferences, but no delete:
"2019-10-17T18:21:24.172234Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:update:null"
"2019-10-17T18:21:24.183272Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:update:200"
"2019-10-17T18:21:24.625529Z|system:unsecured:apps/v1/replicasets/labatf-eu-ejb-bff768d6d:patch:null"
"2019-10-17T18:21:24.632308Z|system:unsecured:apps/v1/replicasets/labatf-eu-ejb-bff768d6d:patch:200"
"2019-10-17T18:21:24.633048Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:update:null"
"2019-10-17T18:21:24.637505Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:update:200"
"2019-10-17T18:21:24.638718Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:patch:null"
"2019-10-17T18:21:24.648674Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:patch:200"

This is the object before its deployment delection:

{
  "kind": "ReplicaSet",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "labatf-eu-web-6b5b5fbc",
    "namespace": "handson-demo",
    "selfLink": "/apis/extensions/v1beta1/namespaces/handson-demo/replicasets/labatf-eu-web-6b5b5fbc",
    "uid": "0feb2bb4-5ee1-4855-a51e-816f6524c914",
    "resourceVersion": "1138722490",
    "generation": 1,
    "creationTimestamp": "2019-10-17T20:12:10Z",
    "labels": {
      "pod-template-hash": "6b5b5fbc",
      "run": "labatf-eu-web"
    },
    "annotations": {
      "deployment.kubernetes.io/desired-replicas": "1",
      "deployment.kubernetes.io/max-replicas": "2",
      "deployment.kubernetes.io/revision": "2"
    },
    "ownerReferences": [
      {
        "apiVersion": "apps/v1",
        "kind": "Deployment",
        "name": "labatf-eu-web",
        "uid": "e382d076-3498-4981-a2c6-128d7712e7ef",
        "controller": true,
        "blockOwnerDeletion": true
      }
    ]
  },

This is the object after deletion of its deployment:

{
  "kind": "ReplicaSet",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "labatf-eu-web-6b5b5fbc",
    "namespace": "handson-demo",
    "selfLink": "/apis/extensions/v1beta1/namespaces/handson-demo/replicasets/labatf-eu-web-6b5b5fbc",
    "uid": "0feb2bb4-5ee1-4855-a51e-816f6524c914",
    "resourceVersion": "1138749550",
    "generation": 1,
    "creationTimestamp": "2019-10-17T20:12:10Z",
    "labels": {
      "pod-template-hash": "6b5b5fbc",
      "run": "labatf-eu-web"
    },
    "annotations": {
      "deployment.kubernetes.io/desired-replicas": "1",
      "deployment.kubernetes.io/max-replicas": "2",
      "deployment.kubernetes.io/revision": "2"
    }
  },

What motivated me to post here is a suspiction triggered by this comment of OP: I noticed my deployments and replicasets are being created by our tool using the apiVersion 'extensions/v1beta1':

# kubectl get deployment -o yaml  | grep -i apiversion
apiVersion: v1
- apiVersion: extensions/v1beta1
- apiVersion: extensions/v1beta1

# kubectl get rs -o yaml  | grep -i apiversion
apiVersion: v1
- apiVersion: extensions/v1beta1
- apiVersion: extensions/v1beta1
    - apiVersion: apps/v1
- apiVersion: extensions/v1beta1
    - apiVersion: apps/v1
- apiVersion: extensions/v1beta1
    - apiVersion: apps/v1
- apiVersion: extensions/v1beta1
    - apiVersion: apps/v1

I'm guessing the problems we are facing are due to clientside operating on extensions/v1beta1 that somehow triggers an aberrant behaviour somewhere in the Controller code.

@calmkart

This comment has been minimized.

Copy link
Contributor

commented Oct 18, 2019

Hey there!

I'm having the same trouble here (kubernetes 1.15.3), but on a very specific situation.

* If I operate on objects with **kubectl**, a **deployment delete** triggers a cascade replicaset delete as expected.

But we don't operate on our cluster using kubectl; we use a custom solution that is kind of outdated.

* If I operate using this custom tools, a **deployment delete**, instead of triggering the cascade delete, triggers a cascade **patch** where the Controller removes the **ownerReferences** section under **metadata**, but doesn't delete the rs object.

This is the apiserver log in a custom format showing what he does:

# Our application sends the delete:
"2019-10-17T18:21:24.164603Z|custom-user:apps/v1/deployments/labatf-eu-ejb:delete:null"
"2019-10-17T18:21:24.171643Z|custom-user:apps/v1/deployments/labatf-eu-ejb:delete:200"

# The controller triggers its operations, that includes a patch that removes its ownerreferences, but no delete:
"2019-10-17T18:21:24.172234Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:update:null"
"2019-10-17T18:21:24.183272Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:update:200"
"2019-10-17T18:21:24.625529Z|system:unsecured:apps/v1/replicasets/labatf-eu-ejb-bff768d6d:patch:null"
"2019-10-17T18:21:24.632308Z|system:unsecured:apps/v1/replicasets/labatf-eu-ejb-bff768d6d:patch:200"
"2019-10-17T18:21:24.633048Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:update:null"
"2019-10-17T18:21:24.637505Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:update:200"
"2019-10-17T18:21:24.638718Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:patch:null"
"2019-10-17T18:21:24.648674Z|system:unsecured:apps/v1/deployments/labatf-eu-ejb:patch:200"

This is the object before its deployment delection:

{
  "kind": "ReplicaSet",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "labatf-eu-web-6b5b5fbc",
    "namespace": "handson-demo",
    "selfLink": "/apis/extensions/v1beta1/namespaces/handson-demo/replicasets/labatf-eu-web-6b5b5fbc",
    "uid": "0feb2bb4-5ee1-4855-a51e-816f6524c914",
    "resourceVersion": "1138722490",
    "generation": 1,
    "creationTimestamp": "2019-10-17T20:12:10Z",
    "labels": {
      "pod-template-hash": "6b5b5fbc",
      "run": "labatf-eu-web"
    },
    "annotations": {
      "deployment.kubernetes.io/desired-replicas": "1",
      "deployment.kubernetes.io/max-replicas": "2",
      "deployment.kubernetes.io/revision": "2"
    },
    "ownerReferences": [
      {
        "apiVersion": "apps/v1",
        "kind": "Deployment",
        "name": "labatf-eu-web",
        "uid": "e382d076-3498-4981-a2c6-128d7712e7ef",
        "controller": true,
        "blockOwnerDeletion": true
      }
    ]
  },

This is the object after deletion of its deployment:

{
  "kind": "ReplicaSet",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "labatf-eu-web-6b5b5fbc",
    "namespace": "handson-demo",
    "selfLink": "/apis/extensions/v1beta1/namespaces/handson-demo/replicasets/labatf-eu-web-6b5b5fbc",
    "uid": "0feb2bb4-5ee1-4855-a51e-816f6524c914",
    "resourceVersion": "1138749550",
    "generation": 1,
    "creationTimestamp": "2019-10-17T20:12:10Z",
    "labels": {
      "pod-template-hash": "6b5b5fbc",
      "run": "labatf-eu-web"
    },
    "annotations": {
      "deployment.kubernetes.io/desired-replicas": "1",
      "deployment.kubernetes.io/max-replicas": "2",
      "deployment.kubernetes.io/revision": "2"
    }
  },

What motivated me to post here is a suspiction triggered by this comment of OP: I noticed my deployments and replicasets are being created by our tool using the apiVersion 'extensions/v1beta1':

# kubectl get deployment -o yaml  | grep -i apiversion
apiVersion: v1
- apiVersion: extensions/v1beta1
- apiVersion: extensions/v1beta1

# kubectl get rs -o yaml  | grep -i apiversion
apiVersion: v1
- apiVersion: extensions/v1beta1
- apiVersion: extensions/v1beta1
    - apiVersion: apps/v1
- apiVersion: extensions/v1beta1
    - apiVersion: apps/v1
- apiVersion: extensions/v1beta1
    - apiVersion: apps/v1
- apiVersion: extensions/v1beta1
    - apiVersion: apps/v1

I'm guessing the problems we are facing are due to clientside operating on extensions/v1beta1 that somehow triggers an aberrant behaviour somewhere in the Controller code.

@ mrrandrade
i think your problem is cause by garbage-colletion config.
in k8s, there is three kind of garbage-collection deletion way

  • orphan
  • Foreground
  • Background

Foreground and Background are the two modes of cascading deletion.
orphan mode just delete the owner but not delete the Dependent.

cascading deletion is default turn on in kubectl client. if you use kubectl to delete owner, such as kubectl delete deployment nginx-deployment, kubectl will delete the replicasets and the pods.

but if you use api to delete a owner, such as curl, the default option is orphan (in ReplicationController、ReplicaSet、StatefulSet、DaemonSet and Deployment).so it will delete the deployment only, but not delete the rs/pods.(only delete the rs's metadata.ownerReferences)

you can set delete options to cascading, like

{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Background"}'

maybe your custom delete tool set the mode to default (orphan), pls change the mode to Foreground or Background will fixed this.

more info pls see:
https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/

@zhangxingdeppon
is there any help to you ?

@zhangxingdeppon

This comment has been minimized.

Copy link
Author

commented Oct 21, 2019

@calmkart okay,great,I need to verify the result, give you a reply later,thanks very much

@zhangxingdeppon

This comment has been minimized.

Copy link
Author

commented Oct 21, 2019

@calmkart I'm a little confused,if our client was set default (orphan), so when I deleted all deployment ,the all rs/pod should't be deleted ,but actually just accidentally happened ,most of time, deleted the deployment ,the rs,pod also will be deleted at the same time ,I want to known more details ,tks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.