Unable to delete pods, stuck in terminating state

[cbluth]

I am not sure the exact steps I took to land myself in this situation, but I cannot delete pods.
I cannot delete ArangoDeployment/arangodb-cluster either.

Here is some info:

user@laptop:~/arangodb$ kubectl delete pod -l app=arangodb
pod "arangodb-cluster-agnt-bnwzveev-4f7109" deleted
pod "arangodb-cluster-agnt-cvikovgy-4f7109" deleted
pod "arangodb-cluster-agnt-fc2fco8j-4f7109" deleted
pod "arangodb-cluster-prmr-lafvlwvb-4f7109" deleted
pod "arangodb-cluster-prmr-r40r3cdf-4f7109" deleted
pod "arangodb-cluster-prmr-skdk4xpp-4f7109" deleted

user@laptop:~/arangodb$ kubectl get pod -l app=arangodb
NAME                                    READY     STATUS        RESTARTS   AGE
arangodb-cluster-agnt-bnwzveev-4f7109   0/1       Terminating   0          22h
arangodb-cluster-agnt-cvikovgy-4f7109   0/1       Terminating   0          22h
arangodb-cluster-agnt-fc2fco8j-4f7109   0/1       Terminating   0          1d
arangodb-cluster-prmr-lafvlwvb-4f7109   0/1       Terminating   0          1d
arangodb-cluster-prmr-r40r3cdf-4f7109   0/1       Terminating   0          1d
arangodb-cluster-prmr-skdk4xpp-4f7109   0/1       Terminating   0          1d
user@laptop:~/arangodb$ 

user@laptop:~/arangodb$ kubectl -n platform describe ArangoDeployment/arangodb-cluster
Name:         arangodb-cluster
Namespace:    platform
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"database.arangodb.com/v1alpha","kind":"ArangoDeployment","metadata":{"annotations":{},"name":"arangodb-cluster","namespace":"platform"},...
API Version:  database.arangodb.com/v1alpha
Kind:         ArangoDeployment
Metadata:
  Cluster Name:                   
  Creation Timestamp:             2018-06-21T07:58:37Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2018-07-17T14:31:04Z
  Finalizers:
    database.arangodb.com/remove-child-finalizers
  Generation:        2
  Resource Version:  11280293
  Self Link:         /apis/database.arangodb.com/v1alpha/namespaces/platform/arangodeployments/arangodb-cluster
  UID:               e7100671-7528-11e8-8e23-525400ac45ea
Spec:
  Agents:
    Count:  0
    Resources:
      Requests:
        Storage:  8Gi
  Auth:
    Jwt Secret Name:  arangodb-cluster-jwt
  Chaos:
    Interval:                  60000000000
    Kill - Pod - Probability:  50
  Coordinators:
    Count:  0
    Resources:
  Dbservers:
    Count:  0
    Resources:
      Requests:
        Storage:  8Gi
  Environment:    Production
  External Access:
  Image:              arangodb/arangodb:3.3.10
  Image Pull Policy:  IfNotPresent
  Mode:               Cluster
  Rocksdb:
    Encryption:
  Single:
    Resources:
      Requests:
        Storage:   8Gi
  Storage Engine:  RocksDB
  Sync:
    Auth:
      Client CA Secret Name:  arangodb-cluster-sync-client-auth-ca
      Jwt Secret Name:        arangodb-cluster-sync-jwt
    External Access:
    Monitoring:
      Token Secret Name:  arangodb-cluster-sync-mt
    Tls:
      Ca Secret Name:  arangodb-cluster-sync-ca
      Ttl:             2610h
  Syncmasters:
    Resources:
  Syncworkers:
    Resources:
  Tls:
    Ca Secret Name:  arangodb-cluster-ca
    Ttl:             2610h
Status:
  Accepted - Spec:
    Agents:
      Count:  0
      Resources:
        Requests:
          Storage:  8Gi
    Auth:
      Jwt Secret Name:  arangodb-cluster-jwt
    Chaos:
      Interval:                  60000000000
      Kill - Pod - Probability:  50
    Coordinators:
      Count:  0
      Resources:
    Dbservers:
      Count:  0
      Resources:
        Requests:
          Storage:  8Gi
    Environment:    Production
    External Access:
    Image:              arangodb/arangodb:3.3.10
    Image Pull Policy:  IfNotPresent
    Mode:               Cluster
    Rocksdb:
      Encryption:
    Single:
      Resources:
        Requests:
          Storage:   8Gi
    Storage Engine:  RocksDB
    Sync:
      Auth:
        Client CA Secret Name:  arangodb-cluster-sync-client-auth-ca
        Jwt Secret Name:        arangodb-cluster-sync-jwt
      External Access:
      Monitoring:
        Token Secret Name:  arangodb-cluster-sync-mt
      Tls:
        Ca Secret Name:  arangodb-cluster-sync-ca
        Ttl:             2610h
    Syncmasters:
      Resources:
    Syncworkers:
      Resources:
    Tls:
      Ca Secret Name:  arangodb-cluster-ca
      Ttl:             2610h
  Arangodb - Images:
    Arangodb - Version:  3.3.10
    Image:               arangodb/arangodb:3.3.10
    Image - Id:          arangodb/arangodb@sha256:fa7fe7c8f2ca3e058440b20b4910d10df9fc4488f52c4a75e4e71ed4f0194a28
  Conditions:
    Last Transition Time:  2018-06-21T07:58:42Z
    Last Update Time:      2018-06-21T07:58:42Z
    Status:                False
    Type:                  Ready
  Members:
    Agents:
      Conditions:
        Last Transition Time:        2018-07-16T15:58:54Z
        Last Update Time:            2018-07-16T15:58:54Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            AGNT-cvikovgy
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-agent-cvikovgy
      Phase:                         Created
      Pod Name:                      arangodb-cluster-agnt-cvikovgy-4f7109
      Recent - Terminations:
        2018-07-16T15:56:21Z
      Conditions:
        Last Transition Time:        2018-07-16T15:58:54Z
        Last Update Time:            2018-07-16T15:58:54Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            AGNT-bnwzveev
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-agent-bnwzveev
      Phase:                         Created
      Pod Name:                      arangodb-cluster-agnt-bnwzveev-4f7109
      Recent - Terminations:
        2018-07-16T15:56:21Z
      Conditions:
        Last Transition Time:        2018-07-16T12:38:17Z
        Last Update Time:            2018-07-16T12:38:17Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            AGNT-fc2fco8j
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-agent-fc2fco8j
      Phase:                         Created
      Pod Name:                      arangodb-cluster-agnt-fc2fco8j-4f7109
      Recent - Terminations:
        2018-06-26T19:56:42Z
        2018-07-16T10:04:55Z
        2018-07-16T12:35:46Z
    Coordinators:
      Conditions:
        Last Transition Time:  2018-06-21T07:59:59Z
        Last Update Time:      2018-06-21T07:59:59Z
        Reason:                Pod Not Ready
        Status:                False
        Type:                  Ready
        Last Transition Time:  2018-06-26T19:56:42Z
        Last Update Time:      2018-06-26T19:56:42Z
        Reason:                Pod Terminated
        Status:                True
        Type:                  Terminated
      Created - At:            2018-06-21T07:58:37Z
      Id:                      CRDN-70m1ufys
      Initialized:             false
      Phase:                   Failed
      Pod Name:                arangodb-cluster-crdn-70m1ufys-4f7109
      Recent - Terminations:
        2018-06-26T19:56:42Z
      Conditions:
        Last Transition Time:  2018-06-21T07:59:59Z
        Last Update Time:      2018-06-21T07:59:59Z
        Reason:                Pod Not Ready
        Status:                False
        Type:                  Ready
        Last Transition Time:  2018-06-26T19:56:42Z
        Last Update Time:      2018-06-26T19:56:42Z
        Reason:                Pod Terminated
        Status:                True
        Type:                  Terminated
      Created - At:            2018-06-21T07:58:37Z
      Id:                      CRDN-lhzbloq9
      Initialized:             false
      Phase:                   Failed
      Pod Name:                arangodb-cluster-crdn-lhzbloq9-4f7109
      Recent - Terminations:
        2018-06-26T19:56:42Z
      Conditions:
        Last Transition Time:  2018-06-21T07:59:59Z
        Last Update Time:      2018-06-21T07:59:59Z
        Reason:                Pod Not Ready
        Status:                False
        Type:                  Ready
        Last Transition Time:  2018-07-16T10:55:09Z
        Last Update Time:      2018-07-16T10:59:04Z
        Reason:                Pod Terminated
        Status:                True
        Type:                  Terminated
      Created - At:            2018-06-21T07:58:37Z
      Id:                      CRDN-w8u5ely9
      Initialized:             false
      Phase:                   Failed
      Pod Name:                arangodb-cluster-crdn-w8u5ely9-4f7109
      Recent - Terminations:
        2018-07-16T10:55:09Z
    Dbservers:
      Conditions:
        Last Transition Time:        2018-07-16T13:37:26Z
        Last Update Time:            2018-07-16T13:37:26Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            PRMR-skdk4xpp
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-dbserver-skdk4xpp
      Phase:                         Created
      Pod Name:                      arangodb-cluster-prmr-skdk4xpp-4f7109
      Recent - Terminations:
        2018-06-26T19:59:14Z
        2018-07-16T10:04:55Z
        2018-07-16T13:35:25Z
      Conditions:
        Last Transition Time:        2018-07-16T12:42:50Z
        Last Update Time:            2018-07-16T12:42:50Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            PRMR-lafvlwvb
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-dbserver-lafvlwvb
      Phase:                         Created
      Pod Name:                      arangodb-cluster-prmr-lafvlwvb-4f7109
      Recent - Terminations:
        2018-07-16T10:04:55Z
        2018-07-16T12:40:48Z
      Conditions:
        Last Transition Time:        2018-07-16T12:44:22Z
        Last Update Time:            2018-07-16T12:44:22Z
        Reason:                      Pod Not Ready
        Status:                      False
        Type:                        Ready
      Created - At:                  2018-06-21T07:58:37Z
      Id:                            PRMR-r40r3cdf
      Initialized:                   false
      Persistent Volume Claim Name:  arangodb-cluster-dbserver-r40r3cdf
      Phase:                         Created
      Pod Name:                      arangodb-cluster-prmr-r40r3cdf-4f7109
      Recent - Terminations:
        2018-07-16T12:42:50Z
  Phase:  Running
  Plan:
    Creation Time:  2018-06-21T08:06:58Z
    Group:          4
    Id:             R32pMTPkJVj8gXiW
    Member ID:      CRDN-70m1ufys
    Type:           RemoveMember
    Creation Time:  2018-06-21T08:06:58Z
    Group:          4
    Id:             czD5LhtMcWGdXepC
    Type:           AddMember
  Secret - Hashes:
    Auth - Jwt:  dd5b638b4bd6a07c333ae638b7b2631c4074a1f0b300deeb56e0e9f45e7f6484
    Tls - Ca:    b22a4afa0e31708668d614ab69d8a92343baf037fd197a0d8ecd7eff7f4729ef
  Service Name:  arangodb-cluster
Events:          <none>
user@laptop:~/arangodb$ 

How can I delete these pods?

[cbluth]

Here is a bit more info. not sure if its needed:

user@laptop:~/arangodb$ kubectl delete CustomResourceDefinition arangodeployments.database.arangodb.com --force --now=true
customresourcedefinition.apiextensions.k8s.io "arangodeployments.database.arangodb.com" deleted
user@laptop:~/arangodb$ kubectl delete CustomResourceDefinition arangodeployments.database.arangodb.com --force --now=true
customresourcedefinition.apiextensions.k8s.io "arangodeployments.database.arangodb.com" deleted
user@laptop:~/arangodb$ kubectl delete CustomResourceDefinition arangodeployments.database.arangodb.com --force --now=true
customresourcedefinition.apiextensions.k8s.io "arangodeployments.database.arangodb.com" deleted
user@laptop:~/arangodb$ kubectl describe CustomResourceDefinition arangodeployments.database.arangodb.com
Name:         arangodeployments.database.arangodb.com
Namespace:    
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"apiextensions.k8s.io/v1beta1","kind":"CustomResourceDefinition","metadata":{"annotations":{},"name":"arangodeployments.database.arangodb...
API Version:  apiextensions.k8s.io/v1beta1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:             2018-06-21T07:40:55Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2018-07-17T14:48:19Z
  Finalizers:
    customresourcecleanup.apiextensions.k8s.io
  Generation:        1
  Resource Version:  11282686
  Self Link:         /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/arangodeployments.database.arangodb.com
  UID:               6e192dab-7526-11e8-8e23-525400ac45ea
Spec:
  Group:  database.arangodb.com
  Names:
    Kind:       ArangoDeployment
    List Kind:  ArangoDeploymentList
    Plural:     arangodeployments
    Short Names:
      arangodb
      arango
    Singular:  arangodeployment
  Scope:       Namespaced
  Version:     v1alpha
Status:
  Accepted Names:
    Kind:       ArangoDeployment
    List Kind:  ArangoDeploymentList
    Plural:     arangodeployments
    Short Names:
      arangodb
      arango
    Singular:  arangodeployment
  Conditions:
    Last Transition Time:  2018-06-21T07:40:55Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2018-06-21T07:40:55Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
    Last Transition Time:  2018-07-17T13:08:27Z
    Message:               CustomResource deletion is in progress
    Reason:                InstanceDeletionInProgress
    Status:                True
    Type:                  Terminating
Events:                    <none>
user@laptop:~/arangodb$ 

[cbluth]

Deploying a new operator from scratch, and then re-creating the cluster with the same name arangodb-cluster fixed my issue.

[ewoutp]

FYI The operator registers finalizers on the resources that is has created (such as Pods and PersistentVolumeClaims) in order to ensure a controlled removal of those resources.
If the operator itself is no longer running, there is no one to remove this finalizers (which prevents the resource from being removed)