New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StatefulSet delete behavior changed in v1.11 #68627

Open
rezroo opened this Issue Sep 13, 2018 · 8 comments

Comments

Projects
None yet
6 participants
@rezroo

rezroo commented Sep 13, 2018

/kind bug

What happened:
In k8s v1.11.1 all statefulset pods are deleted at the same time. This is what I observe:

stack@master:~$ kubectl get pod -w
NAME           READY     STATUS    RESTARTS   AGE
echoserver-0   0/1       Pending   0          0s
echoserver-0   0/1       Pending   0         0s
echoserver-0   0/1       ContainerCreating   0         0s
echoserver-0   0/1       ContainerCreating   0         0s
echoserver-0   1/1       Running   0         1s
echoserver-1   0/1       Pending   0         0s
echoserver-1   0/1       Pending   0         0s
echoserver-1   0/1       ContainerCreating   0         0s
echoserver-1   0/1       ContainerCreating   0         0s
echoserver-1   1/1       Running   0         1s
echoserver-2   0/1       Pending   0         0s
echoserver-2   0/1       Pending   0         0s
echoserver-2   0/1       ContainerCreating   0         0s
echoserver-2   0/1       ContainerCreating   0         0s
echoserver-2   1/1       Running   0         1s
echoserver-0   1/1       Terminating   0         22s
echoserver-2   1/1       Terminating   0         20s
echoserver-1   1/1       Terminating   0         21s
echoserver-1   0/1       Terminating   0         1m
echoserver-0   0/1       Terminating   0         1m
echoserver-0   0/1       Terminating   0         1m
echoserver-0   0/1       Terminating   0         1m
echoserver-2   0/1       Terminating   0         1m
echoserver-2   0/1       Terminating   0         1m
echoserver-2   0/1       Terminating   0         1m
echoserver-2   0/1       Terminating   0         1m
echoserver-1   0/1       Terminating   0         1m
echoserver-1   0/1       Terminating   0         1m

What you expected to happen:
In k8s v1.9.3 StatefulSet deletion adheres to documentation and follows Ordered, graceful deletion and termination. In v1.9.3 when I delete the statefulset this is what I see:

stack@master:~$ kubectl get pod -w
NAME           READY     STATUS    RESTARTS   AGE
echoserver-0   1/1       Running   0          22s
echoserver-1   1/1       Running   0          15s
echoserver-2   1/1       Running   0          9s
echoserver-2   1/1       Terminating   0         29s
echoserver-2   0/1       Terminating   0         30s
echoserver-2   0/1       Terminating   0         38s
echoserver-2   0/1       Terminating   0         38s
echoserver-1   1/1       Terminating   0         44s
echoserver-1   0/1       Terminating   0         45s
echoserver-1   0/1       Terminating   0         55s
echoserver-1   0/1       Terminating   0         55s
echoserver-0   1/1       Terminating   0         1m
echoserver-0   0/1       Terminating   0         1m
echoserver-0   0/1       Terminating   0         1m
echoserver-0   0/1       Terminating   0         1m

How to reproduce it (as minimally and precisely as possible):
Create, the delete the following:

apiVersion: v1
kind: Service
metadata:
  name: echoserver
  labels:
    app: echoserver
spec:
  ports:
  - port: 80
    targetPort: 8080
  clusterIP: None
  selector:
    app: echoserver
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: echoserver
spec:
  serviceName: echoserver
  replicas: 3
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - image: alpine
        name: alpine
        command:
          - sleep
          - '70'
        imagePullPolicy: IfNotPresent

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
    NAME="Ubuntu"
    VERSION="16.04.2 LTS (Xenial Xerus)"
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
@krmayankk

This comment has been minimized.

Show comment
Hide comment
@krmayankk

krmayankk Sep 13, 2018

Contributor

/sig apps

Contributor

krmayankk commented Sep 13, 2018

/sig apps

@k8s-ci-robot k8s-ci-robot added sig/apps and removed needs-sig labels Sep 13, 2018

@krmayankk

This comment has been minimized.

Show comment
Hide comment
@krmayankk

krmayankk Sep 13, 2018

Contributor

It seems the default value of pod management policy changed . You can choose parallel and see f that fixes it https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy

Contributor

krmayankk commented Sep 13, 2018

It seems the default value of pod management policy changed . You can choose parallel and see f that fixes it https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy

@rezroo

This comment has been minimized.

Show comment
Hide comment
@rezroo

rezroo Sep 13, 2018

$ kubectl get statefulsets.apps echoserver -o yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: 2018-09-13T22:21:22Z
  generation: 1
  name: echoserver
  namespace: default
  resourceVersion: "1935"
  selfLink: /apis/apps/v1/namespaces/default/statefulsets/echoserver
  uid: 58639393-b7a3-11e8-9df7-02f13ec36f84
spec:
  podManagementPolicy: OrderedReady
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: echoserver
  serviceName: echoserver
  template:

Creation:

echoserver-0   0/1       Pending   0         0s        <none>    <none>
echoserver-0   0/1       Pending   0         0s        <none>    node2
echoserver-0   0/1       ContainerCreating   0         0s        <none>    node2
echoserver-0   0/1       ContainerCreating   0         1s        <none>    node2
echoserver-0   1/1       Running   0         1s        192.168.2.5   node2
echoserver-1   0/1       Pending   0         0s        <none>    <none>
echoserver-1   0/1       Pending   0         0s        <none>    node1
echoserver-1   0/1       ContainerCreating   0         0s        <none>    node1
echoserver-1   0/1       ContainerCreating   0         1s        <none>    node1
echoserver-1   1/1       Running   0         2s        192.168.1.5   node1
echoserver-2   0/1       Pending   0         0s        <none>    <none>
echoserver-2   0/1       Pending   0         0s        <none>    node2
echoserver-2   0/1       ContainerCreating   0         0s        <none>    node2
echoserver-2   0/1       ContainerCreating   0         0s        <none>    node2
echoserver-2   1/1       Running   0         1s        192.168.2.6   node2

Deletion:

echoserver-2   1/1       Terminating   0         2m        192.168.2.6   node2
echoserver-1   1/1       Terminating   0         2m        192.168.1.5   node1
echoserver-0   1/1       Terminating   0         2m        192.168.2.5   node2
echoserver-2   0/1       Terminating   0         2m        192.168.2.6   node2
echoserver-0   0/1       Terminating   0         2m        192.168.2.5   node2
echoserver-1   0/1       Terminating   0         2m        192.168.1.5   node1
echoserver-0   0/1       Terminating   0         2m        192.168.2.5   node2
echoserver-0   0/1       Terminating   0         2m        192.168.2.5   node2
echoserver-2   0/1       Terminating   0         2m        192.168.2.6   node2
echoserver-2   0/1       Terminating   0         2m        192.168.2.6   node2
echoserver-1   0/1       Terminating   0         2m        192.168.1.5   node1
echoserver-1   0/1       Terminating   0         2m        192.168.1.5   node1

rezroo commented Sep 13, 2018

$ kubectl get statefulsets.apps echoserver -o yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: 2018-09-13T22:21:22Z
  generation: 1
  name: echoserver
  namespace: default
  resourceVersion: "1935"
  selfLink: /apis/apps/v1/namespaces/default/statefulsets/echoserver
  uid: 58639393-b7a3-11e8-9df7-02f13ec36f84
spec:
  podManagementPolicy: OrderedReady
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: echoserver
  serviceName: echoserver
  template:

Creation:

echoserver-0   0/1       Pending   0         0s        <none>    <none>
echoserver-0   0/1       Pending   0         0s        <none>    node2
echoserver-0   0/1       ContainerCreating   0         0s        <none>    node2
echoserver-0   0/1       ContainerCreating   0         1s        <none>    node2
echoserver-0   1/1       Running   0         1s        192.168.2.5   node2
echoserver-1   0/1       Pending   0         0s        <none>    <none>
echoserver-1   0/1       Pending   0         0s        <none>    node1
echoserver-1   0/1       ContainerCreating   0         0s        <none>    node1
echoserver-1   0/1       ContainerCreating   0         1s        <none>    node1
echoserver-1   1/1       Running   0         2s        192.168.1.5   node1
echoserver-2   0/1       Pending   0         0s        <none>    <none>
echoserver-2   0/1       Pending   0         0s        <none>    node2
echoserver-2   0/1       ContainerCreating   0         0s        <none>    node2
echoserver-2   0/1       ContainerCreating   0         0s        <none>    node2
echoserver-2   1/1       Running   0         1s        192.168.2.6   node2

Deletion:

echoserver-2   1/1       Terminating   0         2m        192.168.2.6   node2
echoserver-1   1/1       Terminating   0         2m        192.168.1.5   node1
echoserver-0   1/1       Terminating   0         2m        192.168.2.5   node2
echoserver-2   0/1       Terminating   0         2m        192.168.2.6   node2
echoserver-0   0/1       Terminating   0         2m        192.168.2.5   node2
echoserver-1   0/1       Terminating   0         2m        192.168.1.5   node1
echoserver-0   0/1       Terminating   0         2m        192.168.2.5   node2
echoserver-0   0/1       Terminating   0         2m        192.168.2.5   node2
echoserver-2   0/1       Terminating   0         2m        192.168.2.6   node2
echoserver-2   0/1       Terminating   0         2m        192.168.2.6   node2
echoserver-1   0/1       Terminating   0         2m        192.168.1.5   node1
echoserver-1   0/1       Terminating   0         2m        192.168.1.5   node1
@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Sep 14, 2018

Member

It seems the default value of pod management policy changed

I don't see a default change between v1beta1 and v1

Member

liggitt commented Sep 14, 2018

It seems the default value of pod management policy changed

I don't see a default change between v1beta1 and v1

@jlegrone

This comment has been minimized.

Show comment
Hide comment
@jlegrone

jlegrone Sep 17, 2018

Contributor

/area stateful-apps

Contributor

jlegrone commented Sep 17, 2018

/area stateful-apps

@Pingan2017

This comment has been minimized.

Show comment
Hide comment
@Pingan2017

Pingan2017 Sep 17, 2018

Contributor

maybe the reason is the default deleteoption for kubectl delete changed
before 1.11, it is Foreground , after it is Background
#65908

Contributor

Pingan2017 commented Sep 17, 2018

maybe the reason is the default deleteoption for kubectl delete changed
before 1.11, it is Foreground , after it is Background
#65908

@Pingan2017

This comment has been minimized.

Show comment
Hide comment
@Pingan2017

Pingan2017 Sep 17, 2018

Contributor

/sig cli

Contributor

Pingan2017 commented Sep 17, 2018

/sig cli

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Sep 17, 2018

Member

before 1.11, it is Foreground , after it is Background

That change was made between 1.11.0 and 1.11.1

Member

liggitt commented Sep 17, 2018

before 1.11, it is Foreground , after it is Background

That change was made between 1.11.0 and 1.11.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment