Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StatefulSet - can't rollback from a broken state #67250

Open
MrTrustor opened this issue Aug 10, 2018 · 54 comments
Open

StatefulSet - can't rollback from a broken state #67250

MrTrustor opened this issue Aug 10, 2018 · 54 comments

Comments

@MrTrustor
Copy link

@MrTrustor MrTrustor commented Aug 10, 2018

/kind bug

What happened:

I updated a StatefulSet with a non-existent Docker image. As expected, a pod of the statefulset is destroyed and can't be recreated (ErrImagePull). However, when I change back the StatefulSet with an existing image, the StatefulSet doesn't try to remove the broken pod to replace it by a good one. It keeps trying to pull the non-existing image.
You have to delete the broken pod manually to unblock the situation.

Related Stackoverflow question

What you expected to happen:

When rolling back the bad config, I expected the StatefulSet to remove the broken pod and replace it by a good one.

How to reproduce it (as minimally and precisely as possible):

  1. Deploy the following StatefulSet:
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  serviceName: "nginx"
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "standard"
      resources:
        requests:
          storage: 10Gi
  1. Once the 3 pods are running, update the StatefulSet spec and change the image to k8s.gcr.io/nginx-slim:foobar
  2. Observe the new pod failing to pull the image.
  3. Roll back the change.
  4. Observe the broken pod not being deleted.

Anything else we need to know?:

  • I observed this behaviour both on 1.8 and 1.10.
  • This seems related to the discussion in #18568

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9
.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.5-gke.3", GitCommit:"6265b9797fc8680c8395abeab12c1e3bad14069a", GitTreeState:"clean", BuildDate:"2018-07-19T23:02:51Z", GoVersi
on:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: Google Kubernetes Engine
  • OS (e.g. from /etc/os-release): COS

cc @joe-boyce

@MrTrustor
Copy link
Author

@MrTrustor MrTrustor commented Aug 10, 2018

/sig apps
/sig scheduling

Loading

@joe-boyce
Copy link

@joe-boyce joe-boyce commented Aug 20, 2018

Anybody have any ideas on this one?

Loading

@enisoc
Copy link
Member

@enisoc enisoc commented Aug 21, 2018

As far as I can tell, StatefulSet doesn't make any attempt to support this use case, namely using a rolling update to fix a StatefulSet that's in a broken state. If any of the existing Pods are broken, it appears that StatefulSet bails out before even reaching the rolling update code:

if !isRunningAndReady(replicas[i]) && monotonic {
glog.V(4).Infof(
"StatefulSet %s/%s is waiting for Pod %s to be Running and Ready",
set.Namespace,
set.Name,
replicas[i].Name)
return &status, nil
}

I haven't found any mention of this limitation in the docs, but it's possible that it was a choice made intentionally to err on the side of caution (stop and make the human decide) since stateful data is at stake and stateful Pods often have dependencies on each other (e.g. they may form a cluster/quorum).

With that said, I agree it would be ideal if StatefulSet supported this, at least for clear cases like this one where deleting a Pod that's stuck Pending is unlikely to cause any additional damage.

cc @kow3ns

Loading

@mattmb
Copy link

@mattmb mattmb commented Sep 7, 2018

+1, I just discovered this and had assumed that it would work more like the Deployment controller.

In https://github.com/yelp/paasta we are programmatically creating/updating Deployments and StatefulSets. For that to make sense I really want them to always attempt to converge to the definition.

Loading

bonifaido added a commit to banzaicloud/bank-vaults that referenced this issue Sep 13, 2018
This option gives us the option to workaround current StatefulSet limitations around updates
See: kubernetes/kubernetes#67250
By default it is false.
matyix added a commit to banzaicloud/bank-vaults that referenced this issue Sep 13, 2018
This option gives us the option to workaround current StatefulSet limitations around updates
See: kubernetes/kubernetes#67250
By default it is false.
@fejta-bot
Copy link

@fejta-bot fejta-bot commented Dec 6, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Loading

@fejta-bot
Copy link

@fejta-bot fejta-bot commented Jan 5, 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

Loading

@fejta-bot
Copy link

@fejta-bot fejta-bot commented Feb 4, 2019

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Loading

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Feb 4, 2019

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Loading

@mattmb
Copy link

@mattmb mattmb commented Feb 7, 2019

/reopen

Loading

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Feb 7, 2019

@mattmb: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Loading

@mattmb
Copy link

@mattmb mattmb commented Feb 7, 2019

Heh, well it was worth a go I suppose...

Loading

@MrTrustor
Copy link
Author

@MrTrustor MrTrustor commented Feb 8, 2019

/reopen

Loading

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Feb 8, 2019

@MrTrustor: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Loading

@k8s-ci-robot k8s-ci-robot reopened this Feb 8, 2019
@huzhengchuan
Copy link
Contributor

@huzhengchuan huzhengchuan commented Mar 6, 2019

+1 meet the same problem
I think need to rollback success. but now it blocked

Loading

@dave-tock
Copy link

@dave-tock dave-tock commented Mar 11, 2019

+1. This is a pretty big landmine in using StatefulSet, if you ever make any mistake you're stuck with just destroying your StatefulSet and starting over. IOW, if you ever make a mistake with StatefulSet, you need to cause an outage to recover :(

Loading

@krmayankk
Copy link
Contributor

@krmayankk krmayankk commented Mar 11, 2019

/remove-lifecycle rotten

Loading

@fejta-bot
Copy link

@fejta-bot fejta-bot commented Nov 30, 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Loading

@zerkms
Copy link
Contributor

@zerkms zerkms commented Dec 1, 2020

/remove-lifecycle stale

Loading

@skyfirst93
Copy link

@skyfirst93 skyfirst93 commented Dec 15, 2020

It is this fixed ?

Loading

@piyushkumar13
Copy link

@piyushkumar13 piyushkumar13 commented Dec 16, 2020

I also ran into same issue... is there any timeline to fix this issue or any alternative available ?

Loading

@wu0407
Copy link

@wu0407 wu0407 commented Dec 29, 2020

+1

Loading

@clong-
Copy link

@clong- clong- commented Apr 6, 2021

Still seeing this issue on v1.20.5.

Loading

@sidh
Copy link

@sidh sidh commented Apr 9, 2021

Aaaand we just stepped on this landmine too. :(

Loading

@fejta-bot
Copy link

@fejta-bot fejta-bot commented Jul 8, 2021

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

Loading

@ialidzhikov
Copy link
Contributor

@ialidzhikov ialidzhikov commented Jul 8, 2021

/remove-lifecycle stale

@janetkuo , can you add lifecycle/frozen to this issue?

Loading

@janetkuo
Copy link
Member

@janetkuo janetkuo commented Jul 12, 2021

/lifecycle frozen

Loading

@sftim
Copy link
Contributor

@sftim sftim commented Aug 2, 2021

From #67250 (comment):

alternatively a one-time "acknowledge and proceed" signal that would tell StatefulSet to attempt to force-fix just one Pod but then revert to normal behavior after that.

A new subresource perhaps? Maybe a RollingUpdateOverride? If so, I assume that'd need its own KEP.

Loading

@watkinsmike
Copy link

@watkinsmike watkinsmike commented Aug 6, 2021

This continues to bite us. How are people managing this in the meantime? The manual intervention of deleting the stuck pod is not very ideal.

Loading

@zerkms
Copy link
Contributor

@zerkms zerkms commented Aug 8, 2021

How are people managing this in the meantime?

manually :-(

Loading

@skorobogatydmitry
Copy link

@skorobogatydmitry skorobogatydmitry commented Sep 2, 2021

+1

Loading

@feilengcui008
Copy link

@feilengcui008 feilengcui008 commented Sep 4, 2021

+1, any update or plan for this problem?

Loading

@ashwiniramesha
Copy link

@ashwiniramesha ashwiniramesha commented Nov 5, 2021

+1 this is causing unintended bookkeeping of the statefulset names. and maintaining this information across product releases is becoming an overhead.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.