Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[k8s.io] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Scaling should happen in predictable order and halt if any stateful pod is unhealthy 21m14s #41889

Closed
deads2k opened this issue Feb 22, 2017 · 5 comments · Fixed by #42367
Closed

[k8s.io] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] Scaling should happen in predictable order and halt if any stateful pod is unhealthy 21m14s #41889

deads2k opened this issue Feb 22, 2017 · 5 comments · Fixed by #42367
Assignees
@kow3ns
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/apps Categorizes an issue or PR as relevant to SIG Apps.
Milestone
v1.6

Comments

@deads2k
Copy link
Contributor

deads2k commented Feb 22, 2017

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41821/pull-kubernetes-e2e-gce-gci/18753/

go run hack/e2e.go -v -test --test_args='--ginkgo.focus=StatefulSet\s\[k8s\.io\]\sBasic\sStatefulSet\sfunctionality\s\[StatefulSetBasic\]\sScaling\sshould\shappen\sin\spredictable\sorder\sand\shalt\sif\sany\sstateful\spod\sis\sunhealthy$'

@ncdc do you know the statefulset owners?
@deads2k deads2k added kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Feb 22, 2017
This was referenced Feb 22, 2017
Merged
Merged
@ncdc
Copy link
Member

ncdc commented Feb 22, 2017

cc @foxish @kow3ns @smarterclayton

@sttts sttts mentioned this issue Feb 22, 2017
Merged
4 tasks
@ncdc ncdc mentioned this issue Feb 22, 2017
Merged
@smarterclayton
Copy link
Contributor

I did not see this happening until recently.

@DirectXMan12 DirectXMan12 mentioned this issue Feb 23, 2017
Merged
@0xmichalis 0xmichalis mentioned this issue Feb 23, 2017
Merged
@MrHohn MrHohn mentioned this issue Feb 24, 2017
Merged
@fraenkel fraenkel mentioned this issue Feb 25, 2017
Merged
@calebamiles calebamiles added the sig/apps Categorizes an issue or PR as relevant to SIG Apps. label Feb 27, 2017
@DirectXMan12 DirectXMan12 mentioned this issue Feb 28, 2017
Merged
@0xmichalis 0xmichalis mentioned this issue Mar 1, 2017
Closed
@0xmichalis
Copy link
Contributor

The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.

@screeley44 screeley44 mentioned this issue Mar 1, 2017
Merged
@kow3ns
Copy link
Member

kow3ns commented Mar 1, 2017

I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation

@0xmichalis
Copy link
Contributor

SGTM

@kow3ns kow3ns mentioned this issue Mar 1, 2017
Merged
@calebamiles calebamiles modified the milestone: v1.6 Mar 3, 2017
k8s-github-robot pushed a commit that referenced this issue Mar 4, 2017
Merge pull request #42367 from kow3ns/ss-flakes
f750721 
Automatic merge from submit-queue (batch tested with PRs 42443, 38924, 42367, 42391, 42310)

Fix StatefulSet e2e flake

**What this PR does / why we need it**:
Fixes StatefulSet e2e flake by ensuring that the StatefulSet controller has observed the unreadiness of Pods prior to attempting to exercise scale functionality.
**Which issue this PR fixes** 
fixes #41889
```release-note
NONE
```
@liggitt liggitt mentioned this issue Mar 8, 2017
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kow3ns kow3ns

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/apps Categorizes an issue or PR as relevant to SIG Apps.
Projects
None yet
Milestone
v1.6
Development

Successfully merging a pull request may close this issue.

Fix StatefulSet e2e flake kow3ns/kubernetes
6 participants
@ncdc @calebamiles @smarterclayton @0xmichalis @deads2k @kow3ns