Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flake #99979 increase delay for ready state propagation #103593

Merged
merged 1 commit into from
Aug 5, 2021
Merged

Flake #99979 increase delay for ready state propagation #103593

merged 1 commit into from
Aug 5, 2021

Conversation

matthyx
Copy link
Contributor

@matthyx matthyx commented Jul 8, 2021

What type of PR is this?

/kind flake
/priority important-soon

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #99979

Special notes for your reviewer:

As discussed during Sig-Node CI meeting: increasing delay for state propagation on busy machine.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/flake Categorizes issue or PR as related to a flaky test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. area/test sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 8, 2021
@matthyx
Copy link
Contributor Author

matthyx commented Jul 8, 2021

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 8, 2021
@matthyx matthyx added this to PRs - Needs Reviewer in SIG Node CI/Test Board Jul 8, 2021
@matthyx
Copy link
Contributor Author

matthyx commented Jul 8, 2021

/assign @ehashman

ehashman
ehashman reviewed Jul 8, 2021
View reviewed changes
test/e2e/common/node/container_probe.go Outdated
@@ -445,7 +445,7 @@ var _ = SIGDescribe("Probing container", func() {
if readyIn < 0 {
framework.Failf("Pod became ready before startupProbe succeeded")
}
if readyIn > 5*time.Second {
if readyIn > 12*time.Second {
framework.Failf("Pod became ready in %v, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.", readyIn)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to update the error message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, done

@matthyx
Copy link
Contributor Author

matthyx commented Jul 8, 2021

/hold
Need to work a bit more on that with @ehashman

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 8, 2021
@ehashman
Copy link
Member

ehashman commented Jul 8, 2021

Slack discussion: https://kubernetes.slack.com/archives/C0BP8PW9G/p1625773793079000

@ehashman ehashman moved this from PRs - Needs Reviewer to PRs Waiting on Author in SIG Node CI/Test Board Jul 8, 2021
@matthyx
Copy link
Contributor Author

matthyx commented Jul 8, 2021

@matthyx: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-integration 61291ca link /test pull-kubernetes-integration
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

at least it's not that flake 😅

@matthyx
Copy link
Contributor Author

matthyx commented Jul 10, 2021

Slack discussion: https://kubernetes.slack.com/archives/C0BP8PW9G/p1625773793079000

@ehashman answers to slack points: https://hackmd.io/8BH1yq8fR9W6ZWIauHyT7w?view

@ehashman
Copy link
Member

Discussed in SIG Node CI meeting 07/14 -- we will adopt the proposal outlined here https://hackmd.io/8BH1yq8fR9W6ZWIauHyT7w?view#Proposed-improvement

to update the frequencies in the e2e test to decrease flakiness, and take a backlog item to fix the possible race identified here: https://hackmd.io/8BH1yq8fR9W6ZWIauHyT7w?view#Possible-race-condition-in-prober_manager

@matthyx to file/update.

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 15, 2021
@matthyx
Copy link
Contributor Author

matthyx commented Jul 15, 2021

/retest

@matthyx matthyx mentioned this pull request Jul 15, 2021
Closed
@endocrimes endocrimes added this to Needs Reviewer in SIG Node PR Triage Jul 16, 2021
@endocrimes endocrimes moved this from Needs Reviewer to Waiting on Author in SIG Node PR Triage Jul 16, 2021
ehashman
ehashman reviewed Jul 23, 2021
View reviewed changes
Copy link
Member

@ehashman ehashman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming the underlying code isn't buggy, this seems like a correct adjustment to me. 25s < 30s but gives the startup probe a few tries to run and kick the readiness state.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 23, 2021
@ehashman
Copy link
Member

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 23, 2021
@ehashman ehashman moved this from Waiting on Author to Needs Approver in SIG Node PR Triage Jul 23, 2021
@ehashman ehashman moved this from PRs Waiting on Author to PRs - Needs Approver in SIG Node CI/Test Board Jul 23, 2021
@ehashman ehashman mentioned this pull request Jul 28, 2021
Closed
ehashman
ehashman approved these changes Aug 5, 2021
View reviewed changes
Copy link
Member

@ehashman ehashman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

let's see if this helps with flakiness

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ehashman, matthyx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 5, 2021
@matthyx
Copy link
Contributor Author

matthyx commented Aug 5, 2021

/approve

let's see if this helps with flakiness

Superpowers!

@k8s-ci-robot k8s-ci-robot merged commit 20d5526 into kubernetes:master Aug 5, 2021
SIG Node CI/Test Board automation moved this from PRs - Needs Approver to Done Aug 5, 2021
SIG Node PR Triage automation moved this from Needs Approver to Done Aug 5, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Aug 5, 2021
@matthyx matthyx deleted the container_probe branch November 23, 2021 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ehashman ehashman ehashman approved these changes

@andrewsykim andrewsykim Awaiting requested review from andrewsykim

@SergeyKanzhelev SergeyKanzhelev Awaiting requested review from SergeyKanzhelev

Assignees

@ehashman ehashman

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note-none Denotes a PR that doesn't merit a release note. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node CI/Test Board
Done
SIG Node CI/Test Board
Archived in project
SIG Node PR Triage
Archived in project
SIG Node PR Triage
Done
Milestone
v1.23
Development

Successfully merging this pull request may close these issues.

[flaky test]: [sig-node] Probing container should be ready immediately after startupProbe succeeds
3 participants
@matthyx @ehashman @k8s-ci-robot