Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1901208: Nodes goes into NotReady state (VMware) #466

Merged
merged 2 commits into from
Dec 5, 2020
Merged

Bug 1901208: Nodes goes into NotReady state (VMware) #466

merged 2 commits into from
Dec 5, 2020

Conversation

p0lyn0mial
Copy link

@p0lyn0mial p0lyn0mial commented Nov 19, 2020 •
edited
Loading

we will carry a patch that allows the clients (Kubelet, KCM, etc.) to detect broken connections to the api server more quickly than the default TCP timeout of 15 minutes.

note: this patch will be dropped in future releases once http2/ping is introduced.

the first commit uses openshift/library-go#937

the second commit was created as follows:
manually changed go.mod to pin to github.com/openshift/library-go v0.0.0-20201123125610-83d6d67a1e98 followed by ./hack/update-vendor.sh

@p0lyn0mial
UPSTREAM: <drop>: enable TCP_USER_TIMEOUT on client connections
c085042 
it allows the client to detect broken connections to the api server more quickly than the default TCP timeout of 15 minutes.

note that it will be replaced by an HTTP ping mechanism in the future releases.
@openshift-ci-robot
Copy link

@p0lyn0mial: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

release-4-6: UPSTREAM: : enable TCP_USER_TIMEOUT on client connections

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@p0lyn0mial
Copy link
Author

/assign @sttts

@openshift-ci-robot openshift-ci-robot added the vendor-update Touching vendor dir or related files label Nov 19, 2020
@p0lyn0mial
Copy link
Author

/test integration

@p0lyn0mial
Copy link
Author

/retest

@p0lyn0mial
Copy link
Author

Both integration test (TestCertRotation and TestCertRotationContinuousRequests) seems to be permanently failing. Investigating.

@rphillips
Copy link

/retest

p0lyn0mial
p0lyn0mial commented Nov 20, 2020
View reviewed changes
vendor/github.com/openshift/library-go/pkg/network/dialer_linux.go
@@ -16,28 +15,18 @@ func dialerWithDefaultOptions() DialContext {
nd := &net.Dialer{
// TCP_USER_TIMEOUT does affect the behaviour of connect() which is controlled by this field so we set it to the same value
Timeout: 25 * time.Second,
// KeepAlive must to be set to a negative value to stop std library from applying the default values
// by doing so we ensure that the options we are interested in won't be overwritten
KeepAlive: time.Duration(-1),
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it gets overwritten here https://github.com/golang/go/blob/master/src/net/dial.go#L431

p0lyn0mial
p0lyn0mial commented Nov 20, 2020
View reviewed changes
vendor/github.com/openshift/library-go/pkg/network/dialer_linux.go Outdated
KeepAlive: time.Duration(-1),
Control: func(network, address string, con syscall.RawConn) error {
var err error
err = con.Control(func(fd uintptr) {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now we are using the original file descriptor not a copy

@p0lyn0mial
Copy link
Author

the last commit requires performing manual tests - will do that on Monday

@p0lyn0mial
Copy link
Author

/retest

@p0lyn0mial
Copy link
Author

/test e2e-agnostic-cmd

1 similar comment
@p0lyn0mial
Copy link
Author

/test e2e-agnostic-cmd

@p0lyn0mial p0lyn0mial force-pushed the release-4-6-tcp-usr-timeout-dialer branch from 2782777 to a5615fa Compare November 23, 2020 13:11
@openshift-ci-robot
Copy link

@p0lyn0mial: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

release-4-6: UPSTREAM: : enable TCP_USER_TIMEOUT on client connections

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@p0lyn0mial
Copy link
Author

@rphillips I did manual tests this morning and haven't found any issues (openshift/library-go#944 (comment)). Note that I tested it against my custom applications not kubelet.

@rphillips feel free to retitle so that it points to the BZ you want.

@p0lyn0mial
Copy link
Author

/assign @rphillips

@p0lyn0mial
Copy link
Author

/retest

1 similar comment
@p0lyn0mial
Copy link
Author

/retest

@rphillips
Copy link

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 23, 2020
@p0lyn0mial p0lyn0mial changed the title release-4-6: UPSTREAM: <drop>: enable TCP_USER_TIMEOUT on client connections Bug 1873114: Nodes goes into NotReady state (VMware) Nov 23, 2020
@openshift-ci-robot openshift-ci-robot added the bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. label Nov 23, 2020
@openshift-ci-robot
Copy link

@p0lyn0mial: This pull request references Bugzilla bug 1873114, which is invalid:

  • expected the bug to target the "4.6.z" release, but it targets "4.7.0" instead
  • expected dependent Bugzilla bug 1857446 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ASSIGNED instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1873114: Nodes goes into NotReady state (VMware)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Dec 2, 2020
@openshift-ci-robot
Copy link

@nee1esh: This pull request references Bugzilla bug 1901208, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.6.z) matches configured target release for branch (4.6.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 1873114 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA))
  • dependent Bugzilla bug 1873114 targets the "4.7.0" release, which is one of the valid target releases: 4.7.0
  • bug has dependents

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sdodson sdodson added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Dec 2, 2020
@nee1esh
Copy link

nee1esh commented Dec 2, 2020

/retest

2 similar comments
@nee1esh
Copy link

nee1esh commented Dec 3, 2020

/retest

@nee1esh
Copy link

nee1esh commented Dec 3, 2020

/retest

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

8 similar comments
@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@nee1esh
Copy link

nee1esh commented Dec 4, 2020

/retest

1 similar comment
@sdodson
Copy link
Member

sdodson commented Dec 5, 2020

/retest

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

6 similar comments
@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 7070803 into openshift:release-4.6 Dec 5, 2020
@openshift-ci-robot
Copy link

@p0lyn0mial: All pull requests linked via external trackers have merged:

Bugzilla bug 1901208 has been moved to the MODIFIED state.

In response to this:

Bug 1901208: Nodes goes into NotReady state (VMware)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@cuppett
Copy link
Member

cuppett commented Dec 15, 2020

/cherry-pick release-4.5

@openshift-cherrypick-robot
Copy link

@cuppett: cannot checkout release-4.5: error checking out release-4.5: exit status 1. output: error: pathspec 'release-4.5' did not match any file(s) known to git

In response to this:

/cherry-pick release-4.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sttts
Copy link

sttts commented Dec 15, 2020

@cuppett for 4.5 you have to bring the change into openshift/origin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bparees bparees Awaiting requested review from bparees

@damemi damemi Awaiting requested review from damemi

Assignees

@rphillips rphillips

@sttts sttts

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. lgtm Indicates that a PR is ready to be merged. vendor-update Touching vendor dir or related files
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@p0lyn0mial @openshift-ci-robot @rphillips @sttts @nee1esh @openshift-bot @sdodson @cuppett @openshift-cherrypick-robot @openshift-merge-robot