Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1779163: CNI: Don't wait for missing pods on DEL #280

Merged
merged 1 commit into from Jun 25, 2020
Merged

Bug 1779163: CNI: Don't wait for missing pods on DEL #280

merged 1 commit into from Jun 25, 2020

Conversation

MaysaMacedo
Copy link
Contributor

We've seen some issues related to the fact that on CNI DEL we wait for
pod annotation to appear in CNI registry:

  1. Overloaded kuryr-daemon HTTP server, because many stale DEL requests
    from CRI will saturate the number of free connections, causing CNI to
    answer requests extremely slowly.
  2. Some CRI's will not delete network namespace before getting a
    successful CNI DEL. If we'll assign the same IP (or vlan) to a next
    pod, we might end up with an IP or vlan conflict and NetlinkError.

This commit makes sure we only wait for VIFs up to 5 seconds on DEL
requests, enough time for watcher to populate registry on restart. Also
some code got moved around to make implementating the above simpler and
some logs got clarified to make debugging of such issues easier later
on.

Change-Id: I9221b6bc9166597837a4b53382862aa6c6f3e94c
Closes-Bug: 1882083
Related-Bug: 1854928

@openshift-ci-robot openshift-ci-robot added the bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. label Jun 11, 2020
@openshift-ci-robot
Copy link

@MaysaMacedo: This pull request references Bugzilla bug 1779163, which is invalid:

  • expected dependent Bugzilla bug 1810574 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is MODIFIED instead
  • expected dependent Bugzilla bug 1846225 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ON_QA instead
  • expected dependent Bugzilla bug 1846225 to target a release in 4.5.0, 4.5.z, but it targets "4.6.0" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1779163: CNI: Don't wait for missing pods on DEL

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 11, 2020
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 11, 2020
@MaysaMacedo
Copy link
Contributor Author

/bugzilla refresh

@openshift-ci-robot
Copy link

@MaysaMacedo: This pull request references Bugzilla bug 1779163, which is invalid:

  • expected dependent Bugzilla bug 1810574 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is MODIFIED instead
  • expected dependent Bugzilla bug 1846225 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ON_QA instead
  • expected dependent Bugzilla bug 1846225 to target a release in 4.5.0, 4.5.z, but it targets "4.6.0" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@MaysaMacedo
Copy link
Contributor Author

/bugzilla refresh

@openshift-ci-robot
Copy link

@MaysaMacedo: This pull request references Bugzilla bug 1779163, which is invalid:

  • expected dependent Bugzilla bug 1810574 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is MODIFIED instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@MaysaMacedo
Copy link
Contributor Author

/test verify

@luis5tb
Copy link
Contributor

luis5tb commented Jun 15, 2020

/retest

@MaysaMacedo
Copy link
Contributor Author

/test unit

@luis5tb
Copy link
Contributor

luis5tb commented Jun 16, 2020

/bugzilla refresh

@openshift-ci-robot
Copy link

@luis5tb: This pull request references Bugzilla bug 1779163, which is invalid:

  • expected dependent Bugzilla bug 1810574 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@luis5tb
Copy link
Contributor

luis5tb commented Jun 16, 2020

/test verify

@luis5tb
Copy link
Contributor

luis5tb commented Jun 16, 2020

/test unit

1 similar comment
@luis5tb
Copy link
Contributor

luis5tb commented Jun 17, 2020

/test unit

@luis5tb
Copy link
Contributor

luis5tb commented Jun 17, 2020

/bugzilla refresh

@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jun 17, 2020
@openshift-ci-robot
Copy link

@luis5tb: This pull request references Bugzilla bug 1779163, which is valid.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.4.z) matches configured target release for branch (4.4.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 1810574 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA))
  • dependent Bugzilla bug 1810574 targets the "4.5.0" release, which is one of the valid target releases: 4.5.0, 4.5.z
  • bug has dependents

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@luis5tb
Copy link
Contributor

luis5tb commented Jun 17, 2020

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 17, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

4 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

13 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@MaysaMacedo
Copy link
Contributor Author

/retest

@MaysaMacedo MaysaMacedo force-pushed the dont-wait-for-missing-pods-del branch from 1f06e93 to 90ddd40 Compare June 18, 2020 16:01
@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jun 18, 2020
@openshift-ci-robot openshift-ci-robot mentioned this pull request Jun 18, 2020
Merged
@dulek @MaysaMacedo
CNI: Don't wait for missing pods on DEL
586d639 
We've seen some issues related to the fact that on CNI DEL we wait for
pod annotation to appear in CNI registry:

1. Overloaded kuryr-daemon HTTP server, because many stale DEL requests
   from CRI will saturate the number of free connections, causing CNI to
   answer requests extremely slowly.
2. Some CRI's will not delete network namespace before getting a
   successful CNI DEL. If we'll assign the same IP (or vlan) to a next
   pod, we might end up with an IP or vlan conflict and NetlinkError.

This commit makes sure we only wait for VIFs up to 5 seconds on DEL
requests, enough time for watcher to populate registry on restart. Also
some code got moved around to make implementating the above simpler and
some logs got clarified to make debugging of such issues easier later
on.

Change-Id: I9221b6bc9166597837a4b53382862aa6c6f3e94c
Closes-Bug: 1882083
Related-Bug: 1854928
@MaysaMacedo MaysaMacedo force-pushed the dont-wait-for-missing-pods-del branch from 90ddd40 to 586d639 Compare June 19, 2020 07:53
@MaysaMacedo
Copy link
Contributor Author

/test images

2 similar comments
@MaysaMacedo
Copy link
Contributor Author

/test images

@MaysaMacedo
Copy link
Contributor Author

/test images

@luis5tb
Copy link
Contributor

luis5tb commented Jun 19, 2020

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 19, 2020
@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: luis5tb, MaysaMacedo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [MaysaMacedo,luis5tb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sdodson sdodson added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jun 25, 2020
@openshift-merge-robot openshift-merge-robot merged commit 84672f8 into openshift:release-4.4 Jun 25, 2020
@openshift-ci-robot
Copy link

@MaysaMacedo: All pull requests linked via external trackers have merged: openshift/kuryr-kubernetes#280, openshift/kuryr-kubernetes#246, openshift/kuryr-kubernetes#182, openshift/kuryr-kubernetes#119. Bugzilla bug 1779163 has been moved to the MODIFIED state.

In response to this:

Bug 1779163: CNI: Don't wait for missing pods on DEL

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

MaysaMacedo pushed a commit to MaysaMacedo/kuryr-kubernetes-1 that referenced this pull request Jul 1, 2021
@openshift-merge-robot
Merge pull request openshift#280 from danehans/proxy_reconciler_fix
146bc27 
Fixes proxy validation error formatting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@celebdor celebdor Awaiting requested review from celebdor

@luis5tb luis5tb Awaiting requested review from luis5tb

Assignees

@luis5tb luis5tb

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@MaysaMacedo @openshift-ci-robot @luis5tb @openshift-bot @sdodson @openshift-merge-robot @dulek