Bug 2014360: [4.7z] Scale fixes for pods/exgws #799

trozet · 2021-10-15T14:13:30Z

Also includes missing exgw fixes.

Address set operations like add and remove are idempotent. We can get away with only RLocking there, which will greatly improve pod add performance. There is also no need to store the ips in the addressSet struct. Signed-off-by: Tim Rozet <trozet@redhat.com>

This happens when the pod was already created but a new event of the pod is generated. I managed to see it after a ovnkube-master manual restart. Signed-off-by: Federico Paolinelli <fpaoline@redhat.com> (cherry picked from commit 7828dff)

When a gw pod gets the external gateway annotation, it adds the specific routes to the external gateway for existing pods, but it does not remove the SNAT that was added when the pod was created. Signed-off-by: Federico Paolinelli <fpaoline@redhat.com> (cherry picked from commit 8783628)

When adding routes to pod, we fail inside the inner loop returning an error. What happens is that if the pod has two ip addresses, and the gw is set only for the second address, the function will mistakenly return an error. Signed-off-by: Federico Paolinelli <fpaoline@redhat.com> (cherry picked from commit d2e0593) (cherry picked from commit fae2540)

Previously nsInfo was holding not only a map of gateways per namespace, but all of the routes per pod in an external gateway enabled namespace. This means that during all external gateway route adds/deletes nsInfo would need to be locked. This creates heavy contention in cluster specifically using external gateway functionality. This breaks out the pod routes portion into its own cache, which has individual locks on a per pod basis. This allows exgw routes to be added and removed without needing nsInfo lock. Additionally, since locks are on a per pod basis, it provides less overall contention across the cache. Signed-off-by: Tim Rozet <trozet@redhat.com> (cherry picked from commit c6db422)

When a pod n number of gateways there will be n number of calls to create the same 501 policy. This commit reduces it to a single call. Signed-off-by: Tim Rozet <trozet@redhat.com> (cherry picked from commit dd836a7)

openshift-ci · 2021-10-15T14:13:35Z

@trozet: This pull request references Bugzilla bug 2014360, which is invalid:

expected dependent Bugzilla bug 2014332 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

trozet · 2021-10-15T14:13:44Z

/assign @fedepaol

trozet · 2021-10-15T14:13:55Z

/assign @dcbw

trozet · 2021-10-16T00:59:25Z

/retest

fedepaol · 2021-10-18T06:24:38Z

/lgtm

openshift-bot · 2021-10-19T01:00:40Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-10-19T01:00:44Z

@openshift-bot: This pull request references Bugzilla bug 2014360, which is invalid:

expected dependent Bugzilla bug 2014332 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mffiedler · 2021-10-19T22:51:48Z

Failed verification on a cluster-bot cluster built from this PR (similar to the 4.9 bug 1997072 - see that bz for must-gather).

Cluster was a 120 node OVN cluster on AWS and the workload was node-density light. Many FailedCreatePodSandBox events with reason "timed out waiting for annotations" are seen and pods take a long time for all to go Running.

On 4.10 latest nightly, the issue can not be reproduced - no annotation timeout events for node-density light in the same cluster configuration

openshift-bot · 2021-10-20T01:01:53Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-10-20T01:02:45Z

@openshift-bot: This pull request references Bugzilla bug 2014360, which is invalid:

expected dependent Bugzilla bug 2014332 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-10-21T01:00:21Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-10-21T01:00:32Z

@openshift-bot: This pull request references Bugzilla bug 2014360, which is invalid:

expected dependent Bugzilla bug 2014332 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-10-22T01:00:21Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-10-22T01:00:26Z

@openshift-bot: This pull request references Bugzilla bug 2014360, which is invalid:

expected dependent Bugzilla bug 2014332 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-10-23T01:00:15Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-bot · 2021-11-01T01:00:29Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-01T01:00:33Z

@openshift-bot: This pull request references Bugzilla bug 2014360, which is invalid:

expected dependent Bugzilla bug 2014332 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-11-02T01:00:32Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-02T01:00:36Z

@openshift-bot: This pull request references Bugzilla bug 2014360, which is invalid:

expected dependent Bugzilla bug 2014332 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-11-03T01:00:46Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-03T01:00:50Z

@openshift-bot: This pull request references Bugzilla bug 2014360, which is invalid:

expected dependent Bugzilla bug 2014332 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-11-04T01:00:27Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-04T01:00:34Z

@openshift-bot: This pull request references Bugzilla bug 2014360, which is invalid:

expected dependent Bugzilla bug 2014332 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-11-05T01:00:32Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-05T01:00:36Z

@openshift-bot: This pull request references Bugzilla bug 2014360, which is invalid:

expected dependent Bugzilla bug 2014332 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mffiedler · 2021-11-05T14:50:14Z

Verified on cluster-bot AWS cluster build from this PR using the workload described in https://bugzilla.redhat.com/show_bug.cgi?id=2014360#c4. No annotation timeout errors and all pods came to Running state

/label qe-approved

trozet · 2021-11-05T14:59:22Z

/label backport-risk-assessed

openshift-bot · 2021-11-06T01:00:34Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-06T01:00:39Z

@openshift-bot: This pull request references Bugzilla bug 2014360, which is invalid:

expected dependent Bugzilla bug 2014332 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mffiedler · 2021-11-06T18:36:13Z

/bugzilla refresh

openshift-ci · 2021-11-06T18:36:18Z

@mffiedler: This pull request references Bugzilla bug 2014360, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.z) matches configured target release for branch (4.7.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 2014332 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 2014332 targets the "4.8.z" release, which is one of the valid target releases: 4.8.0, 4.8.z
bug has dependents

Requesting review from QA contact:
/cc @mffiedler

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mffiedler · 2021-11-06T18:38:04Z

/label cherry-pick-approved

openshift-ci · 2021-11-06T18:39:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fedepaol, mffiedler, trozet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [trozet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-11-06T19:57:23Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-06T20:22:24Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-06T20:35:25Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-11-06T21:54:21Z

@trozet: All pull requests linked via external trackers have merged:

openshift/ovn-kubernetes#799

Bugzilla bug 2014360 has been moved to the MODIFIED state.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

trozet and others added 6 commits October 14, 2021 22:38

Reduce 501 logical route policy creates/deletes

f40a055

When a pod n number of gateways there will be n number of calls to create the same 501 policy. This commit reduces it to a single call. Signed-off-by: Tim Rozet <trozet@redhat.com> (cherry picked from commit dd836a7)

openshift-ci bot added the bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. label Oct 15, 2021

openshift-ci bot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Oct 15, 2021

openshift-ci bot assigned fedepaol Oct 15, 2021

openshift-ci bot assigned dcbw Oct 15, 2021

openshift-ci bot requested review from abhat and juanluisvaladas October 15, 2021 14:14

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 15, 2021

dcbw changed the title ~~Bug 2014360: Scale fixes for pod and external gateways~~ Bug 2014360: [4.7z] Scale fixes for pods/exgws Oct 15, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 18, 2021

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Nov 5, 2021

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Nov 5, 2021

openshift-ci bot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Nov 6, 2021

openshift-ci bot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Nov 6, 2021

openshift-ci bot requested a review from mffiedler November 6, 2021 18:36

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Nov 6, 2021

mffiedler approved these changes Nov 6, 2021

View reviewed changes

openshift-merge-robot merged commit 8ca2b7c into openshift:release-4.7 Nov 6, 2021

Bug 2014360: [4.7z] Scale fixes for pods/exgws #799

Bug 2014360: [4.7z] Scale fixes for pods/exgws #799

Conversation

trozet commented Oct 15, 2021

openshift-ci bot commented Oct 15, 2021

trozet commented Oct 15, 2021

trozet commented Oct 15, 2021

trozet commented Oct 16, 2021

fedepaol commented Oct 18, 2021

openshift-bot commented Oct 19, 2021

openshift-ci bot commented Oct 19, 2021

mffiedler commented Oct 19, 2021

openshift-bot commented Oct 20, 2021

openshift-ci bot commented Oct 20, 2021

openshift-bot commented Oct 21, 2021

openshift-ci bot commented Oct 21, 2021

openshift-bot commented Oct 22, 2021

openshift-ci bot commented Oct 22, 2021

openshift-bot commented Oct 23, 2021

openshift-bot commented Nov 1, 2021

openshift-ci bot commented Nov 1, 2021

openshift-bot commented Nov 2, 2021

openshift-ci bot commented Nov 2, 2021

openshift-bot commented Nov 3, 2021

openshift-ci bot commented Nov 3, 2021

openshift-bot commented Nov 4, 2021

openshift-ci bot commented Nov 4, 2021

openshift-bot commented Nov 5, 2021

openshift-ci bot commented Nov 5, 2021

mffiedler commented Nov 5, 2021

trozet commented Nov 5, 2021

openshift-bot commented Nov 6, 2021

openshift-ci bot commented Nov 6, 2021

mffiedler commented Nov 6, 2021

openshift-ci bot commented Nov 6, 2021

mffiedler commented Nov 6, 2021

openshift-ci bot commented Nov 6, 2021

openshift-bot commented Nov 6, 2021

openshift-bot commented Nov 6, 2021

openshift-bot commented Nov 6, 2021

openshift-ci bot commented Nov 6, 2021