Bug 2014332: [4.8z] Scale fixes for pods/exgws #798

trozet · 2021-10-15T02:33:43Z

Performance and scale fixes with pods and multiple external gateways.

Address set operations like add and remove are idempotent. We can get away with only RLocking there, which will greatly improve pod add performance. There is also no need to store the ips in the addressSet struct. Signed-off-by: Tim Rozet <trozet@redhat.com>

This happens when the pod was already created but a new event of the pod is generated. I managed to see it after a ovnkube-master manual restart. Signed-off-by: Federico Paolinelli <fpaoline@redhat.com> (cherry picked from commit 7828dff)

openshift-ci · 2021-10-15T02:33:47Z

@trozet: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead
expected dependent Bugzilla bug 1997072 to target a release in 4.9.0, but it targets "4.9.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

trozet · 2021-10-15T02:34:05Z

/assign @dcbw

Dan please review this closely, there were quite a few merge conflicts.

When a gw pod gets the external gateway annotation, it adds the specific routes to the external gateway for existing pods, but it does not remove the SNAT that was added when the pod was created. Signed-off-by: Federico Paolinelli <fpaoline@redhat.com> (cherry picked from commit 8783628)

Previously nsInfo was holding not only a map of gateways per namespace, but all of the routes per pod in an external gateway enabled namespace. This means that during all external gateway route adds/deletes nsInfo would need to be locked. This creates heavy contention in cluster specifically using external gateway functionality. This breaks out the pod routes portion into its own cache, which has individual locks on a per pod basis. This allows exgw routes to be added and removed without needing nsInfo lock. Additionally, since locks are on a per pod basis, it provides less overall contention across the cache. Signed-off-by: Tim Rozet <trozet@redhat.com> (cherry picked from commit c6db422)

When a pod n number of gateways there will be n number of calls to create the same 501 policy. This commit reduces it to a single call. Signed-off-by: Tim Rozet <trozet@redhat.com> (cherry picked from commit dd836a7)

trozet · 2021-10-15T14:34:28Z

/assign @fedepaol

trozet · 2021-10-16T01:00:05Z

/retest

trozet · 2021-10-16T13:06:18Z

/retest

fedepaol · 2021-10-18T06:42:16Z

/lgtm

openshift-ci · 2021-10-18T06:43:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fedepaol, trozet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [trozet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-10-19T01:00:40Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-10-19T01:00:44Z

@openshift-bot: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

anuragthehatter · 2021-10-19T14:30:26Z

/bugzilla cc-qa

openshift-ci · 2021-10-19T14:32:32Z

@anuragthehatter: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mffiedler · 2021-10-19T21:58:16Z

Failed verification on a cluster-bot cluster built from this PR (similar to the 4.9 bug 1997072 - see that bz for must-gather).

Cluster was a 120 node OVN cluster on AWS and the workload was node-density light. Many FailedCreatePodSandBox events with reason "timed out waiting for annotations" are seen and pods take a long time for all to go Running.

On 4.10 latest nightly, the issue can not be reproduced - no annotation timeout events for node-density light in the same cluster configuration

openshift-bot · 2021-10-20T01:01:58Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-bot · 2021-10-20T01:02:07Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-bot · 2021-10-20T01:02:22Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-10-20T01:02:52Z

@openshift-bot: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-10-29T01:00:35Z

@openshift-bot: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-10-30T01:00:50Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-10-30T01:00:54Z

@openshift-bot: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-10-31T01:00:28Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-10-31T01:00:33Z

@openshift-bot: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-11-01T01:00:29Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-01T01:00:33Z

@openshift-bot: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-11-02T01:00:32Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-02T01:00:37Z

@openshift-bot: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-11-03T01:00:47Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-03T01:00:50Z

@openshift-bot: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-11-04T01:00:27Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-04T01:00:35Z

@openshift-bot: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mffiedler · 2021-11-04T18:39:05Z

Tested on 4.8 cluster-bot cluster using the workload from https://bugzilla.redhat.com/show_bug.cgi?id=2014332#c6
/label qe-approved

openshift-bot · 2021-11-05T01:00:32Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-05T01:00:36Z

@openshift-bot: This pull request references Bugzilla bug 2014332, which is invalid:

expected dependent Bugzilla bug 1997072 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dcbw · 2021-11-05T14:27:30Z

/bugzilla refresh

openshift-ci · 2021-11-05T14:27:36Z

@dcbw: This pull request references Bugzilla bug 2014332, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.z) matches configured target release for branch (4.8.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1997072 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 1997072 targets the "4.9.z" release, which is one of the valid target releases: 4.9.0, 4.9.z
bug has dependents

Requesting review from QA contact:
/cc @anuragthehatter

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

trozet · 2021-11-05T14:35:37Z

/label backport-risk-assessed

mffiedler · 2021-11-05T15:28:25Z

/label cherry-pick-approved

openshift-bot · 2021-11-05T17:03:24Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-11-05T17:29:23Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-11-05T18:56:01Z

@trozet: All pull requests linked via external trackers have merged:

openshift/ovn-kubernetes#798

Bugzilla bug 2014332 has been moved to the MODIFIED state.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

trozet and others added 2 commits October 14, 2021 21:35

openshift-ci bot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Oct 15, 2021

openshift-ci bot assigned dcbw Oct 15, 2021

openshift-ci bot requested review from dcbw and rcarrillocruz October 15, 2021 02:34

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 15, 2021

fedepaol and others added 3 commits October 15, 2021 10:22

Reduce 501 logical route policy creates/deletes

e55038f

When a pod n number of gateways there will be n number of calls to create the same 501 policy. This commit reduces it to a single call. Signed-off-by: Tim Rozet <trozet@redhat.com> (cherry picked from commit dd836a7)

trozet force-pushed the bz2014332 branch from 01bc700 to e55038f Compare October 15, 2021 14:34

openshift-ci bot assigned fedepaol Oct 15, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 18, 2021

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Nov 4, 2021

openshift-ci bot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Nov 5, 2021

openshift-ci bot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Nov 5, 2021

openshift-ci bot requested a review from anuragthehatter November 5, 2021 14:27

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Nov 5, 2021

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Nov 5, 2021

openshift-merge-robot merged commit f1d74e3 into openshift:release-4.8 Nov 5, 2021

Bug 2014332: [4.8z] Scale fixes for pods/exgws #798

Bug 2014332: [4.8z] Scale fixes for pods/exgws #798

Conversation

trozet commented Oct 15, 2021

openshift-ci bot commented Oct 15, 2021

trozet commented Oct 15, 2021

trozet commented Oct 15, 2021

trozet commented Oct 16, 2021

trozet commented Oct 16, 2021

fedepaol commented Oct 18, 2021

openshift-ci bot commented Oct 18, 2021

openshift-bot commented Oct 19, 2021

openshift-ci bot commented Oct 19, 2021

anuragthehatter commented Oct 19, 2021

openshift-ci bot commented Oct 19, 2021

mffiedler commented Oct 19, 2021

openshift-bot commented Oct 20, 2021

openshift-bot commented Oct 20, 2021

openshift-bot commented Oct 20, 2021

openshift-ci bot commented Oct 20, 2021

openshift-ci bot commented Oct 29, 2021

openshift-bot commented Oct 30, 2021

openshift-ci bot commented Oct 30, 2021

openshift-bot commented Oct 31, 2021

openshift-ci bot commented Oct 31, 2021

openshift-bot commented Nov 1, 2021

openshift-ci bot commented Nov 1, 2021

openshift-bot commented Nov 2, 2021

openshift-ci bot commented Nov 2, 2021

openshift-bot commented Nov 3, 2021

openshift-ci bot commented Nov 3, 2021

openshift-bot commented Nov 4, 2021

openshift-ci bot commented Nov 4, 2021

mffiedler commented Nov 4, 2021

openshift-bot commented Nov 5, 2021

openshift-ci bot commented Nov 5, 2021

dcbw commented Nov 5, 2021

openshift-ci bot commented Nov 5, 2021

trozet commented Nov 5, 2021

mffiedler commented Nov 5, 2021

openshift-bot commented Nov 5, 2021

openshift-bot commented Nov 5, 2021

openshift-ci bot commented Nov 5, 2021