Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure ovnkube-controller does not go remote->local #3735

Merged
merged 1 commit into from Jul 6, 2023

Conversation

trozet
Copy link
Contributor

@trozet trozet commented Jun 30, 2023

At ovnkube-controller start up, it may attempt to start adding resources for a node that does not yet have its annotation, but should be considered local. This is especially the case in OpenShift when going from phase1 -> phase 2 upgrade and ovnkube-node on the node will annotate its zone id. In this case ovnkube-controller will start too early and program NBDB resources as remote for the node. Then when the node is annotated, it will program NBDB resources as local.

Since we cannot be sure that all remote feature configuration in NBDB is removed when going from remote -> local we need to avoid this case. It also makes no sense to program NBDB when there are no nodes in the zone.

This commit adds a wait to make sure at least one node in the cluster is in the ovnkube-controller's managed zone.

Thanks @numansiddique for the idea.

@trozet
Copy link
Contributor Author

trozet commented Jun 30, 2023

@tssurya @numansiddique FYI

@trozet trozet force-pushed the fix_remote_to_local_ic branch 4 times, most recently from 637ba65 to 733e073 Compare July 1, 2023 04:12
@coveralls
Copy link

Coverage Status

coverage: 53.362% (-0.08%) from 53.444% when pulling 733e073 on trozet:fix_remote_to_local_ic into 6870575 on ovn-org:master.

Copy link
Contributor

@jcaamano jcaamano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would have benefited from a waitForValidZone method somewhere

At ovnkube-controller start up, it may attempt to start adding resources
for a node that does not yet have its annotation, but should be
considered local. This is especially the case in OpenShift when going
from phase1 -> phase 2 upgrade and ovnkube-node on the node will
annotate its zone id. In this case ovnkube-controller will start too
early and program NBDB resources as remote for the node. Then when the
node is annotated, it will program NBDB resources as local.

Since we cannot be sure that all remote feature configuration in NBDB is
removed when going from remote -> local we need to avoid this case. It
also makes no sense to program NBDB when there are no nodes in the zone.

This commit adds a wait to make sure at least one node in the cluster
is in the ovnkube-controller's managed zone.

Thanks @numansiddique for the idea.

Signed-off-by: Tim Rozet <trozet@redhat.com>
@jcaamano jcaamano merged commit a3dd7de into ovn-org:master Jul 6, 2023
26 checks passed
@tssurya
Copy link
Member

tssurya commented Jul 6, 2023

Since we cannot be sure that all remote feature configuration in NBDB is removed when going from remote -> local we need to avoid this case. It also makes no sense to program NBDB when there are no nodes in the zone.

hmm I thought we spoke about local->remote here on friday, I was a bit confused about @trozet 's comment here: https://github.com/ovn-org/ovn-kubernetes/pull/3735/files#diff-cf3d012028107d0065b038f3ddf3f2fd82b14518b09aa7193b45bb5ef7eefebfR334 which mentions local -> remote but the title and description in the PR are focused on remote -> local, was about to talk to @trozet when he is back, but I guess its ok we merged this since its harmless.

@flavio-fernandes
Copy link
Contributor

this since its harmless

... unless there are no nodes! 😛
See: https://issues.redhat.com/browse/OCPBUGS-16767

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants