Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OCPBUGS-16629: Increase gateway migration rollout for OVNK-IC #42165

Merged

Conversation

tssurya
Copy link
Contributor

@tssurya tssurya commented Aug 9, 2023

@openshift-ci-robot openshift-ci-robot added the jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. label Aug 9, 2023
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 9, 2023
@openshift-ci-robot
Copy link
Contributor

@tssurya: This pull request references Jira Issue OCPBUGS-16629, which is invalid:

  • expected the bug to target the "4.14.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/cc @anuragthehatter @jluhrsen @trozet

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Aug 9, 2023
@tssurya
Copy link
Contributor Author

tssurya commented Aug 9, 2023

/jira refresh

@openshift-ci-robot
Copy link
Contributor

@tssurya: This pull request references Jira Issue OCPBUGS-16629, which is invalid:

  • expected the bug to target the "4.14.0" version, but it targets "4.14" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@tssurya
Copy link
Contributor Author

tssurya commented Aug 9, 2023

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 9, 2023
@openshift-ci-robot
Copy link
Contributor

@tssurya: This pull request references Jira Issue OCPBUGS-16629, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.14.0) matches configured target version for branch (4.14.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @anuragthehatter

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jluhrsen
Copy link
Contributor

jluhrsen commented Aug 9, 2023

/hold
I'm trying to nail down what/where this extra timing got introduced and I'm pushing PRs and running this job to
investigate. if this makes it in, the jobs will pass and it will be harder to figure out. Unless there is some other
way I can do this?

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 9, 2023
@tssurya
Copy link
Contributor Author

tssurya commented Aug 10, 2023

/hold I'm trying to nail down what/where this extra timing got introduced and I'm pushing PRs and running this job to investigate. if this makes it in, the jobs will pass and it will be harder to figure out. Unless there is some other way I can do this?

I think from the job itself we should be able to measure the time for OVNK components versus K8s rollout perhaps and go with that? You should also be able to do this manually - the migration switch I mean and see what's happening there..

We anyways need to do this I guess for IC. But I am OK to wait if you think this will help your investigations...
I'll let you decide when you want to unhold...

@tssurya
Copy link
Contributor Author

tssurya commented Aug 14, 2023

/pj-rehearse

@jluhrsen
Copy link
Contributor

/hold cancel
/approve
/lgtm

This is going to affect all jobs, not just master. There has definitely been a regression so adding this extra timeout is just ignoring that for now. I have a bug assigned to me that we can use to track this regression. It's possible we will just ignore it, but I need to point out that this is not fixing anything just allowing more time for whatever started taking
longer.

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 15, 2023
@openshift-ci openshift-ci bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Aug 15, 2023
@jluhrsen
Copy link
Contributor

/hold

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 15, 2023
# FIXME: Increasing timeout to 15minutes for OVNK IC deployments (original value was 360seconds)
# See https://issues.redhat.com/browse/OCPBUGS-16629 for details
timeout 900s oc rollout status ds/ovnkube-node -n openshift-ovn-kubernetes
timeout 900s oc rollout status ds/ovnkube-master -n openshift-ovn-kubernetes
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+ timeout 900s oc rollout status ds/ovnkube-master -n openshift-ovn-kubernetes
Error from server (NotFound): daemonsets.apps "ovnkube-master" not found

this is not ovnkube-master any more in IC world, its ovnkube-control-plane

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also its not daemonset anymore,
its deployment

@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Aug 17, 2023
Signed-off-by: Surya Seetharaman <suryaseetharaman.9@gmail.com>
@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@tssurya: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-cluster-network-operator-master-e2e-aws-ovn-local-to-shared-gateway-mode-migration openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.15-e2e-aws-ovn-local-to-shared-gateway-mode-migration openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.14-e2e-aws-ovn-local-to-shared-gateway-mode-migration openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-master-e2e-aws-ovn-shared-to-local-gateway-mode-migration-periodic openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.15-e2e-aws-ovn-shared-to-local-gateway-mode-migration-periodic openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.14-e2e-aws-ovn-shared-to-local-gateway-mode-migration-periodic openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-local-to-shared-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.15-e2e-aws-ovn-local-to-shared-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.14-e2e-aws-ovn-local-to-shared-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn-local-to-shared-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.12-e2e-aws-ovn-local-to-shared-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.11-e2e-aws-ovn-local-to-shared-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.10-e2e-aws-ovn-local-to-shared-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-shared-to-local-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.15-e2e-aws-ovn-shared-to-local-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.14-e2e-aws-ovn-shared-to-local-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn-shared-to-local-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.12-e2e-aws-ovn-shared-to-local-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.11-e2e-aws-ovn-shared-to-local-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
pull-ci-openshift-ovn-kubernetes-release-4.10-e2e-aws-ovn-shared-to-local-gateway-mode-migration openshift/ovn-kubernetes presubmit Registry content changed
periodic-ci-openshift-ovn-kubernetes-release-4.10-e2e-aws-ovn-shared-to-local-gateway-mode-migration-periodic N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.11-amd64-nightly-vsphere-ipi-ovn-shared-to-local-gateway-mode-migration-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.11-amd64-nightly-vsphere-ipi-ovn-local-to-shared-gateway-mode-migration-f28 N/A periodic Registry content changed
periodic-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-shared-to-local-gateway-mode-migration-periodic N/A periodic Registry content changed
periodic-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-local-to-shared-gateway-mode-migration-periodic N/A periodic Registry content changed
periodic-ci-openshift-ovn-kubernetes-release-4.12-e2e-aws-ovn-shared-to-local-gateway-mode-migration-periodic N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.10-amd64-nightly-vsphere-ipi-ovn-local-to-shared-gateway-mode-migration-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.10-amd64-nightly-vsphere-ipi-ovn-shared-to-local-gateway-mode-migration-f28 N/A periodic Registry content changed
periodic-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn-shared-to-local-gateway-mode-migration-periodic N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-vsphere-ipi-ovn-shared-to-local-gateway-mode-migration-p2-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-vsphere-ipi-ovn-local-to-shared-gateway-mode-migration-p2-f14 N/A periodic Registry content changed
periodic-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn-local-to-shared-gateway-mode-migration-periodic N/A periodic Registry content changed
periodic-ci-openshift-ovn-kubernetes-release-4.11-e2e-aws-ovn-local-to-shared-gateway-mode-migration-periodic N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-ec-vsphere-ipi-ovn-shared-to-local-gateway-mode-migration-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-vsphere-ipi-ovn-local-to-shared-gateway-mode-migration-f14 N/A periodic Registry content changed

A total of 43 jobs have been affected by this change. The above listing is non-exhaustive and limited to 35 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 10 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 20 rehearsals
Comment: /pj-rehearse max to run up to 35 rehearsals
Comment: /pj-rehearse auto-ack to run up to 10 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@tssurya
Copy link
Contributor Author

tssurya commented Aug 17, 2023

/pj-rehearse

@tssurya
Copy link
Contributor Author

tssurya commented Aug 17, 2023

/retest

1 similar comment
@tssurya
Copy link
Contributor Author

tssurya commented Aug 21, 2023

/retest

@tssurya
Copy link
Contributor Author

tssurya commented Aug 21, 2023

/pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.14-e2e-aws-ovn-shared-to-local-gateway-mode-migration
/pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.15-e2e-aws-ovn-local-to-shared-gateway-mode-migration
/pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn-local-to-shared-gateway-mode-migration

@tssurya
Copy link
Contributor Author

tssurya commented Aug 21, 2023

/pj-rehearse pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-local-to-shared-gateway-mode-migration
/pj-rehearse pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-shared-to-local-gateway-mode-migration

@tssurya
Copy link
Contributor Author

tssurya commented Aug 21, 2023

/pj-rehearse pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-shared-to-local-gateway-mode-migration

@tssurya
Copy link
Contributor Author

tssurya commented Aug 21, 2023

/pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.15-e2e-aws-ovn-local-to-shared-gateway-mode-migration

@tssurya
Copy link
Contributor Author

tssurya commented Aug 21, 2023

/pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn-local-to-shared-gateway-mode-migration

@tssurya
Copy link
Contributor Author

tssurya commented Aug 21, 2023

/pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.15-e2e-aws-ovn-shared-to-local-gateway-mode-migration

@tssurya
Copy link
Contributor Author

tssurya commented Aug 21, 2023

/hold cancel
fixed the issue we were facing

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 21, 2023
@tssurya
Copy link
Contributor Author

tssurya commented Aug 22, 2023

/pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.14-e2e-aws-ovn-shared-to-local-gateway-mode-migration

@tssurya
Copy link
Contributor Author

tssurya commented Aug 22, 2023

/pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.14-e2e-aws-ovn-local-to-shared-gateway-mode-migration

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 22, 2023

@tssurya: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/openshift/ovn-kubernetes/release-4.10/e2e-aws-ovn-local-to-shared-gateway-mode-migration 4e40324 link unknown /pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.10-e2e-aws-ovn-local-to-shared-gateway-mode-migration
ci/rehearse/openshift/ovn-kubernetes/release-4.12/e2e-aws-ovn-local-to-shared-gateway-mode-migration 4e40324 link unknown /pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.12-e2e-aws-ovn-local-to-shared-gateway-mode-migration
ci/rehearse/openshift/ovn-kubernetes/release-4.11/e2e-aws-ovn-local-to-shared-gateway-mode-migration 4e40324 link unknown /pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.11-e2e-aws-ovn-local-to-shared-gateway-mode-migration
ci/rehearse/openshift/ovn-kubernetes/release-4.13/e2e-aws-ovn-local-to-shared-gateway-mode-migration 4e40324 link unknown /pj-rehearse pull-ci-openshift-ovn-kubernetes-release-4.13-e2e-aws-ovn-local-to-shared-gateway-mode-migration

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@jluhrsen
Copy link
Contributor

/lgtm

@jluhrsen
Copy link
Contributor

/pj-rehearse ack

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Aug 22, 2023
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 22, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 22, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jluhrsen, tssurya

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit 5e7ec83 into openshift:master Aug 22, 2023
17 of 21 checks passed
@openshift-ci-robot
Copy link
Contributor

@tssurya: Jira Issue OCPBUGS-16629: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-16629 has been moved to the MODIFIED state.

In response to this:

/cc @anuragthehatter @jluhrsen @trozet

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

prb112 pushed a commit to prb112/openshift-release that referenced this pull request Sep 12, 2023
Signed-off-by: Surya Seetharaman <suryaseetharaman.9@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged
Projects
None yet
4 participants