Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MGMT-15926: Operators installation fails due to timeout on failed jobs #748

Merged
merged 1 commit into from Nov 9, 2023
Merged

MGMT-15926: Operators installation fails due to timeout on failed jobs #748

merged 1 commit into from Nov 9, 2023

Conversation

danmanor
Copy link
Contributor

@danmanor danmanor commented Nov 6, 2023 •
edited

Installing operators fails due to failing job, The job are starting and timing out because workers aren't ready.
deleting the failed job to allow retry of operator installation. The solution is removing failed OLM jobs and subscription install plans during operator initialization check not only in the case of notFound error. Linking test installation same as in the origin failure - http://rdu-infra-edge-06.infra-edge.lab.eng.rdu2.redhat.com:6008/clusters/5a46e946-60fe-43ab-8650-5a50c0429bf6

@danmanor
MGMT-15926: Delete failed OLM jobs and subscription install plans in …
a5c999e 
…all cases during initialization check, not only in case of not found error
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 6, 2023
@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 6, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@danmanor
Copy link
Contributor Author

danmanor commented Nov 6, 2023

/test all

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 6, 2023
@openshift-ci-robot
Copy link

openshift-ci-robot commented Nov 6, 2023 •
edited by openshift-ci bot

@danmanor: This pull request references MGMT-15926 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.15.0" version, but no target version was set.

In response to this:

Currently when installing a high availability cluster with MCE operator, MCE fails to install and the cluster will be degraded.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 6, 2023
@codecov Codecov
Copy link

codecov bot commented Nov 6, 2023 •
edited

Codecov Report

Merging #748 (a5c999e) into master (b7c9d2e) will decrease coverage by 0.02%.
Report is 2 commits behind head on master.
The diff coverage is 50.00%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #748      +/-   ##
==========================================
- Coverage   56.04%   56.02%   -0.02%     
==========================================
  Files          15       15              
  Lines        3203     3202       -1     
==========================================
- Hits         1795     1794       -1     
  Misses       1246     1246              
  Partials      162      162
Files Coverage Δ
.../assisted_installer_controller/operator_handler.go 78.52% <50.00%> (-0.15%) ⬇️

@openshift-ci-robot
Copy link

openshift-ci-robot commented Nov 6, 2023 •
edited by openshift-ci bot

@danmanor: This pull request references MGMT-15926 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.15.0" version, but no target version was set.

In response to this:

Currently when installing a high availability cluster with MCE operator, MCE fails to install and the cluster will be degraded. This PR fixing this bug.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@danmanor danmanor marked this pull request as ready for review November 6, 2023 09:28
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 6, 2023
@danmanor
Copy link
Contributor Author

danmanor commented Nov 6, 2023

/cc @avishayt @eifrach

@danmanor
Copy link
Contributor Author

danmanor commented Nov 6, 2023

/uncc @paul-maidment @eranco74

@danmanor
Copy link
Contributor Author

danmanor commented Nov 6, 2023

/cc @tsorya

@openshift-ci openshift-ci bot requested a review from tsorya November 6, 2023 09:30
@danmanor
Copy link
Contributor Author

danmanor commented Nov 6, 2023

@eifrach @avishayt @tsorya Any description suggestions will be appreciated

@danmanor danmanor changed the title MGMT-15926: Delete failed OLM jobs and subscription install plans in all cases during initialization check, not only in case of not found error MGMT-15926: Operators installation fails due to timeout on failed jobs Nov 6, 2023
@openshift-ci-robot
Copy link

openshift-ci-robot commented Nov 6, 2023 •
edited by openshift-ci bot

@danmanor: This pull request references MGMT-15926 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.15.0" version, but no target version was set.

In response to this:

installing operators fails do to failing job, The job are starting and timing out because workers aren't ready.
deleting the failed job to allow retry of operator installation.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Nov 6, 2023 •
edited by openshift-ci bot

@danmanor: This pull request references MGMT-15926 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.15.0" version, but no target version was set.

In response to this:

Installing operators fails do to failing job, The job are starting and timing out because workers aren't ready.
deleting the failed job to allow retry of operator installation.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@avishayt
Copy link
Contributor

avishayt commented Nov 6, 2023

How did you test?

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 6, 2023

@danmanor: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

  • /test e2e-agent-compact-ipv4
  • /test edge-e2e-ai-operator-ztp
  • /test edge-e2e-metal-assisted
  • /test edge-format-check
  • /test edge-images
  • /test edge-lint
  • /test edge-unit-test
  • /test images
  • /test mce-images

The following commands are available to trigger optional jobs:

  • /test e2e-agent-ha-dualstack
  • /test e2e-agent-sno-ipv6
  • /test edge-e2e-metal-assisted-cnv
  • /test edge-e2e-metal-assisted-ipv6
  • /test edge-e2e-metal-assisted-lvm
  • /test edge-e2e-metal-assisted-odf
  • /test edge-e2e-metal-assisted-single-node
  • /test edge-e2e-oci-assisted

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-assisted-installer-master-e2e-agent-compact-ipv4
  • pull-ci-openshift-assisted-installer-master-edge-e2e-ai-operator-ztp
  • pull-ci-openshift-assisted-installer-master-edge-e2e-metal-assisted
  • pull-ci-openshift-assisted-installer-master-edge-format-check
  • pull-ci-openshift-assisted-installer-master-edge-images
  • pull-ci-openshift-assisted-installer-master-edge-lint
  • pull-ci-openshift-assisted-installer-master-edge-unit-test
  • pull-ci-openshift-assisted-installer-master-images
  • pull-ci-openshift-assisted-installer-master-mce-images

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 6, 2023

@danmanor: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test e2e-agent-compact-ipv4
  • /test edge-e2e-ai-operator-ztp
  • /test edge-e2e-metal-assisted
  • /test edge-format-check
  • /test edge-images
  • /test edge-lint
  • /test edge-unit-test
  • /test images
  • /test mce-images

The following commands are available to trigger optional jobs:

  • /test e2e-agent-ha-dualstack
  • /test e2e-agent-sno-ipv6
  • /test edge-e2e-metal-assisted-cnv
  • /test edge-e2e-metal-assisted-ipv6
  • /test edge-e2e-metal-assisted-lvm
  • /test edge-e2e-metal-assisted-odf
  • /test edge-e2e-metal-assisted-single-node
  • /test edge-e2e-oci-assisted

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-assisted-installer-master-e2e-agent-compact-ipv4
  • pull-ci-openshift-assisted-installer-master-edge-e2e-ai-operator-ztp
  • pull-ci-openshift-assisted-installer-master-edge-e2e-metal-assisted
  • pull-ci-openshift-assisted-installer-master-edge-format-check
  • pull-ci-openshift-assisted-installer-master-edge-images
  • pull-ci-openshift-assisted-installer-master-edge-lint
  • pull-ci-openshift-assisted-installer-master-edge-unit-test
  • pull-ci-openshift-assisted-installer-master-images
  • pull-ci-openshift-assisted-installer-master-mce-images

In response to this:

/test e2e-metal-assisted-mce

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@danmanor
Copy link
Contributor Author

danmanor commented Nov 6, 2023

/test edge-e2e-metal-assisted-mce

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 6, 2023

@danmanor: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test e2e-agent-compact-ipv4
  • /test edge-e2e-ai-operator-ztp
  • /test edge-e2e-metal-assisted
  • /test edge-format-check
  • /test edge-images
  • /test edge-lint
  • /test edge-unit-test
  • /test images
  • /test mce-images

The following commands are available to trigger optional jobs:

  • /test e2e-agent-ha-dualstack
  • /test e2e-agent-sno-ipv6
  • /test edge-e2e-metal-assisted-cnv
  • /test edge-e2e-metal-assisted-ipv6
  • /test edge-e2e-metal-assisted-lvm
  • /test edge-e2e-metal-assisted-odf
  • /test edge-e2e-metal-assisted-single-node
  • /test edge-e2e-oci-assisted

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-assisted-installer-master-e2e-agent-compact-ipv4
  • pull-ci-openshift-assisted-installer-master-edge-e2e-ai-operator-ztp
  • pull-ci-openshift-assisted-installer-master-edge-e2e-metal-assisted
  • pull-ci-openshift-assisted-installer-master-edge-format-check
  • pull-ci-openshift-assisted-installer-master-edge-images
  • pull-ci-openshift-assisted-installer-master-edge-lint
  • pull-ci-openshift-assisted-installer-master-edge-unit-test
  • pull-ci-openshift-assisted-installer-master-images
  • pull-ci-openshift-assisted-installer-master-mce-images

In response to this:

/test edge-e2e-metal-assisted-mce

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@danmanor
Copy link
Contributor Author

danmanor commented Nov 8, 2023

/test e2e-agent-compact-ipv4

@openshift-ci-robot
Copy link

openshift-ci-robot commented Nov 8, 2023 •
edited by openshift-ci bot

@danmanor: This pull request references MGMT-15926 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.15.0" version, but no target version was set.

In response to this:

Installing operators fails do to failing job, The job are starting and timing out because workers aren't ready.
deleting the failed job to allow retry of operator installation. The solution is removing failed OLM jobs and subscription install plans during operator initialization check not only in the case of notFound error. Linking test installation same as in the origin failure - http://rdu-infra-edge-06.infra-edge.lab.eng.rdu2.redhat.com:6008/clusters/5a46e946-60fe-43ab-8650-5a50c0429bf6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Nov 8, 2023 •
edited by openshift-ci bot

@danmanor: This pull request references MGMT-15926 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.15.0" version, but no target version was set.

In response to this:

Installing operators fails do to failing job, The job are starting and timing out because workers aren't ready.
deleting the failed job to allow retry of operator installation. The solution is removing failed OLM jobs and subscription install plans during operator initialization check not only in the case of notFound error. Linking test installation same as in the origin failure - http://rdu-infra-edge-06.infra-edge.lab.eng.rdu2.redhat.com:6008/clusters/5a46e946-60fe-43ab-8650-5a50c0429bf6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@danmanor danmanor marked this pull request as ready for review November 8, 2023 18:45
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 8, 2023
@danmanor
Copy link
Contributor Author

danmanor commented Nov 8, 2023

/cc @avishayt @tsorya @eifrach

@danmanor
Copy link
Contributor Author

danmanor commented Nov 8, 2023

/uncc @eliorerz

@openshift-ci openshift-ci bot removed the request for review from eliorerz November 8, 2023 18:46
@danmanor
Copy link
Contributor Author

danmanor commented Nov 8, 2023

\unhold

@openshift-ci-robot
Copy link

openshift-ci-robot commented Nov 8, 2023 •
edited by openshift-ci bot

@danmanor: This pull request references MGMT-15926 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.15.0" version, but no target version was set.

In response to this:

Installing operators fails due to failing job, The job are starting and timing out because workers aren't ready.
deleting the failed job to allow retry of operator installation. The solution is removing failed OLM jobs and subscription install plans during operator initialization check not only in the case of notFound error. Linking test installation same as in the origin failure - http://rdu-infra-edge-06.infra-edge.lab.eng.rdu2.redhat.com:6008/clusters/5a46e946-60fe-43ab-8650-5a50c0429bf6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@danmanor
Copy link
Contributor Author

danmanor commented Nov 9, 2023

/cc @gamli75

@openshift-ci openshift-ci bot requested a review from gamli75 November 9, 2023 07:16
@eifrach
Copy link
Contributor

eifrach commented Nov 9, 2023

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 9, 2023
@danmanor
Copy link
Contributor Author

danmanor commented Nov 9, 2023

/unhold

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 9, 2023
@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 9, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danmanor, eifrach, gamli75

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 9, 2023
@gamli75
Copy link
Contributor

gamli75 commented Nov 9, 2023

/refresh

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 9, 2023

@danmanor: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-bot openshift-merge-bot bot merged commit ccc8e3e into openshift:master Nov 9, 2023
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gamli75 gamli75 gamli75 approved these changes

@eifrach eifrach eifrach approved these changes

@romfreiman romfreiman Awaiting requested review from romfreiman

@avishayt avishayt Awaiting requested review from avishayt

@tsorya tsorya Awaiting requested review from tsorya

Assignees

@gamli75 gamli75

@eifrach eifrach

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@danmanor @openshift-ci-robot @avishayt @eifrach @gamli75