Bug 1998643: Revert "bump RHCOS boot images for 4.9"

[ardaguclu]

e2e-metal-ipi-ovn-ipv6 and e2e-metal-ipi-ovn-dualstack jobs are failing in bootstrap stage. Service network endpoint can not be up and pods are crashlooping.

This PR reverts #5168

[openshift-ci]

@ardaguclu: This pull request references Bugzilla bug 1981999, which is invalid:

• expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is VERIFIED instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

WIP: Revert "Bug 1981999: bump RHCOS boot images for 4.9"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ardaguclu]

/test e2e-metal-ipi-ovn-ipv6

[openshift-ci]

@ardaguclu: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

WIP: Revert

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ardaguclu]

/test e2e-metal-ipi-ovn-ipv6

[ardaguclu]

/test e2e-metal-ipi-ovn-ipv6

[ardaguclu]

/retest

[ardaguclu]

/test e2e-metal-ipi-virtualmedia

[openshift-ci]

@ardaguclu: This pull request references Bugzilla bug 1998643, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.9.0) matches configured target release for branch (4.9.0)
• bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (augol@redhat.com), skipping review request.

In response to this:

Bug 1998643: Revert "bump RHCOS boot images for 4.9"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ardaguclu]

/bugzilla refresh

[openshift-ci]

@ardaguclu: This pull request references Bugzilla bug 1998643, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.9.0) matches configured target release for branch (4.9.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (augol@redhat.com), skipping review request.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ardaguclu]

/test e2e-metal-ipi-ovn-ipv6

[ardaguclu]

/test e2e-metal-ipi-ovn-ipv6

[ardaguclu]

/assign @miabbott @jlebon @bgilbert @patrickdillon

[ashcrow]

Note: This will revert multiple fixes. Is there more information on what it crashing? From the original merge:

1967483 - coreos-installer fails to download Ignition (DNS error, failed to lookup address)
1974411 - Installation with multipath parameters in parmfile fails (DNS resolution missing)
1980679 - On a Azure IPI installation MCO fails to create new nodes

[ardaguclu]

@ashcrow @miabbott investigations continue but so far there is no clue yet what is causing this network problem. It can not complete cluster setup within installer timeout, but on local after waiting more(of course getting the same error as CI jobs), it becomes ready.

[miabbott]

I really don't like the idea of reverting the boot image without knowing the underlying root cause, but if we are blocked on shipping in this state, it seems like this is the best path forward.

/approve

[patrickdillon]

/approve

@miabbott I am giving high-level installer approval. I will leave it to you to lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: miabbott, patrickdillon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [patrickdillon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[miabbott]

One thing to note is that we have identified another blocker in RHCOS that we need a new boot image update to fully solve - https://bugzilla.redhat.com/show_bug.cgi?id=1999577

Which means we need to sort out the underlying problem that required us to pursue this revert before we can fully fix BZ#1999577

I'm going to give the /lgtm in hopes that we can unblock other folks from more reliable CI jobs, but we really need to understand the root cause that is requiring us to revert.

cc: @ardaguclu @hardys

/lgtm

[ashcrow]

@ardaguclu @hardys please keep us updated with your debugging. Note that https://bugzilla.redhat.com/show_bug.cgi?id=1999577 is also related to metal installs.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci]

@ardaguclu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-workers-rhel7	f6ece2a	link	/test e2e-aws-workers-rhel7
ci/prow/e2e-aws-workers-rhel8	f6ece2a	link	/test e2e-aws-workers-rhel8
ci/prow/e2e-openstack-provider-network	f6ece2a	link	/test e2e-openstack-provider-network
ci/prow/e2e-crc	f6ece2a	link	/test e2e-crc

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci]

@ardaguclu: All pull requests linked via external trackers have merged:

• openshift/installer#5180

Bugzilla bug 1998643 has been moved to the MODIFIED state.

In response to this:

Bug 1998643: Revert "bump RHCOS boot images for 4.9"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.