[WIP] [release-4.14] OCPBUGS-XXXXX: UPSTREAM: 6277: openshift: Fix OCPBUGS-XXXXX (CI Broken)

[gcs278]

4.14 Cherry-Pick of coredns#6277 which fixes OCPBUGS-28200 by adding logic that more gracefully handles overflow errors by setting the Truncated Bit and clearing the response. This indicates to DNS clients to retry with TCP, whereas previously they would have always gotten a SERVFAIL.

This becomes more important/exposed with CoreDNS 1.10.1 (OCP 4.13) onwards, given the changes introduced by coredns#5671. This modification only uses EDNS on the upstream DNS request when the client query used EDNS. It appears that certain DNS resolvers may not handle non-EDNS queries in a compliant manner.

Cherry-Pick Merge Conflict:

CONFLICT (modify/delete): plugin/pkg/proxy/proxy_test.go deleted in HEAD and modified in 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397).  Version 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397) of plugin/pkg/proxy/proxy_test.go left in tree.

• plugin/pkg/proxy directory doesn't exist in release-4.14 due to refactoring, so I placed the unit tests in plugin/forward/proxy_test.go
• Required some refactoring due to function signature changes and other refactoring

[openshift-ci-robot]

@gcs278: This pull request references Jira Issue OCPBUGS-28200, which is invalid:

• expected the bug to target either version "4.16." or "openshift-4.16.", but it targets "4.14.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

4.14 Cherry-Pick of coredns#6277 which fixes OCPBUGS-28200 by adding logic that more gracefully handles overflow errors by setting the Truncated Bit and clearing the response. This indicates to DNS clients to retry with TCP, whereas previously they would have always gotten a SERVFAIL.

This becomes more important/exposed with CoreDNS 1.10.1 (OCP 4.13) onwards, given the changes introduced by coredns#5671. This modification only uses EDNS on the upstream DNS request when the client query used EDNS. It appears that certain DNS resolvers may not handle non-EDNS queries in a compliant manner.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@gcs278: This pull request references Jira Issue OCPBUGS-28200, which is invalid:

• expected dependent Jira Issue OCPBUGS-27904 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is MODIFIED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

4.14 Cherry-Pick of coredns#6277 which fixes OCPBUGS-28200 by adding logic that more gracefully handles overflow errors by setting the Truncated Bit and clearing the response. This indicates to DNS clients to retry with TCP, whereas previously they would have always gotten a SERVFAIL.

This becomes more important/exposed with CoreDNS 1.10.1 (OCP 4.13) onwards, given the changes introduced by coredns#5671. This modification only uses EDNS on the upstream DNS request when the client query used EDNS. It appears that certain DNS resolvers may not handle non-EDNS queries in a compliant manner.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[gcs278]

/jira refresh

[openshift-ci-robot]

@gcs278: This pull request references Jira Issue OCPBUGS-28200, which is invalid:

• expected dependent Jira Issue OCPBUGS-27904 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is MODIFIED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[Miciah]

Thanks!
/approve
/lgtm

Adding the branch to the PR title...
/retitle [release-4.14] OCPBUGS-28200: UPSTREAM: 6277: openshift: Fix OCPBUGS-28200

As with the original change, this change only affects the behavior when we already have an error and the upstream resolver is probably misbehaving.
/label backport-risk-assessed

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Miciah

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Miciah]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@gcs278: This pull request references Jira Issue OCPBUGS-28200, which is invalid:

• expected dependent Jira Issue OCPBUGS-27904 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is MODIFIED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

4.14 Cherry-Pick of coredns#6277 which fixes OCPBUGS-28200 by adding logic that more gracefully handles overflow errors by setting the Truncated Bit and clearing the response. This indicates to DNS clients to retry with TCP, whereas previously they would have always gotten a SERVFAIL.

This becomes more important/exposed with CoreDNS 1.10.1 (OCP 4.13) onwards, given the changes introduced by coredns#5671. This modification only uses EDNS on the upstream DNS request when the client query used EDNS. It appears that certain DNS resolvers may not handle non-EDNS queries in a compliant manner.

Cherry-Pick Merge Conflict:

CONFLICT (modify/delete): plugin/pkg/proxy/proxy_test.go deleted in HEAD and modified in 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397).  Version 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397) of plugin/pkg/proxy/proxy_test.go left in tree.

• plugin/pkg/proxy directory doesn't exist in release-4.14 due to refactoring, so I placed the unit tests in plugin/forward/proxy_test.go
• Required some refactoring due to function signature changes and other refactoring

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[melvinjoseph86]

/label cherry-pick-approved

[sreber84]

/retest-required

[gcs278]

event happened 21 times, something is wrong: ns/openshift-monitoring statefulset/alertmanager-main hmsg/f8ed7dbffe - pathological/true reason/RecreatingTerminatedPod StatefulSet openshift-monitoring/alertmanager-main is recreating terminated Pod alertmanager-main-1 From: 07:33:54Z To: 07:33:55Z result=reject }
/test e2e-aws-ovn-upgrade: Not related.

[openshift-ci]

@gcs278: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

• /test e2e-aws-ovn
• /test e2e-aws-ovn-upgrade
• /test e2e-gcp-sdn-serial
• /test images
• /test unit

The following commands are available to trigger optional jobs:

• /test e2e-metal-ipi-ovn
• /test e2e-metal-ipi-ovn-dualstack

Use /test all to run the following jobs that were automatically triggered:

• pull-ci-openshift-coredns-release-4.14-e2e-aws-ovn
• pull-ci-openshift-coredns-release-4.14-e2e-aws-ovn-upgrade
• pull-ci-openshift-coredns-release-4.14-e2e-gcp-sdn-serial
• pull-ci-openshift-coredns-release-4.14-e2e-metal-ipi-ovn
• pull-ci-openshift-coredns-release-4.14-images
• pull-ci-openshift-coredns-release-4.14-unit

In response to this:

event happened 21 times, something is wrong: ns/openshift-monitoring statefulset/alertmanager-main hmsg/f8ed7dbffe - pathological/true reason/RecreatingTerminatedPod StatefulSet openshift-monitoring/alertmanager-main is recreating terminated Pod alertmanager-main-1 From: 07:33:54Z To: 07:33:55Z result=reject }
/test e2e-aws-ovn-upgrade: Not related.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gcs278]

/test e2e-aws-ovn-upgrade

[gcs278]

Different error this time:
ns/openshift-authentication route/oauth-openshift disruption/ingress-to-oauth-server connection/new should be available throughout the test
I see some related discussion about this.
/test e2e-aws-ovn-upgrade

[melvinjoseph86]

/retest-required

[openshift-ci-robot]

@gcs278: This pull request references Jira Issue OCPBUGS-28200, which is valid.

6 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.14.z) matches configured target version for branch (4.14.z)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• dependent bug Jira Issue OCPBUGS-27904 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-27904 targets the "4.15.0" version, which is one of the valid target versions: 4.15.0
• bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

In response to this:

4.14 Cherry-Pick of coredns#6277 which fixes OCPBUGS-28200 by adding logic that more gracefully handles overflow errors by setting the Truncated Bit and clearing the response. This indicates to DNS clients to retry with TCP, whereas previously they would have always gotten a SERVFAIL.

This becomes more important/exposed with CoreDNS 1.10.1 (OCP 4.13) onwards, given the changes introduced by coredns#5671. This modification only uses EDNS on the upstream DNS request when the client query used EDNS. It appears that certain DNS resolvers may not handle non-EDNS queries in a compliant manner.

Cherry-Pick Merge Conflict:

CONFLICT (modify/delete): plugin/pkg/proxy/proxy_test.go deleted in HEAD and modified in 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397).  Version 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397) of plugin/pkg/proxy/proxy_test.go left in tree.

• plugin/pkg/proxy directory doesn't exist in release-4.14 due to refactoring, so I placed the unit tests in plugin/forward/proxy_test.go
• Required some refactoring due to function signature changes and other refactoring

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@openshift-ci-robot: GitHub didn't allow me to request PR reviews from the following users: melvinjoseph86.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@gcs278: This pull request references Jira Issue OCPBUGS-28200, which is valid.

6 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.14.z) matches configured target version for branch (4.14.z)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• dependent bug Jira Issue OCPBUGS-27904 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-27904 targets the "4.15.0" version, which is one of the valid target versions: 4.15.0
• bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

In response to this:

4.14 Cherry-Pick of coredns#6277 which fixes OCPBUGS-28200 by adding logic that more gracefully handles overflow errors by setting the Truncated Bit and clearing the response. This indicates to DNS clients to retry with TCP, whereas previously they would have always gotten a SERVFAIL.

This becomes more important/exposed with CoreDNS 1.10.1 (OCP 4.13) onwards, given the changes introduced by coredns#5671. This modification only uses EDNS on the upstream DNS request when the client query used EDNS. It appears that certain DNS resolvers may not handle non-EDNS queries in a compliant manner.

Cherry-Pick Merge Conflict:

CONFLICT (modify/delete): plugin/pkg/proxy/proxy_test.go deleted in HEAD and modified in 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397).  Version 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397) of plugin/pkg/proxy/proxy_test.go left in tree.

• plugin/pkg/proxy directory doesn't exist in release-4.14 due to refactoring, so I placed the unit tests in plugin/forward/proxy_test.go
• Required some refactoring due to function signature changes and other refactoring

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gcs278]

/retest

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 8341fa7 and 2 for PR HEAD 7a0f883 in total

[gcs278]

install failed
/test e2e-aws-ovn-upgrdae

[openshift-ci]

@gcs278: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

• /test e2e-aws-ovn
• /test e2e-aws-ovn-upgrade
• /test e2e-gcp-sdn-serial
• /test images
• /test unit

The following commands are available to trigger optional jobs:

• /test e2e-metal-ipi-ovn
• /test e2e-metal-ipi-ovn-dualstack

Use /test all to run the following jobs that were automatically triggered:

• pull-ci-openshift-coredns-release-4.14-e2e-aws-ovn
• pull-ci-openshift-coredns-release-4.14-e2e-aws-ovn-upgrade
• pull-ci-openshift-coredns-release-4.14-e2e-gcp-sdn-serial
• pull-ci-openshift-coredns-release-4.14-e2e-metal-ipi-ovn
• pull-ci-openshift-coredns-release-4.14-images
• pull-ci-openshift-coredns-release-4.14-unit

In response to this:

install failed
/test e2e-aws-ovn-upgrdae

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gcs278]

/test e2e-aws-ovn-upgrade

[Miciah]

/refresh

[Miciah]

/tide refresh

[openshift-ci-robot]

@gcs278: This pull request references Jira Issue OCPBUGS-28200. The bug has been updated to no longer refer to the pull request using the external bug tracker. All external bug links have been closed. The bug has been moved to the NEW state.

In response to this:

4.14 Cherry-Pick of coredns#6277 which fixes OCPBUGS-28200 by adding logic that more gracefully handles overflow errors by setting the Truncated Bit and clearing the response. This indicates to DNS clients to retry with TCP, whereas previously they would have always gotten a SERVFAIL.

This becomes more important/exposed with CoreDNS 1.10.1 (OCP 4.13) onwards, given the changes introduced by coredns#5671. This modification only uses EDNS on the upstream DNS request when the client query used EDNS. It appears that certain DNS resolvers may not handle non-EDNS queries in a compliant manner.

Cherry-Pick Merge Conflict:

CONFLICT (modify/delete): plugin/pkg/proxy/proxy_test.go deleted in HEAD and modified in 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397).  Version 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397) of plugin/pkg/proxy/proxy_test.go left in tree.

• plugin/pkg/proxy directory doesn't exist in release-4.14 due to refactoring, so I placed the unit tests in plugin/forward/proxy_test.go
• Required some refactoring due to function signature changes and other refactoring

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@gcs278: This pull request references Jira Issue OCPBUGS-28200, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.14.z) matches configured target version for branch (4.14.z)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
• dependent bug Jira Issue OCPBUGS-27904 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-27904 targets the "4.15.0" version, which is one of the valid target versions: 4.15.0
• bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

4.14 Cherry-Pick of coredns#6277 which fixes OCPBUGS-28200 by adding logic that more gracefully handles overflow errors by setting the Truncated Bit and clearing the response. This indicates to DNS clients to retry with TCP, whereas previously they would have always gotten a SERVFAIL.

This becomes more important/exposed with CoreDNS 1.10.1 (OCP 4.13) onwards, given the changes introduced by coredns#5671. This modification only uses EDNS on the upstream DNS request when the client query used EDNS. It appears that certain DNS resolvers may not handle non-EDNS queries in a compliant manner.

Cherry-Pick Merge Conflict:

CONFLICT (modify/delete): plugin/pkg/proxy/proxy_test.go deleted in HEAD and modified in 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397).  Version 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397) of plugin/pkg/proxy/proxy_test.go left in tree.

• plugin/pkg/proxy directory doesn't exist in release-4.14 due to refactoring, so I placed the unit tests in plugin/forward/proxy_test.go
• Required some refactoring due to function signature changes and other refactoring

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@openshift-ci-robot: GitHub didn't allow me to request PR reviews from the following users: melvinjoseph86.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@gcs278: This pull request references Jira Issue OCPBUGS-28200, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.14.z) matches configured target version for branch (4.14.z)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
• dependent bug Jira Issue OCPBUGS-27904 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-27904 targets the "4.15.0" version, which is one of the valid target versions: 4.15.0
• bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

4.14 Cherry-Pick of coredns#6277 which fixes OCPBUGS-28200 by adding logic that more gracefully handles overflow errors by setting the Truncated Bit and clearing the response. This indicates to DNS clients to retry with TCP, whereas previously they would have always gotten a SERVFAIL.

This becomes more important/exposed with CoreDNS 1.10.1 (OCP 4.13) onwards, given the changes introduced by coredns#5671. This modification only uses EDNS on the upstream DNS request when the client query used EDNS. It appears that certain DNS resolvers may not handle non-EDNS queries in a compliant manner.

Cherry-Pick Merge Conflict:

CONFLICT (modify/delete): plugin/pkg/proxy/proxy_test.go deleted in HEAD and modified in 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397).  Version 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397) of plugin/pkg/proxy/proxy_test.go left in tree.

• plugin/pkg/proxy directory doesn't exist in release-4.14 due to refactoring, so I placed the unit tests in plugin/forward/proxy_test.go
• Required some refactoring due to function signature changes and other refactoring

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Miciah]

/test e2e-aws-ovn-upgrade

[gcs278]

/test e2e-gcp-sdn-serial

[openshift-ci]

@gcs278: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/images	7a0f883	link	true	/test images
ci/prow/e2e-aws-ovn-upgrade	7a0f883	link	true	/test e2e-aws-ovn-upgrade
ci/prow/e2e-metal-ipi-ovn	7a0f883	link	false	/test e2e-metal-ipi-ovn
ci/prow/unit	7a0f883	link	true	/test unit
ci/prow/e2e-metal-ipi-ovn	7a0f883	link	false	/test e2e-metal-ipi-ovn
ci/prow/e2e-gcp-serial	7a0f883	link	true	/test e2e-gcp-serial
ci/prow/e2e-aws-ovn	7a0f883	link	true	/test e2e-aws-ovn

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[gcs278]

Switching to #114 as something seems wrong with CI. Now, the tide link is referencing #114.
I'll leave this open for a bit in case anyone is curious what is happening.
/hold

[openshift-ci-robot]

@gcs278: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

In response to this:

4.14 Cherry-Pick of coredns#6277 which fixes OCPBUGS-28200 by adding logic that more gracefully handles overflow errors by setting the Truncated Bit and clearing the response. This indicates to DNS clients to retry with TCP, whereas previously they would have always gotten a SERVFAIL.

This becomes more important/exposed with CoreDNS 1.10.1 (OCP 4.13) onwards, given the changes introduced by coredns#5671. This modification only uses EDNS on the upstream DNS request when the client query used EDNS. It appears that certain DNS resolvers may not handle non-EDNS queries in a compliant manner.

Cherry-Pick Merge Conflict:

CONFLICT (modify/delete): plugin/pkg/proxy/proxy_test.go deleted in HEAD and modified in 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397).  Version 37a9afe69 (UPSTREAM: 6277: openshift: Fix OCPBUGS-27397) of plugin/pkg/proxy/proxy_test.go left in tree.

• plugin/pkg/proxy directory doesn't exist in release-4.14 due to refactoring, so I placed the unit tests in plugin/forward/proxy_test.go
• Required some refactoring due to function signature changes and other refactoring

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[gcs278]

/unassign @Miciah

[gcs278]

Note: This PR wasn't explicitly merged, but somehow is showing that it merged because a PR with identical commits merged: #114