CI: ingress-conformance-test-shared: An Ingress with no rules should send all requests to the default backend: expected status code 200 but 503 was returned

[joestringer]

The following test may fail on master branch. Workflow is ingress-conformance-test-shared

At a glance it seems like there is roughly one failure every ~3 days or so, vs. ~10-15 successful runs over the same period.

  Scenario Outline: An Ingress with no rules should send all requests to the default backend # features/default_backend.feature:21
    Given a new random namespace                                                             # feature.go:61 -> aNewRandomNamespace
    Given an Ingress resource named "default-backend" with this spec:                        # feature.go:71 -> anIngressResourceNamedWithThisSpec
      """
      defaultBackend:
        service:
          name: echo-service
          port:
            number: 8080
      """
    Then The Ingress status shows the IP address or FQDN where it is exposed                 # feature.go:92 -> theIngressStatusShowsTheIPAddressOrFQDNWhereItIsExposed
    When I send a "<method>" request to http://"<host>"/"<path>"                             # feature.go:102 -> iSendARequestToHttp
    Then the response status-code must be 200                                                # feature.go:106 -> theResponseStatuscodeMustBe
    And the response must be served by the "echo-service" service                            # feature.go:110 -> theResponseMustBeServedByTheService
    And the response proto must be "HTTP/1.1"                                                # feature.go:114 -> theResponseProtoMustBe
    And the response headers must contain <key> with matching <value>                        # feature.go:118 -> theResponseHeadersMustContainKeyWithMatchingValue
      | key            | value |
      | Content-Length | *     |
      | Content-Type   | *     |
      | Date           | *     |
      | Server         | *     |
    And the request method must be "<method>"                                                # feature.go:122 -> theRequestMethodMustBe
    And the request path must be "<path>"                                                    # feature.go:126 -> theRequestPathMustBe
    And the request proto must be "HTTP/1.1"                                                 # feature.go:130 -> theRequestProtoMustBe
    And the request headers must contain <key> with matching <value>                         # feature.go:134 -> theRequestHeadersMustContainKeyWithMatchingValue
      | key        | value              |
      | User-Agent | Go-http-client/1.1 |

    Examples:
      | method | host      | path     |
      | GET    | my-host   |          |
      | GET    | my-host   | sub-path |
      | POST   | some-host |          |
      | PUT    |           | resource |
        expected status code 200 but 503 was returned
      | DELETE | some-host | resource |
      | PATCH  | my-host   | resource |

--- Failed steps:

  Scenario Outline: An Ingress with no rules should send all requests to the default backend # features/default_backend.feature:21
    Then the response status-code must be 200 # features/default_backend.feature:23
      Error: expected status code 200 but 503 was returned

Similar sort of failure shows up in:

• An Ingress with no rules should send all requests to the default backend
  • https://github.com/cilium/cilium/actions/runs/3377922745/jobs/5607477240
• An Ingress with prefix path rules should send traffic to the matching backend service
  • https://github.com/cilium/cilium/actions/runs/3368492373/jobs/5587097717
  • This one may possibly be different, in which case we can file a fresh issue for it.

[giorio94]

I've just hit this issue in #24979: https://github.com/cilium/cilium/actions/runs/4743823239/jobs/8423871578?pr=24979

    Examples:
      | method | host      | path     |
      | GET    | my-host   |          |
      | GET    | my-host   | sub-path |
      | POST   | some-host |          |
      | PUT    |           | resource |
      | DELETE | some-host | resource |
        expected status code 200 but 503 was returned
      | PATCH  | my-host   | resource |

--- Failed steps:

  Scenario Outline: An Ingress with no rules should send all requests to the default backend # features/default_backend.feature:21
    Then the response status-code must be 200 # features/default_backend.feature:23
      Error: expected status code 200 but 503 was returned

[joestringer]

@giorio94 I autoclosed this due to merging #25244, but if you find that the issue recurs again, feel free to reopen.

[sayboras]

The unintentional revert of the original fix could be the reason for re-occurring. Feel free to ping me if it happens again. Thanks all.