Bug 1986575: Add e2e test cases for haproxy timeout api fields, and reject negative timeout values #644

rfredette · 2021-08-17T18:26:08Z

This PR adds 2 tests to the e2e suite:

TestHAProxyTimeouts:

Set timeouts for all 6 new timeout fields, and verify that all timeout values are reflected in the router deployment's environment.
Most of the values are set to durations that, if converted directly from Duration to string, would be invalid strings to HAProxy. E.g. 90 * time.Second would string-ify as "1m30s", but HAProxy requires it to only use one time unit, such as "90s". This makes sure the durationToHAProxyTimespec function is working as intended.
One timeout value is set higher than the maximum HAProxy will allow, to make sure that the timeout is clipped to HAProxy's max.

TestHAProxyTimeoutsRejection:

Sets timeouts for all 6 new timeout fields that are allowed by the API, but are not actually valid timeout periods, like 0 or negative values. Verify that none of the environment variables are set, forcing the router pods to use the default timeouts.

In addition to the tests, this PR also includes a change to reject timeout values that are zero or less, instead of just rejecting zero-length timeouts. The router image will fall back to using default timeouts if negative ones are set, but rejecting them before they are set in the deployment environment is more obvious to the user.

openshift-ci · 2021-08-17T18:27:37Z

@rfredette: This pull request references Bugzilla bug 1986575, which is invalid:

expected the bug to target the "4.9.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1986575: Add e2e test cases for haproxy timeout api fields, and reject negative timeout values

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rfredette · 2021-08-17T18:56:58Z

/bugzilla refresh

openshift-ci · 2021-08-17T19:00:11Z

@rfredette: An error was encountered querying GitHub for users with public email (hongli@redhat.com) for bug 1986575 on the Bugzilla server at https://bugzilla.redhat.com. No known errors were detected, please see the full error message for details.

Full error message.


non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again.\"\n}\n"

Please contact an administrator to resolve this issue, then request a bug refresh with /bugzilla refresh.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rfredette · 2021-08-17T19:04:25Z

/bugzilla refresh

openshift-ci · 2021-08-17T19:05:41Z

@rfredette: This pull request references Bugzilla bug 1986575, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @lihongan

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

candita · 2021-08-17T21:55:28Z

pkg/operator/controller/ingress/deployment.go

@@ -521,22 +521,22 @@ func desiredRouterDeployment(ci *operatorv1.IngressController, ingressController
 	}
 	env = append(env, corev1.EnvVar{Name: RouterHAProxyThreadsEnvName, Value: strconv.Itoa(threads)})

-	if ci.Spec.TuningOptions.ClientTimeout != nil && ci.Spec.TuningOptions.ClientTimeout.Duration != 0*time.Second {
+	if ci.Spec.TuningOptions.ClientTimeout != nil && ci.Spec.TuningOptions.ClientTimeout.Duration > 0*time.Second {


What if Spec.TuningOptions is nil? Is that possible?

What if Spec.TuningOptions is nil? Is that possible?

It isn't possible. Spec.TuningOptions is not a pointer type.

candita · 2021-08-17T22:09:31Z

test/e2e/haproxy_timeouts_test.go

+				t.Errorf("expected %s = %q, got %q", envVar.Name, tlsInspectDelayOutput, envVar.Value)
+			}
+		}
+	}


It would be best to actually see these show up in the haproxy.config file with the values that you specified. You would need to scan a router pod's haproxy.config for the timeout values that should be there.

Good point. I will add checks for the values in haproxy's config.

Miciah · 2021-08-23T17:47:02Z

test/e2e/haproxy_timeouts_test.go

+	conditions := []operatorv1.OperatorCondition{
+		{Type: operatorv1.IngressControllerAvailableConditionType, Status: operatorv1.ConditionTrue},
+		{Type: operatorv1.LoadBalancerManagedIngressConditionType, Status: operatorv1.ConditionFalse},
+		{Type: operatorv1.DNSManagedIngressConditionType, Status: operatorv1.ConditionFalse},
+	}


You can use IngressControllerAvailableConditionType.

Miciah · 2021-08-23T17:51:25Z

test/e2e/haproxy_timeouts_test.go

+		case "ROUTER_DEFAULT_CLIENT_TIMEOUT":
+			fallthrough
+		case "ROUTER_CLIENT_FIN_TIMEOUT":
+			fallthrough
+		case "ROUTER_DEFAULT_SERVER_TIMEOUT":
+			fallthrough
+		case "ROUTER_DEFAULT_SERVER_FIN_TIMEOUT":
+			fallthrough
+		case "ROUTER_DEFAULT_TUNNEL_TIMEOUT":
+			fallthrough
+		case "ROUTER_INSPECT_DELAY":


Suggested change

case "ROUTER_DEFAULT_CLIENT_TIMEOUT":

fallthrough

case "ROUTER_CLIENT_FIN_TIMEOUT":

fallthrough

case "ROUTER_DEFAULT_SERVER_TIMEOUT":

fallthrough

case "ROUTER_DEFAULT_SERVER_FIN_TIMEOUT":

fallthrough

case "ROUTER_DEFAULT_TUNNEL_TIMEOUT":

fallthrough

case "ROUTER_INSPECT_DELAY":

case "ROUTER_DEFAULT_CLIENT_TIMEOUT", "ROUTER_CLIENT_FIN_TIMEOUT", "ROUTER_DEFAULT_SERVER_TIMEOUT", "ROUTER_DEFAULT_SERVER_FIN_TIMEOUT", "ROUTER_DEFAULT_TUNNEL_TIMEOUT", "ROUTER_INSPECT_DELAY":

Edit: Ignore this suggestion if you're replacing this logic with a check for the settings in haproxy.config.

Miciah

There are a few spots we can tighten up the code or stop the test early where later steps depend on an earlier step that has failed. Otherwise, looks good. I look forward to using podExec in other E2E tests!

Miciah · 2021-08-25T17:19:40Z

test/e2e/haproxy_timeouts_test.go

+		return false, nil
+	})
+	if pollErr != nil {
+		t.Errorf("Router pod %s failed to become ready: %v", routerPod.Name, pollErr)


Suggested change

t.Errorf("Router pod %s failed to become ready: %v", routerPod.Name, pollErr)

t.Fatalf("Router pod %s failed to become ready: %v", routerPod.Name, pollErr)

Miciah · 2021-08-25T17:20:22Z

test/e2e/haproxy_timeouts_test.go

+			t.Errorf("Error executing %s: %v", strings.Join(cmd, " "), err)
+			t.Errorf("stderr: %v", stderr)


Suggested change

t.Errorf("Error executing %s: %v", strings.Join(cmd, " "), err)

t.Errorf("stderr: %v", stderr)

t.Errorf("Error executing %s: %v", strings.Join(cmd, " "), err)

t.Errorf("stderr: %v", stderr)

continue

Miciah · 2021-08-25T17:22:30Z

test/e2e/haproxy_timeouts_test.go

+	}
+	values := strings.Split(strings.TrimSpace(stdout.String()), "\n")
+	// tcp-request inspect-delay is set in 2 places, but both should match
+	if len(values) != 2 {
+		t.Errorf("Expected 2 instances of \"tcp-request inspect-delay\", got %v", len(values))
+	}
+	inspectDelayDefault := "5s"
+	if strings.TrimSpace(values[0]) != inspectDelayDefault ||
+		strings.TrimSpace(values[1]) != inspectDelayDefault {
+		t.Errorf("Expected value for \"tcp-request inspect-delay\" to be %v, got %v", []string{"5s", "5s"}, values)
+	}


Suggested change

}

values := strings.Split(strings.TrimSpace(stdout.String()), "\n")

// tcp-request inspect-delay is set in 2 places, but both should match

if len(values) != 2 {

t.Errorf("Expected 2 instances of \"tcp-request inspect-delay\", got %v", len(values))

}

inspectDelayDefault := "5s"

if strings.TrimSpace(values[0]) != inspectDelayDefault ||

strings.TrimSpace(values[1]) != inspectDelayDefault {

t.Errorf("Expected value for \"tcp-request inspect-delay\" to be %v, got %v", []string{"5s", "5s"}, values)

}

} else {

values := strings.Split(strings.TrimSpace(stdout.String()), "\n")

// tcp-request inspect-delay is set in 2 places, but both should match

if len(values) != 2 {

t.Errorf("Expected 2 instances of \"tcp-request inspect-delay\", got %v", len(values))

} else {

inspectDelayDefault := "5s"

if strings.TrimSpace(values[0]) != inspectDelayDefault ||

strings.TrimSpace(values[1]) != inspectDelayDefault {

t.Errorf(`Expected value for "tcp-request inspect-delay" to be %v, got %v`, []string{"5s", "5s"}, values)

}

}

}

Miciah · 2021-08-25T17:22:56Z

test/e2e/util.go

+	err = exec.Stream(remotecommand.StreamOptions{
+		Stdout: stdout,
+		Stderr: stderr,
+	})
+	if err != nil {
+		return err
+	}
+	return nil


Suggested change

err = exec.Stream(remotecommand.StreamOptions{

Stdout: stdout,

Stderr: stderr,

})

if err != nil {

return err

}

return nil

return exec.Stream(remotecommand.StreamOptions{

Stdout: stdout,

Stderr: stderr,

})

Miciah

Sorry, just noticed a couple more small things that should be addressed.

Miciah · 2021-08-25T19:08:11Z

test/e2e/haproxy_timeouts_test.go

+	}
+	pollErr := wait.PollImmediate(2*time.Second, 5*time.Minute, func() (bool, error) {
+		if err := kclient.List(context.TODO(), podList, client.InNamespace(deployment.Namespace), client.MatchingLabels(labels)); err != nil {
+			t.Errorf("failed to list pods for ingress controllers %s: %v", ic.Name, err)


Using t.Errorf isn't quite the right thing to do here because it won't terminate the polling loop, but it will cause the test to fail even if the polling loop succeeds on the next try. Better to log and return to retry immediately:

Suggested change

t.Errorf("failed to list pods for ingress controllers %s: %v", ic.Name, err)

t.Logf("failed to list pods for ingress controllers %s: %v", ic.Name, err)

return false, nil

Miciah · 2021-08-25T19:08:54Z

test/e2e/haproxy_timeouts_test.go

+			t.Errorf("failed to list pods for ingress controllers %s: %v", ic.Name, err)
+		}
+
+		routerPod = podList.Items[0]


The List could have successfully listed 0 pods:

Suggested change

routerPod = podList.Items[0]

if len(podList.Items) == 0 {

t.Logf("failed to find any pods for ingress controllers %s", ic.Name)

return false, nil

}

routerPod = podList.Items[0]

Miciah · 2021-08-25T19:10:28Z

test/e2e/haproxy_timeouts_test.go

+			t.Errorf("failed to list pods for ingress controllers %s: %v", ic.Name, err)
+		}
+
+		routerPod = podList.Items[0]


My previous two comments apply here as well.

…gative timeout values HAProxy won't accept timeouts with negative values, so if one makes it into the ingresscontroller config, discard it and use the default value

rfredette · 2021-08-25T20:09:54Z

Thanks @Miciah! I've incorporated your suggested changes.

Miciah · 2021-08-25T20:50:41Z

/lgtm
No remaining issues from my side.
/hold
@candita, I'm adding a hold in case you want to give this another look now that Ryan has added the haproxy.config checks.

openshift-ci · 2021-08-25T20:53:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Miciah, rfredette

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Miciah,rfredette]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Miciah · 2021-08-27T18:03:24Z

/hold cancel

openshift-bot · 2021-08-28T03:23:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T03:36:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T07:11:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T07:23:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T07:59:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T08:11:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T08:23:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T08:35:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T08:47:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T10:59:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T11:11:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T11:23:49Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T11:35:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T11:47:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T11:59:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T12:11:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T12:23:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T12:36:10Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T13:11:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T13:23:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T13:36:09Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T13:47:53Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T13:59:50Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-08-28T15:16:53Z

@rfredette: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-single-node	`855e6de`	link	`/test e2e-aws-single-node`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2021-08-28T15:35:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T15:47:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T18:47:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-28T18:59:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-08-28T19:20:54Z

@rfredette: All pull requests linked via external trackers have merged:

openshift/cluster-ingress-operator#644

Bugzilla bug 1986575 has been moved to the MODIFIED state.

In response to this:

Bug 1986575: Add e2e test cases for haproxy timeout api fields, and reject negative timeout values

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Aug 17, 2021

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 17, 2021

openshift-ci bot requested review from frobware and lmzuccarelli August 17, 2021 18:35

openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Aug 17, 2021

openshift-ci bot requested a review from lihongan August 17, 2021 19:06

candita reviewed Aug 17, 2021

View reviewed changes

Miciah reviewed Aug 23, 2021

View reviewed changes

Miciah reviewed Aug 25, 2021

View reviewed changes

rfredette force-pushed the haproxy-timeout-tests branch from f93a823 to 8f66ed0 Compare August 25, 2021 18:39

Miciah reviewed Aug 25, 2021

View reviewed changes

Add test cases for haproxy timeout api fields, and protect against ne…

855e6de

…gative timeout values HAProxy won't accept timeouts with negative values, so if one makes it into the ingresscontroller config, discard it and use the default value

rfredette force-pushed the haproxy-timeout-tests branch from 8f66ed0 to 855e6de Compare August 25, 2021 20:08

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 25, 2021

openshift-ci bot assigned Miciah Aug 25, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 25, 2021

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 27, 2021

openshift-merge-robot merged commit fbd8fcf into openshift:master Aug 28, 2021

	t.Errorf("Router pod %s failed to become ready: %v", routerPod.Name, pollErr)
	t.Fatalf("Router pod %s failed to become ready: %v", routerPod.Name, pollErr)

		t.Errorf("Error executing %s: %v", strings.Join(cmd, " "), err)
		t.Errorf("stderr: %v", stderr)

	t.Errorf("failed to list pods for ingress controllers %s: %v", ic.Name, err)
	t.Logf("failed to list pods for ingress controllers %s: %v", ic.Name, err)
	return false, nil

Bug 1986575: Add e2e test cases for haproxy timeout api fields, and reject negative timeout values #644

Bug 1986575: Add e2e test cases for haproxy timeout api fields, and reject negative timeout values #644

Conversation

rfredette commented Aug 17, 2021

openshift-ci bot commented Aug 17, 2021

rfredette commented Aug 17, 2021

openshift-ci bot commented Aug 17, 2021

rfredette commented Aug 17, 2021

openshift-ci bot commented Aug 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Miciah Aug 23, 2021 • edited

Choose a reason for hiding this comment

Miciah left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Miciah left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rfredette commented Aug 25, 2021

Miciah commented Aug 25, 2021

openshift-ci bot commented Aug 25, 2021

Miciah commented Aug 27, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-ci bot commented Aug 28, 2021 • edited

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-bot commented Aug 28, 2021

openshift-ci bot commented Aug 28, 2021

Miciah Aug 23, 2021 •

edited

openshift-ci bot commented Aug 28, 2021 •

edited