Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't follow non-local redirects from HTTP probes #75416

Merged
merged 2 commits into from Mar 16, 2019

Conversation

@tallclair
Copy link
Member

commented Mar 15, 2019

What type of PR is this?
/kind bug

What this PR does / why we need it:

HTTP probes currently follow redirects, and perform the health check against the endpoint redirect to. This behavior differs from the documented behavior, which states that any HTTP response with a status code between 200 and 400 should be treated as success.

As a backwards-compatibility compromise, this PR modifies the probe logic follow redirects if they go to the same hostname (e.g. a relative redirect to a different path). If the redirect points to a different hostname, then the response is simply treated as a success.

Which issue(s) this PR fixes:
Fixes #73172

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

ACTION REQUIRED: Health check (liveness & readiness) probes using an HTTPGetAction will no longer follow redirects to different hostnames from the original probe request. Instead, these non-local redirects will be treated as a Success (the documented behavior). In this case an event with reason "ProbeWarning" will be generated, indicating that the redirect was ignored. If you were previously relying on the redirect to run health checks against different endpoints, you will need to perform the healthcheck logic outside the Kubelet, for instance by proxying the external endpoint rather than redirecting to it.

/assign @thockin @liggitt
/cc @redbaron
/sig node network
/priority important-soon

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2019

@tallclair: GitHub didn't allow me to request PR reviews from the following users: redbaron.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

What type of PR is this?
/kind bug

What this PR does / why we need it:

HTTP probes currently follow redirects, and perform the health check against the endpoint redirect to. This behavior differs from the documented behavior, which states that any HTTP response with a status code between 200 and 400 should be treated as success.

As a backwards-compatibility compromise, this PR modifies the probe logic follow redirects if they go to the same hostname (e.g. a relative redirect to a different path). If the redirect points to a different hostname, then the response is simply treated as a success.

Which issue(s) this PR fixes:
Fixes #73172

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

ACTION REQUIRED: Health check (liveness & readiness) probes using an HTTPGetAction will no longer follow redirects to different hostnames from the original probe request. Instead, these non-local redirects will be treated as a Success (the documented behavior). In this case an event with reason "ProbeWarning" will be generated, indicating that the redirect was ignored. If you were previously relying on the redirect to run health checks against different endpoints, you will need to perform the healthcheck logic outside the Kubelet, for instance by proxying the external endpoint rather than redirecting to it.

/assign @thockin @liggitt
/cc @redbaron
/sig node network
/priority important-soon

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@@ -215,7 +215,7 @@ func initImageConfigs() map[int]Config {
configs[Iperf] = Config{e2eRegistry, "iperf", "1.0"}
configs[JessieDnsutils] = Config{e2eRegistry, "jessie-dnsutils", "1.0"}
configs[Kitten] = Config{e2eRegistry, "kitten", "1.0"}
configs[Liveness] = Config{e2eRegistry, "liveness", "1.0"}
configs[Liveness] = Config{e2eRegistry, "liveness", "1.1"}

This comment has been minimized.

Copy link
@thockin

thockin Mar 15, 2019

Member

Has this been pushed already?

This comment has been minimized.

Copy link
@tallclair

tallclair Mar 15, 2019

Author Member

Yeah.

@thockin
Copy link
Member

left a comment

It's an ugly compromise, but I guess that's what a compromise is.

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label Mar 15, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tallclair, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@thockin

This comment has been minimized.

Copy link
Member

commented Mar 15, 2019

/retest

@spiffxp

This comment has been minimized.

Copy link
Member

commented Mar 15, 2019

/milestone v1.14
v1.14 release lead here, this is the sort of change I wouldn't want to see cherry-picked into v1.14.1 as it changes behavior... and this seems worthwhile to include now instead of waiting until v1.15

@k8s-ci-robot k8s-ci-robot added this to the v1.14 milestone Mar 15, 2019

@fejta-bot

This comment has been minimized.

Copy link

commented Mar 16, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

1 similar comment
@fejta-bot

This comment has been minimized.

Copy link

commented Mar 16, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

}

return func(req *http.Request, via []*http.Request) error {
if req.URL.Hostname() != via[0].URL.Hostname() {

This comment has been minimized.

Copy link
@redbaron

redbaron Mar 16, 2019

Contributor

AFAIK relative redirects can have no hostname, just "Location: /another/endpoint" , does go http client normalize them to have same hostname?

This comment has been minimized.

Copy link
@tallclair

tallclair Mar 18, 2019

Author Member

Yes, the URL in the request is parsed relative to the previous request: https://golang.org/src/net/http/client.go?s=18538:18870#L563

This case is also verified by the unit test.

@fejta-bot

This comment has been minimized.

Copy link

commented Mar 16, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

1 similar comment
@fejta-bot

This comment has been minimized.

Copy link

commented Mar 16, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 3512757 into kubernetes:master Mar 16, 2019

17 checks passed

cla/linuxfoundation tallclair authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.