New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use SSL health checks for ELBs when backend protocol is SSL/HTTPS #70309

Merged
merged 3 commits into from Feb 1, 2019

Conversation

@2rs2ts
Copy link
Contributor

2rs2ts commented Oct 26, 2018

Fixes #45746

What type of PR is this?

/kind feature

What this PR does / why we need it:
Previously, all health checks were HTTP or TCP type, which meant that if the backend was actually listening for SSL traffic, there would be SSL handshake errors, and out of the box this causes many servers to log a lot of noise about said handshake errors. By setting the health check protocol to HTTPS/SSL, these errors can be avoided.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #45746

Special notes for your reviewer:
I didn't see any established tests for this code path, and since it's happening in a very long function I didn't feel comfortable trying to write a new test, so I apologize for not including any tests here.

Does this PR introduce a user-facing change?:

AWS ELB health checks will now use HTTPS/SSL protocol for HTTPS/SSL backends.

/sig aws

@2rs2ts

This comment has been minimized.

Copy link
Contributor Author

2rs2ts commented Oct 26, 2018

/assign @justinsb

@idealhack

This comment has been minimized.

Copy link
Member

idealhack commented Oct 29, 2018

/ok-to-test

@sstarcher sstarcher referenced this pull request Nov 19, 2018

Closed

TLS enabled error spam #355

@idealhack

This comment has been minimized.

Copy link
Member

idealhack commented Nov 19, 2018

kindly ping @2rs2ts for rebase

@2rs2ts

This comment has been minimized.

Copy link
Contributor Author

2rs2ts commented Nov 27, 2018

can this PR be merged now?

@idealhack

This comment has been minimized.

Copy link
Member

idealhack commented Nov 28, 2018

@mcrute

This comment has been minimized.

Copy link
Member

mcrute commented Nov 28, 2018

/lgtm

@micahhausler

This comment has been minimized.

Copy link
Member

micahhausler commented Nov 28, 2018

Is this going to work with self-signed certificates on the backend? What about a kubernetes-CA signed kubelet-server certificate? @2rs2ts have you tested this?

@mcrute

This comment has been minimized.

Copy link
Member

mcrute commented Nov 28, 2018

Health checks do not consider certificate chain trust, just the ability to connect via TLS and to read an HTTP response.

@2rs2ts

This comment has been minimized.

Copy link
Contributor Author

2rs2ts commented Nov 29, 2018

@micahhausler I did in fact test this with a backend cert signed by a company-internal CA that isn't explicitly trusted by the ELBs. The ELBs don't have backend cert trust policies set up on them anyway, so they will trust anything.

@2rs2ts

This comment has been minimized.

Copy link
Contributor Author

2rs2ts commented Dec 31, 2018

/unassign @justinsb

(I hear he is no longer reviewing PRs)

@k8s-ci-robot k8s-ci-robot added the lgtm label Jan 3, 2019

@2rs2ts

This comment has been minimized.

Copy link
Contributor Author

2rs2ts commented Jan 4, 2019

I do not understand these errors, to be quite honest they seem unrelated to my changes.

/retest

@M00nF1sh

This comment has been minimized.

Copy link
Contributor

M00nF1sh commented Jan 4, 2019

I do not understand these errors, to be quite honest they seem unrelated to my changes.

/retest

I don't think it's related to your PR, we'll investigate it and let you know

@2rs2ts

This comment has been minimized.

Copy link
Contributor Author

2rs2ts commented Jan 11, 2019

@M00nF1sh how goes the investigation?

@M00nF1sh

This comment has been minimized.

Copy link
Contributor

M00nF1sh commented Jan 11, 2019

/test pull-kubernetes-e2e-kops-aws
/test pull-kubernetes-integration
i don't got time to investigate this yet, these tests have been flake for a long time.(trying rerun :D)

@2rs2ts

This comment has been minimized.

Copy link
Contributor Author

2rs2ts commented Jan 28, 2019

/test pull-kubernetes-e2e-kops-aws

@2rs2ts

This comment has been minimized.

Copy link
Contributor Author

2rs2ts commented Jan 29, 2019

What should I do? Rope in sig-testing?

@justinsb justinsb added kind/bug and removed kind/feature labels Jan 31, 2019

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Jan 31, 2019

Not sure where you heard I'm no longer reviewing PRs, though I'm certainly not able to keep up in a timely manner...

The test failures are due to an issue with the payments on the AWS account used for testing and are being worked on.

Code looks good and because it's an beta annotation I think we're OK to fix it as a straight bug.

/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 31, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 2rs2ts, justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Jan 31, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

3 similar comments
@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Jan 31, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Jan 31, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Jan 31, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@2rs2ts

This comment has been minimized.

Copy link
Contributor Author

2rs2ts commented Jan 31, 2019

@justinsb someone on the k8s slack told me so, but it was a long time ago so I don't have the link to the message :(

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Jan 31, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

8 similar comments
@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Jan 31, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Jan 31, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Feb 1, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Feb 1, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Feb 1, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Feb 1, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Feb 1, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Feb 1, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Feb 1, 2019

@2rs2ts: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-e2e-kops-aws 72895a8 link /test pull-kubernetes-e2e-kops-aws

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot merged commit b0a455b into kubernetes:master Feb 1, 2019

19 checks passed

cla/linuxfoundation 2rs2ts authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kops-aws Context retired without replacement.
pull-kubernetes-e2e-kubeadm-gce Skipped
pull-kubernetes-godeps Skipped
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details

@2rs2ts 2rs2ts deleted the 2rs2ts:ssl-healthchecks branch Feb 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment