ELB SSL handshake broken in us-east-2

[justinsb]

At least in us-east-2, with --topology=private, ELB reports master instances as being out of service when the health check is configured for SSL. Manually changing to TCP fixes things immediately.

apiserver is logging http: TLS handshake error from 172.20.1.222:38426: EOF during the failed health checks. It logs it with the TCP health-checks as well though!

Reported here: #6172 (comment)

I was able to reproduce in us-east-2

[justinsb]

Likely related to 2accc73

cc @johanneswuerbach

Although honestly we should do a better health check anyway....

[justinsb]

(by better health check I mean one that actually checks the health of apiserver & etcd, not just whether or not apiserver is listening - this predates 2accc73)

[johanneswuerbach]

Strange, we run all our clusters in us-east-1 and eu-west-1 with API ELBs configured with Ping Target | SSL:443 and never experienced any issues.

[johanneswuerbach]

Maybe its caused by --topology=private and SSL checks actually doing more back-and-forth then plain TCP checks?

We had similar issues in the past with misconfigured security groups or route tables on one-side of the connection, which allowed to open a connection, but they stalled as responses never arrived.

[markine]

The us-east-1 vs us-east-2 contrast can be seen by using:

kops create cluster NAME --cloud aws --networking kopeio-vxlan --master-zones ZONES --zones ZONES --ssh-public-key KEY --master-size SIZE --master-volume-size SIZE --node-size SIZE --node-volume-size SIZE --node-count COUNT --output yaml --topology private --bastion --image AMI

and changing just the zones and the AMI (have to pick the AMI for the right zone). us-east-1 works, us-east-2 does not.

kops 1.10.0 (and kubernetes 1.10.11)

[justinsb]

Yes, what's weird is that I think sometimes us-east-2 works. I thought it was a kubernetes 1.10 vs 1.11 thing, but now I'm leaning towards it being random.

I've captured some packets, and it looks like the ELB health check is behaving more like a TCP health check - opening the connection and then closing it again. No TLS handshake being initiated.

The SSL code has been in kops for a few releases now, so I am leaning towards it being a problem with ELB in us-east-2. But nothing on the AWS status and nobody else reporting it, which is odd.

[alexander-semenets]

The same issue for our infrastructure. We have clusters provisioned with the same codebase in us-east-1 and us-east-2. Us-east-2 has ssl checks failed while the east-1 works well. Trying to investigate with AWS Support now.

[alexander-semenets]

I've just got a confirmation from AWS Support team that there was an issue with CLB update at us-east-2 region. The problem was fixed and CLB works great again at my env.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.