Skip to content

use TCPServer for rake-task readiness checks#1975

Merged
tcdowney merged 1 commit intomasterfrom
update-tcp-probe-175388005
Nov 30, 2020
Merged

use TCPServer for rake-task readiness checks#1975
tcdowney merged 1 commit intomasterfrom
update-tcp-probe-175388005

Conversation

@tcdowney
Copy link
Copy Markdown
Member

The socket we were opening would pass the readiness probe checks initially, but cause it to fail later with errors like:

  ----     ------     ----                  ----                                          -------
  Normal   Scheduled  10m                   default-scheduler                             Successfully assigned cf-system/cf-api-clock-65df859bf7-m2nw5 to gke-tim2-default-pool-e0832f49-cpq3
  Normal   Pulling    10m                   kubelet, gke-tim2-default-pool-e0832f49-cpq3  Pulling image "gcr.io/cf-capi-arya/dev-ccng:kbld-rand-1606256436134787000-1515422298211"
  Normal   Pulled     10m                   kubelet, gke-tim2-default-pool-e0832f49-cpq3  Successfully pulled image "gcr.io/cf-capi-arya/dev-ccng:kbld-rand-1606256436134787000-1515422298211"
  Normal   Created    10m                   kubelet, gke-tim2-default-pool-e0832f49-cpq3  Created container cf-api-clock
  Normal   Started    10m                   kubelet, gke-tim2-default-pool-e0832f49-cpq3  Started container cf-api-clock
  Warning  Unhealthy  9m55s (x3 over 10m)   kubelet, gke-tim2-default-pool-e0832f49-cpq3  Readiness probe failed: dial tcp 10.4.2.195:4446: connect: connection refused
  Warning  Unhealthy  18s (x91 over 4m48s)  kubelet, gke-tim2-default-pool-e0832f49-cpq3  Readiness probe failed: dial tcp 10.4.2.195:4446: i/o timeout

This replaces the raw socket with a TCPServer that runs in a background thread that can accept multiple readiness probe connections.

Relates to: cloudfoundry/capi-k8s-release#91

Tracker Story: #175388005

- the Socket we were opening would pass the readiness probe checks
initially but begin failing later
- this replaces it with a TCPServer that runs in a background thread
that can accept multiple readiness probe connections
- addresses cloudfoundry/capi-k8s-release#91

[#175388005](https://www.pivotaltracker.com/story/show/175388005)

Authored-by: Tim Downey <tdowney@vmware.com>
Copy link
Copy Markdown
Contributor

@cwlbraa cwlbraa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Before we tried to unit test this, we had implemented the full server with the accepts. Did y'all ever actually see errors? What did they look like?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants