rest: retry on connection refused and apiserver shutdown #75368

mfojtik · 2019-03-14T12:11:52Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

The client-go seems to retry on connection refused in case of GET requests. There are more transient errors that we can retry instead of making users implement retry in their code. Two examples I can
this of are: "connection refused" errors and "apiserver is shutting down"... Both should be transient and I think we can safely retry those.

NONE

caesarxuchao · 2019-03-14T20:08:47Z

/assign @logicalhan

deads2k · 2019-03-18T15:21:17Z

@smarterclayton this is in keeping with a pull you wrote a while back I think.

@kubernetes/sig-api-machinery-misc

staging/src/k8s.io/client-go/rest/request.go

liggitt · 2019-03-18T15:27:43Z

staging/src/k8s.io/client-go/rest/request.go

 			// Thus in case of "GET" operations, we simply retry it.
 			// We are not automatically retrying "write" operations, as
 			// they are not idempotent.
-			if !net.IsConnectionReset(err) || r.verb != "GET" {
+			if !net.IsConnectionReset(err) || !IsConnectionRefused(err) || !isAPIServerShutdown(err) || r.verb != "GET" {


this logic isn't right... this will not retry unless a single error is simultaneously a connection reset AND a connection refused AND an "apiserver is shutting down" error.

~~this demonstrates the existing retry logic isn't exercised in tests :-/~~ unit tests did fail

it is, the unit test failed, mea culpa.

staging/src/k8s.io/client-go/rest/request.go

sttts · 2019-03-18T15:48:30Z

staging/src/k8s.io/apiserver/pkg/server/filters/waitgroup.go

@@ -38,6 +38,7 @@ func WithWaitGroup(handler http.Handler, longRunning apirequest.LongRunningReque

 		if !longRunning(req, requestInfo) {
 			if err := wg.Add(1); err != nil {
+				w.Header().Add("Retry-After", "1")
 				http.Error(w, "apiserver is shutting down.", http.StatusInternalServerError)


can we also switch this to a Status object?

i would do this as a follow up (it might require more plumbing and there are more places in code that return plain text errors via http.Error()).

is 1 appropriate?

@smarterclayton my assumption is that HA clusters have 2 other endpoints the client-go can reconnect when it gets apiserver shutdown error. So faster retry will lead to faster success?

staging/src/k8s.io/client-go/rest/request.go

smarterclayton · 2019-03-18T16:06:30Z

staging/src/k8s.io/client-go/rest/request.go

+				// TODO: Should we clean the original response if it exists?
+				resp = &http.Response{
+					StatusCode: http.StatusInternalServerError,
+					Header:     http.Header{"Retry-After": []string{"1"}},


1 seems too short

1 is pre-existing, and fires on connectionreset errors... 1 second doesn't seem awful for that, but does maybe seem too short for apiserver is shutting down

i would guess on apiserver shutdown the client-go retry request and possibly hit different endpoint that is not shutting down (in HA env).

Will this retry infinitely? Connection refused could be non-transient as well. We should at least fail eventually.

if seconds, wait := checkWait(resp); wait && retries < maxRetries { limits us to maxRetries

review with ignore whitespace, this is all preexisting.

staging/src/k8s.io/apiserver/pkg/server/filters/waitgroup.go

fejta-bot · 2019-10-13T14:15:01Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2019-10-13T14:15:09Z

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

deads2k · 2019-11-13T13:17:48Z

huh, can I force this to re-open?

deads2k · 2019-11-13T13:20:50Z

staging/src/k8s.io/apiserver/pkg/server/filters/waitgroup.go

@@ -38,7 +43,13 @@ func WithWaitGroup(handler http.Handler, longRunning apirequest.LongRunningReque

 		if !longRunning(req, requestInfo) {
 			if err := wg.Add(1); err != nil {
-				http.Error(w, "apiserver is shutting down.", http.StatusInternalServerError)
+				// When apiserver is shutting down, signal clients to retry
+				w.Header().Add("Retry-After", "1")


Add a comment explaining that the expectation is that with a load-balancer, you'll hit a different server, so a tight retry is good for client responsiveness.

comment added

deads2k · 2019-11-13T18:51:07Z

staging/src/k8s.io/apimachinery/pkg/util/net/util.go

-// Returns if the given err is "connection reset by peer" error.
-func IsConnectionReset(err error) bool {
+// Returns true if the given err is "connection refused" error.
+func IsConnectionRefused(err error) bool {


@mfojtik in your rebase, these got reordered. Check to see if this change is even needed anymore.

this is weird, but I see both functions (reset and refused) are present in file and needed, not sure why this was re-rdered.

…ing down

deads2k · 2019-11-20T16:45:34Z

/lgtm
/approve

k8s-ci-robot · 2019-11-20T16:47:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, mfojtik

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/apiserver/OWNERS~~ [deads2k]
~~staging/src/k8s.io/client-go/OWNERS~~ [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mfojtik · 2019-11-20T17:16:33Z

/retest

liggitt · 2020-01-24T01:23:56Z

Before this PR:

$ kubectl get pods --v=6
I0123 20:19:10.159029   99567 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:19:10.159153   99567 cached_discovery.go:121] skipped caching discovery info due to Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
I0123 20:19:10.159951   99567 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 0 milliseconds
I0123 20:19:10.159988   99567 cached_discovery.go:121] skipped caching discovery info due to Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
I0123 20:19:10.160015   99567 shortcut.go:89] Error loading discovery information: Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
I0123 20:19:10.160673   99567 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 0 milliseconds
I0123 20:19:10.160694   99567 cached_discovery.go:121] skipped caching discovery info due to Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
I0123 20:19:10.161307   99567 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 0 milliseconds
I0123 20:19:10.161337   99567 cached_discovery.go:121] skipped caching discovery info due to Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
I0123 20:19:10.161940   99567 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 0 milliseconds
I0123 20:19:10.161978   99567 cached_discovery.go:121] skipped caching discovery info due to Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
I0123 20:19:10.162007   99567 helpers.go:217] Connection error: Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused

The connection to the server localhost:8080 was refused - did you specify the right host or port?

After this PR:

$ kubectl get pods --v=6
I0123 20:21:58.562543    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:21:58.562669    4063 request.go:853] Got a Retry-After 1s response for attempt 1 to http://localhost:8080/api?timeout=32s
I0123 20:21:59.569996    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 2 milliseconds
I0123 20:21:59.570039    4063 request.go:853] Got a Retry-After 1s response for attempt 2 to http://localhost:8080/api?timeout=32s
I0123 20:22:00.577074    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:00.577129    4063 request.go:853] Got a Retry-After 1s response for attempt 3 to http://localhost:8080/api?timeout=32s
I0123 20:22:01.582189    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:01.582229    4063 request.go:853] Got a Retry-After 1s response for attempt 4 to http://localhost:8080/api?timeout=32s
I0123 20:22:02.587684    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:02.587739    4063 request.go:853] Got a Retry-After 1s response for attempt 5 to http://localhost:8080/api?timeout=32s
I0123 20:22:03.591833    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:03.591901    4063 request.go:853] Got a Retry-After 1s response for attempt 6 to http://localhost:8080/api?timeout=32s
I0123 20:22:04.596874    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:04.596954    4063 request.go:853] Got a Retry-After 1s response for attempt 7 to http://localhost:8080/api?timeout=32s
I0123 20:22:05.599101    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:05.599146    4063 request.go:853] Got a Retry-After 1s response for attempt 8 to http://localhost:8080/api?timeout=32s
I0123 20:22:06.601063    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:06.601101    4063 request.go:853] Got a Retry-After 1s response for attempt 9 to http://localhost:8080/api?timeout=32s
I0123 20:22:07.607365    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:07.607484    4063 cached_discovery.go:121] skipped caching discovery info due to an error on the server ("") has prevented the request from succeeding
I0123 20:22:07.608900    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:07.608943    4063 request.go:853] Got a Retry-After 1s response for attempt 1 to http://localhost:8080/api?timeout=32s
I0123 20:22:08.615392    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:08.615443    4063 request.go:853] Got a Retry-After 1s response for attempt 2 to http://localhost:8080/api?timeout=32s
I0123 20:22:09.621900    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:09.621947    4063 request.go:853] Got a Retry-After 1s response for attempt 3 to http://localhost:8080/api?timeout=32s
I0123 20:22:10.625904    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:10.625956    4063 request.go:853] Got a Retry-After 1s response for attempt 4 to http://localhost:8080/api?timeout=32s
I0123 20:22:11.629615    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:11.629660    4063 request.go:853] Got a Retry-After 1s response for attempt 5 to http://localhost:8080/api?timeout=32s
I0123 20:22:12.636470    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:12.636521    4063 request.go:853] Got a Retry-After 1s response for attempt 6 to http://localhost:8080/api?timeout=32s
I0123 20:22:13.642258    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:13.642307    4063 request.go:853] Got a Retry-After 1s response for attempt 7 to http://localhost:8080/api?timeout=32s
I0123 20:22:14.647828    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:14.647873    4063 request.go:853] Got a Retry-After 1s response for attempt 8 to http://localhost:8080/api?timeout=32s
I0123 20:22:15.655124    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:15.655173    4063 request.go:853] Got a Retry-After 1s response for attempt 9 to http://localhost:8080/api?timeout=32s
I0123 20:22:16.658749    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:16.658847    4063 cached_discovery.go:121] skipped caching discovery info due to an error on the server ("") has prevented the request from succeeding
I0123 20:22:16.658876    4063 shortcut.go:89] Error loading discovery information: an error on the server ("") has prevented the request from succeeding
I0123 20:22:16.660121    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:16.660178    4063 request.go:853] Got a Retry-After 1s response for attempt 1 to http://localhost:8080/api?timeout=32s
I0123 20:22:17.665749    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:17.665816    4063 request.go:853] Got a Retry-After 1s response for attempt 2 to http://localhost:8080/api?timeout=32s
I0123 20:22:18.671799    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:18.671845    4063 request.go:853] Got a Retry-After 1s response for attempt 3 to http://localhost:8080/api?timeout=32s
I0123 20:22:19.676815    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:19.676853    4063 request.go:853] Got a Retry-After 1s response for attempt 4 to http://localhost:8080/api?timeout=32s
I0123 20:22:20.679810    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:20.679896    4063 request.go:853] Got a Retry-After 1s response for attempt 5 to http://localhost:8080/api?timeout=32s
I0123 20:22:21.683950    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:21.683997    4063 request.go:853] Got a Retry-After 1s response for attempt 6 to http://localhost:8080/api?timeout=32s
I0123 20:22:22.689865    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:22.689910    4063 request.go:853] Got a Retry-After 1s response for attempt 7 to http://localhost:8080/api?timeout=32s
I0123 20:22:23.692097    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:23.692161    4063 request.go:853] Got a Retry-After 1s response for attempt 8 to http://localhost:8080/api?timeout=32s
I0123 20:22:24.695713    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:24.695763    4063 request.go:853] Got a Retry-After 1s response for attempt 9 to http://localhost:8080/api?timeout=32s
I0123 20:22:25.702348    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:25.702429    4063 cached_discovery.go:121] skipped caching discovery info due to an error on the server ("") has prevented the request from succeeding
I0123 20:22:25.703672    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:25.703704    4063 request.go:853] Got a Retry-After 1s response for attempt 1 to http://localhost:8080/api?timeout=32s
I0123 20:22:26.708807    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:26.708855    4063 request.go:853] Got a Retry-After 1s response for attempt 2 to http://localhost:8080/api?timeout=32s
I0123 20:22:27.716044    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:27.716091    4063 request.go:853] Got a Retry-After 1s response for attempt 3 to http://localhost:8080/api?timeout=32s
I0123 20:22:28.718190    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:28.718231    4063 request.go:853] Got a Retry-After 1s response for attempt 4 to http://localhost:8080/api?timeout=32s
I0123 20:22:29.724726    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:29.724783    4063 request.go:853] Got a Retry-After 1s response for attempt 5 to http://localhost:8080/api?timeout=32s
I0123 20:22:30.730743    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:30.730800    4063 request.go:853] Got a Retry-After 1s response for attempt 6 to http://localhost:8080/api?timeout=32s
I0123 20:22:31.735632    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:31.735681    4063 request.go:853] Got a Retry-After 1s response for attempt 7 to http://localhost:8080/api?timeout=32s
I0123 20:22:32.738399    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:32.738443    4063 request.go:853] Got a Retry-After 1s response for attempt 8 to http://localhost:8080/api?timeout=32s
I0123 20:22:33.744887    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:33.744932    4063 request.go:853] Got a Retry-After 1s response for attempt 9 to http://localhost:8080/api?timeout=32s
I0123 20:22:34.750506    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:34.750598    4063 cached_discovery.go:121] skipped caching discovery info due to an error on the server ("") has prevented the request from succeeding
I0123 20:22:34.751898    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:34.751936    4063 request.go:853] Got a Retry-After 1s response for attempt 1 to http://localhost:8080/api?timeout=32s
I0123 20:22:35.759263    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 2 milliseconds
I0123 20:22:35.759323    4063 request.go:853] Got a Retry-After 1s response for attempt 2 to http://localhost:8080/api?timeout=32s
I0123 20:22:36.761142    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:36.761181    4063 request.go:853] Got a Retry-After 1s response for attempt 3 to http://localhost:8080/api?timeout=32s
I0123 20:22:37.766472    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:37.766562    4063 request.go:853] Got a Retry-After 1s response for attempt 4 to http://localhost:8080/api?timeout=32s
I0123 20:22:38.773317    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:38.773360    4063 request.go:853] Got a Retry-After 1s response for attempt 5 to http://localhost:8080/api?timeout=32s
I0123 20:22:39.776635    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:39.776680    4063 request.go:853] Got a Retry-After 1s response for attempt 6 to http://localhost:8080/api?timeout=32s
I0123 20:22:40.782795    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:40.782853    4063 request.go:853] Got a Retry-After 1s response for attempt 7 to http://localhost:8080/api?timeout=32s
I0123 20:22:41.788655    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:41.788720    4063 request.go:853] Got a Retry-After 1s response for attempt 8 to http://localhost:8080/api?timeout=32s
I0123 20:22:42.791264    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:42.791306    4063 request.go:853] Got a Retry-After 1s response for attempt 9 to http://localhost:8080/api?timeout=32s
I0123 20:22:43.798359    4063 round_trippers.go:443] GET http://localhost:8080/api?timeout=32s  in 1 milliseconds
I0123 20:22:43.798463    4063 cached_discovery.go:121] skipped caching discovery info due to an error on the server ("") has prevented the request from succeeding
I0123 20:22:43.798689    4063 helpers.go:203] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "an error on the server (\"\") has prevented the request from succeeding",
  "reason": "InternalError",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse"
      }
    ],
    "retryAfterSeconds": 1
  },
  "code": 500
}]

Error from server (InternalError): an error on the server ("") has prevented the request from succeeding

This makes kubectl get against an invalid host/port retry many times, and ultimately error with a confusing "InternalError"

liggitt · 2020-01-24T01:27:16Z

opened #87510 to track this

k8s-ci-robot requested review from errordeveloper and gmarek March 14, 2019 12:13

k8s-ci-robot assigned logicalhan Mar 14, 2019

liggitt reviewed Mar 18, 2019

View reviewed changes

staging/src/k8s.io/client-go/rest/request.go Outdated Show resolved Hide resolved

deads2k reviewed Mar 18, 2019

View reviewed changes

staging/src/k8s.io/client-go/rest/request.go Outdated Show resolved Hide resolved

liggitt reviewed Mar 18, 2019

View reviewed changes

mfojtik force-pushed the retry-on-errors branch from 28a598d to c7c19b8 Compare March 18, 2019 15:28

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 18, 2019

mfojtik force-pushed the retry-on-errors branch from c7c19b8 to d910cea Compare March 18, 2019 15:30

liggitt reviewed Mar 18, 2019

View reviewed changes

staging/src/k8s.io/client-go/rest/request.go Outdated Show resolved Hide resolved

mfojtik force-pushed the retry-on-errors branch from d910cea to ecc4c87 Compare March 18, 2019 15:47

sttts reviewed Mar 18, 2019

View reviewed changes

mfojtik force-pushed the retry-on-errors branch from ecc4c87 to 4e49c60 Compare March 18, 2019 15:50

k8s-ci-robot added the area/apiserver label Mar 18, 2019

liggitt reviewed Mar 18, 2019

View reviewed changes

staging/src/k8s.io/client-go/rest/request.go Outdated Show resolved Hide resolved

smarterclayton reviewed Mar 18, 2019

View reviewed changes

mfojtik force-pushed the retry-on-errors branch from 4e49c60 to 9fda5a4 Compare March 19, 2019 09:41

sttts reviewed Mar 19, 2019

View reviewed changes

staging/src/k8s.io/apiserver/pkg/server/filters/waitgroup.go Outdated Show resolved Hide resolved

mfojtik force-pushed the retry-on-errors branch from 9fda5a4 to 9f4cf47 Compare March 19, 2019 09:44

k8s-ci-robot added the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Sep 13, 2019

k8s-ci-robot closed this Oct 13, 2019

deads2k reopened this Nov 13, 2019

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 13, 2019

deads2k reviewed Nov 13, 2019

View reviewed changes

mfojtik force-pushed the retry-on-errors branch from 35db800 to 377ad81 Compare November 13, 2019 13:26

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 13, 2019

mfojtik force-pushed the retry-on-errors branch from 377ad81 to edc52e3 Compare November 13, 2019 13:33

deads2k reviewed Nov 13, 2019

View reviewed changes

mfojtik force-pushed the retry-on-errors branch 2 times, most recently from 57f8bb7 to 14aea7c Compare November 20, 2019 15:58

mfojtik added 2 commits November 20, 2019 17:30

client-go: add connection refused to list of transient errors

9aa68a5

apiserver: add Retry-After header to response when apiserver is shutt…

a3c82e8

…ing down

mfojtik force-pushed the retry-on-errors branch from 14aea7c to a3c82e8 Compare November 20, 2019 16:31

k8s-ci-robot assigned deads2k Nov 20, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 20, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 20, 2019

k8s-ci-robot merged commit 6666177 into kubernetes:master Nov 23, 2019

k8s-ci-robot added this to the v1.18 milestone Nov 23, 2019

This was referenced Jan 24, 2020

Kubectl not working on GCE master #87481

Closed

connection refused errors retry slowly and ultimately return incorrect error #87510

Closed

p0lyn0mial mentioned this pull request Jun 17, 2020

client-go library should retry on connection refused errors #92218

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rest: retry on connection refused and apiserver shutdown #75368

rest: retry on connection refused and apiserver shutdown #75368

mfojtik commented Mar 14, 2019

caesarxuchao commented Mar 14, 2019

deads2k commented Mar 18, 2019

liggitt Mar 18, 2019 •

edited

mfojtik Mar 18, 2019

sttts Mar 18, 2019

mfojtik Mar 18, 2019

smarterclayton Mar 18, 2019

mfojtik Mar 18, 2019

smarterclayton Mar 18, 2019

liggitt Mar 18, 2019

mfojtik Mar 18, 2019

sttts Apr 9, 2019

liggitt Apr 9, 2019

deads2k Nov 13, 2019

fejta-bot commented Oct 13, 2019

k8s-ci-robot commented Oct 13, 2019

deads2k commented Nov 13, 2019

deads2k Nov 13, 2019

mfojtik Nov 15, 2019

deads2k Nov 13, 2019

mfojtik Nov 15, 2019

deads2k commented Nov 20, 2019

k8s-ci-robot commented Nov 20, 2019

mfojtik commented Nov 20, 2019

liggitt commented Jan 24, 2020 •

edited

liggitt commented Jan 24, 2020

rest: retry on connection refused and apiserver shutdown #75368

rest: retry on connection refused and apiserver shutdown #75368

Conversation

mfojtik commented Mar 14, 2019

caesarxuchao commented Mar 14, 2019

deads2k commented Mar 18, 2019

liggitt Mar 18, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fejta-bot commented Oct 13, 2019

k8s-ci-robot commented Oct 13, 2019

deads2k commented Nov 13, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deads2k commented Nov 20, 2019

k8s-ci-robot commented Nov 20, 2019

mfojtik commented Nov 20, 2019

liggitt commented Jan 24, 2020 • edited

liggitt commented Jan 24, 2020

liggitt Mar 18, 2019 •

edited

liggitt commented Jan 24, 2020 •

edited