Implement transparent retries for gRFC A6 #1597

dfawley · 2017-10-19T22:38:31Z

Also fixes #1532

tamird · 2017-10-21T21:38:11Z

call.go

+		// Check to make sure the context has expired.  This will prevent us from
+		// looping forever if an error occurs for wait-for-ready RPCs where no data
+		// is sent on the wire.
+		select {


there's no reason to use a select here, and doing so can result in worse performance than the simpler alternative:

if err := ctx.Err(); err != nil { return toRPCErr(err) }

It turns out this is not true. I made some benchmarks, and found the following results:

BenchmarkCancelContextErrNoErr-12 100000000 52.2 ns/op BenchmarkCancelContextErrGotErr-12 100000000 51.9 ns/op BenchmarkCancelContextChannelNoErr-12 200000000 19.9 ns/op BenchmarkCancelContextChannelGotErr-12 100000000 36.1 ns/op

Using the channel is faster as of today. I believe the problem is the defer in context.Err():

https://github.com/golang/net/blob/cd69bc3fc700721b709c3a59e16e24c67b58f6ff/context/pre_go17.go#L162

These numbers line up well with our benchmarks for mutexes vs. channels and defer vs no-defer:

BenchmarkMutexWithDefer-12 20000000 51.2 ns/op BenchmarkMutex-12 100000000 14.5 ns/op BenchmarkSelectClosed-12 50000000 20.3 ns/op BenchmarkSelectOpen-12 300000000 5.74 ns/op

Note how the select is faster than the mutex with a defer, but slower than the mutex with an inline Unlock().

Quite surprising, thanks for the details. Your comment points to pre_go17.go but I assume these benchmarks were run on 1.9?

Also interesting that selecting on the open channel is faster than the mutex with inline Unlock.

By the way, there's something fishy here - how can BenchmarkCancelContextChannelGotErr ever be faster than BenchmarkCancelContextErrGotErr? The former does strictly more work.

Quite surprising, thanks for the details. Your comment points to pre_go17.go but I
assume these benchmarks were run on 1.9?

Yes -- presumably the std context package is implemented identically (based on the results that makes sense).

The former does strictly more work.

Good catch. Actually, it does not do strictly more work, because I forgot to call Err() in this case. Adding it results in:

BenchmarkCancelContextErrGotErr-12 100000000 52.4 ns/op BenchmarkCancelContextChannelGotErr-12 50000000 94.8 ns/op BenchmarkTimerContextErrGotErr-12 100000000 52.5 ns/op BenchmarkTimerContextChannelGotErr-12 50000000 86.7 ns/op

I would still lean toward the existing approach, as it optimizes the non-error case. But maybe the Go team should think about removing that defer...

cc @Sajmani -- looks like that defer has been there since inception; perhaps it's worth replacing with the an inline Unlock as @dfawley has shown.

FWIW I filed golang/go#22401 for this.

tamird · 2017-10-21T21:38:25Z

stream.go

+		// Check to make sure the context has expired.  This will prevent us from
+		// looping forever if an error occurs for wait-for-ready RPCs where no data
+		// is sent on the wire.
+		select {


Ditto:

if err := ctx.Err(); err != nil { return toRPCErr(err) }

menghanl

The changes LGTM.
Some possible goaway cleanup in transport.

menghanl · 2017-10-26T17:49:31Z

transport/transport.go

@@ -689,6 +672,12 @@ var (
 	// ErrStreamDrain indicates that the stream is rejected by the server because
 	// the server stops accepting new RPCs.
 	ErrStreamDrain = streamErrorf(codes.Unavailable, "the server stops accepting new RPCs")
+	// StatusStreamRefused indicates that the server sent a RST_STREAM with the
+	// REFUSED_STREAM code, meaning the stream was entirely unprocessed.
+	StatusStreamRefused = status.New(codes.Unavailable, "the server refused the stream")


Does this need to be exported?

Deleted entirely (it was unused).

menghanl · 2017-10-26T18:00:27Z

transport/http2_client.go

+			// The stream was unprocessed by the server.
+			stream.mu.Lock()
+			stream.unprocessed = true
+			stream.finish(StatusGoAway)


What's the purpose of this finish()?
finish closes s.done, which may result in race in wait().

See following comment on ErrStreamDrain.
If we do a cleanup on stream goaway, this finish() may become necessary.

I'll leave both the finish and the closing of s.done for now; we will clean things up in the future.

Unexported statusGoAway.

menghanl · 2017-10-26T22:27:15Z

transport/transport.go

@@ -689,6 +672,12 @@ var (
 	// ErrStreamDrain indicates that the stream is rejected by the server because
 	// the server stops accepting new RPCs.
 	ErrStreamDrain = streamErrorf(codes.Unavailable, "the server stops accepting new RPCs")


Do we need this anymore? Seems it's only returned by transport but never check.

IIRC, the purpose of this error is exactly what Unprocessed() is for.

And if we no longer need this error, we can also (maybe) remove s.goaway channel.
But should probably make this a TODO for another cleanup PR.

Unexported, added TODO. We have a lot more cleaning up we can do.

dfawley · 2017-11-03T16:09:30Z

Ping: rebased & resolved conflicts.

menghanl · 2017-10-31T21:13:15Z

test/end2end_test.go

+	for ; state == connectivity.Ready && cc.WaitForStateChange(ctx, state); state = cc.GetState() {
+	}
+	if state == connectivity.Ready {
+		t.Fatalf("Want connection state to be Ready, got %v", state)


want state to be not Ready?

menghanl · 2017-11-03T01:21:13Z

test/end2end_test.go

+	if state == connectivity.Ready {
+		t.Fatalf("Want connection state to be Ready, got %v", state)
+	}
+	cancel()


context will be leaked if t.Fafalf before this.
Move this before the if check?

menghanl · 2017-11-03T01:27:05Z

test/end2end_test.go

 		t.Fatalf("TestService/EmptyCall(_, _) = _, %v, want _, <nil>", err)
 	}
+	cancel()


Not cancel on t.Fatalf?
defer?

dfawley

Thanks, PTAL.

dfawley · 2017-11-03T17:30:43Z

test/end2end_test.go

+	for ; state == connectivity.Ready && cc.WaitForStateChange(ctx, state); state = cc.GetState() {
+	}
+	if state == connectivity.Ready {
+		t.Fatalf("Want connection state to be Ready, got %v", state)


dfawley · 2017-11-03T17:31:38Z

test/end2end_test.go

+	if state == connectivity.Ready {
+		t.Fatalf("Want connection state to be Ready, got %v", state)
+	}
+	cancel()


dfawley · 2017-11-03T17:34:17Z

test/end2end_test.go

 		t.Fatalf("TestService/EmptyCall(_, _) = _, %v, want _, <nil>", err)
 	}
+	cancel()


menghanl

🎉

dfawley requested a review from menghanl October 19, 2017 23:44

dfawley assigned menghanl and dfawley and unassigned dfawley Oct 19, 2017

dfawley mentioned this pull request Oct 20, 2017

New grpclb implementation #1558

Merged

tamird reviewed Oct 21, 2017

View reviewed changes

menghanl reviewed Oct 26, 2017

View reviewed changes

dfawley added 3 commits November 3, 2017 08:53

Implement transparent retries for gRFC A6

0946f33

remove commented-out code

46290b7

cleanups from review

12d4904

dfawley force-pushed the transparent_retries2 branch from 0928cf8 to 12d4904 Compare November 3, 2017 16:09

menghanl reviewed Nov 3, 2017

View reviewed changes

test cleanups

09c482d

dfawley commented Nov 3, 2017

View reviewed changes

menghanl approved these changes Nov 6, 2017

View reviewed changes

dfawley merged commit 8ff8683 into grpc:master Nov 6, 2017

menghanl added this to the 1.8 Release milestone Nov 7, 2017

menghanl added the Type: Feature New features or improvements in behavior label Nov 7, 2017

dfawley deleted the transparent_retries2 branch April 30, 2018 16:12

lock bot locked as resolved and limited conversation to collaborators Oct 27, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement transparent retries for gRFC A6 #1597

Implement transparent retries for gRFC A6 #1597

dfawley commented Oct 19, 2017

tamird Oct 21, 2017

dfawley Oct 23, 2017

tamird Oct 23, 2017

tamird Oct 23, 2017

dfawley Oct 23, 2017

tamird Oct 23, 2017 •

edited

Loading

dfawley Oct 23, 2017

tamird Oct 21, 2017

menghanl left a comment

menghanl Oct 26, 2017

dfawley Oct 31, 2017

menghanl Oct 26, 2017

dfawley Oct 31, 2017

menghanl Oct 26, 2017 •

edited

Loading

dfawley Oct 31, 2017

dfawley commented Nov 3, 2017

menghanl Oct 31, 2017

dfawley Nov 3, 2017

menghanl Nov 3, 2017

dfawley Nov 3, 2017

menghanl Nov 3, 2017

dfawley Nov 3, 2017

dfawley left a comment

dfawley Nov 3, 2017

dfawley Nov 3, 2017

dfawley Nov 3, 2017

menghanl left a comment

Implement transparent retries for gRFC A6 #1597

Implement transparent retries for gRFC A6 #1597

Conversation

dfawley commented Oct 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tamird Oct 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

menghanl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

menghanl Oct 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfawley commented Nov 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfawley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

menghanl left a comment

Choose a reason for hiding this comment

tamird Oct 23, 2017 •

edited

Loading

menghanl Oct 26, 2017 •

edited

Loading