Allow blackholing requests #681

peats-bond · 2018-02-16T20:23:48Z

This enables us to blackhole requests and release unnecessary long-lived goroutines.

Confirmed with pprof and YARPC that the only goroutines left running are for each connection.

codecov · 2018-02-16T20:28:26Z

Codecov Report

Merging #681 into dev will increase coverage by 0.18%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##              dev     #681      +/-   ##
==========================================
+ Coverage   86.68%   86.87%   +0.18%     
==========================================
  Files          39       39              
  Lines        3959     3961       +2     
==========================================
+ Hits         3432     3441       +9     
+ Misses        403      399       -4     
+ Partials      124      121       -3

Impacted Files	Coverage Δ
inbound.go	`81.05% <100%> (+0.2%)`	⬆️
mex.go	`72.09% <100%> (-7.91%)`	⬇️
outbound.go	`78.73% <0%> (-2.88%)`	⬇️
channel.go	`88.03% <0%> (+1.42%)`	⬆️
connection.go	`85.64% <0%> (+6.04%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 75acd77...f4db7cc. Read the comment docs.

prashantv

I'd rather avoid a new channel allocation in this path. I think we can reuse response.cancel as-is right now. Just call that method, and the current code will hit the ctx.Done() case, and will expire the inbound.

No strong opinions on this, but instead of a new method, should this be a new error that we can detect in SendSystemError?

peats-bond · 2018-02-22T22:52:20Z

I used response.cancel() as suggested and confirmed again with pprof that we're not keeping goroutines running after blackholing the request.

I'd rather use this method than create a new error with SendSystemError. I think this is a bit more straightforward and would otherwise cause SendSystemError to be a misnomer.

I've also added a test, but it relies on the context timing out so its essentially a sleep for 10 milliseconds. I'm not sure what your thoughts on this are @prashantv, but I think that this is the most reasonable way for me to test the change.

HelloGrayson

Suggest eyes from other reviewers as well :)

HelloGrayson · 2018-02-23T00:55:36Z

mex.go

@@ -390,7 +390,7 @@ func (mexset *messageExchangeSet) removeExchange(msgID uint32) {
 // will write to the send channel.
 func (mexset *messageExchangeSet) expireExchange(msgID uint32) {
 	mexset.log.Debugf(
-		"Removing %s message exchange %d due to timeout or cancelation",
+		"Removing %s message exchange %d due to timeout, cancelation or blackhole",


cancellation*

Indeed. Corroboration for consistency (since technically both spellings are accepted):

tchannel-go ❯ git grep -i cancellation CHANGELOG.md:* Fix context cancellation not cancelling Dial attempts (#541) inbound.go: // TODO(prashant): This is an expensive way to check for cancellation. Use a heap for timeouts. inbound.go: call.log.Debugf("Wait for timeout/cancellation interrupted by error: %v", call.mex.errCh.err) mex.go:// It returns any existing errors (timeout, cancellation, connection errors). mex.go: // 1. Timeouts/cancellation (mex.ctx errors) mex.go: // 1. Timeouts/cancellation (mex.ctx errors) tchannel-go ❯ git grep -i cancelation mex.go: "Removing %s message exchange %d due to timeout or cancelation",

HelloGrayson · 2018-02-23T00:56:07Z

inbound.go

@@ -361,6 +361,12 @@ func (response *InboundCallResponse) SetApplicationError() error {
 	return nil
 }

+// Blackhole indicates that there should be no response and provides
+// an opportunity to clean up resources.
+func (response *InboundCallResponse) Blackhole() {


Simpler than I thought it was going to be.

prashantv · 2018-02-23T19:40:30Z

inbound_test.go

+		clientCh := tServer.NewClient(testutils.NewOpts())
+		defer clientCh.Close()
+
+		ctx, cancel := NewContext(time.Millisecond * 10)


nit: use testutils.Timeout here

prashantv · 2018-02-23T19:41:01Z

inbound_test.go

+	testutils.WithTestServer(t, nil, func(tServer *testutils.TestServer) {
+		serviceName := tServer.ServiceName()
+
+		server := tServer.NewServer(testutils.NewOpts())


why are you creating a new server here? the tServer can have methods registered directly

yup, my bad. I overlooked the register function completely.

prashantv · 2018-02-23T19:41:14Z

inbound_test.go

+			inbound.Response().Blackhole()
+		}))
+
+		clientCh := tServer.NewClient(testutils.NewOpts())


nit: pass in nil for default options to be consistent with other tests

prashantv · 2018-02-23T19:48:14Z

inbound_test.go

+		subCh := server.GetSubChannel(serviceName)
+		subCh.SetHandler(HandlerFunc(func(ctx context.Context, inbound *InboundCall) {
+			// blackhole all requests
+			inbound.Response().Blackhole()


If you replace the implementation of Blackhole with a time.Sleep, the test will still pass.

diff --git a/inbound.go b/inbound.go index a7f7739b..605bc52f 100644 --- a/inbound.go +++ b/inbound.go @@ -364,7 +364,8 @@ func (response *InboundCallResponse) SetApplicationError() error { // Blackhole indicates that there should be no response and provides // an opportunity to clean up resources. func (response *InboundCallResponse) Blackhole() { - response.cancel() + time.Sleep(200 * time.Millisecond) + //response.cancel() }

> go test -v -run TestBlackhole === RUN TestBlackhole --- PASS: TestBlackhole (0.42s) PASS ok github.com/uber/tchannel-go 0.439s

That means the test isn't covering that the blackhole implementation is more efficient than a non-blackhole implementation.

Some suggests on how the test can actually validate it:

Before the Blackhole call, if you introspect, you should see that there's a pending message exchange
https://godoc.org/github.com/uber/tchannel-go#ExchangeSetRuntimeState

After the Blackhole call, there should be no exchanges.

Instead of using a short timeout, you could actually:

make a call with a huge timeout (e.g., 1 hour), so we know the exchange isn't being cleared due to timeout

after the Blackhole command, we can cancel the client context, so the client returns

prashantv · 2018-02-23T19:49:57Z

inbound.go

@@ -361,6 +361,12 @@ func (response *InboundCallResponse) SetApplicationError() error {
 	return nil
 }

+// Blackhole indicates that there should be no response and provides


more context here,

// Blackhole indicates no response will be sent, and cleans up any resources associated with this request. This allows for services to trigger a timeout in clients without holding on to any goroutines on the server.

This enables us to blackhole requests and release all long-lived go routines. This needs more testing, but I've confirmed with pprof + YARPC that the only goroutines left running are per connection. Relates to yarpc/yarpc-go/#1436.

peats-bond · 2018-03-02T23:32:01Z

Updated with your suggestions @prashantv. Since the message exchange cleanup happens in a separate goroutine, I still found the need to add time.Sleep to give time for clean up. Tests pass with a 1ms timeout, but I upped it to 10ms to reduce potential flakiness.

prashantv · 2018-03-23T18:25:28Z

inbound_test.go

+			inbound.Response().Blackhole()
+
+			// give time for exchange to cleanup
+			time.Sleep(10 * time.Millisecond)


instead of sleep, take a look at testutils.WaitFor -- it's a lot less flaky for testing since it'll sleep a small amount initially, check whether what we're waiting for is true, if not, sleep a little longer, and repeat.

The issue with a single time.Sleep is that it could finish in 1ms, in which case you're wasting 9ms, or it could take 11ms, in which case the test becomes flaky and fails. The only way to avoid flakiness is to use a huge value like a second or more, but then you're likely wasting 900ms for each run just doing nothing.

WaitFor is a compromise that lets you sleep (since some things can't be tested without waiting for some background work to be scheduled), without making the test slower or flaky.

prashantv · 2018-03-23T18:26:41Z

inbound_test.go

+		defer clientCh.Close()
+
+		_, _, _, err := raw.Call(ctx, clientCh, server.HostPort(), serviceName, handlerName, nil, nil)
+		require.Error(t, err, "expected to timeout")


I think you're actually expecting a cancelled error, since you call cancel.

Can you also add a assert.Equal(t, ErrCodeCanceled, GetSystemErrorCode(err)) (or similar) to make sure we're getting an error for the right reason.

prashantv

Pleasantly surprised at how small this change turned out to be!

prashantv · 2018-03-23T20:39:00Z

inbound_test.go

+		require.Error(t, err, "expected call error")
+
+		errCode := GetSystemErrorCode(err)
+		assert.Equal(t, ErrCodeCancelled, errCode, "expected cancelled error code, got: %q", errCode)


no need for "got %v", assert will print the value it expected and what it got. can just make error message something like "got unexpected error code"

As discussed in person, this would output the byte value (eg. 0x02) since SystemErrCode is an alias of byte. I'll update with a comment to indicate why I wrote it out this way.

peats-bond requested review from kriskowal and prashantv February 16, 2018 20:23

peats-bond mentioned this pull request Feb 16, 2018

Black-hole TChannel requests on resource exhausted errors yarpc/yarpc-go#1436

Merged

prashantv reviewed Feb 16, 2018

View reviewed changes

peats-bond changed the title ~~[WIP] Allow blackholing requests~~ Allow blackholing requests Feb 22, 2018

HelloGrayson approved these changes Feb 23, 2018

View reviewed changes

kriskowal approved these changes Feb 23, 2018

View reviewed changes

prashantv suggested changes Feb 23, 2018

View reviewed changes

peats-bond added 4 commits March 2, 2018 15:17

Allow blackholing requests

de94094

This enables us to blackhole requests and release all long-lived go routines. This needs more testing, but I've confirmed with pprof + YARPC that the only goroutines left running are per connection. Relates to yarpc/yarpc-go/#1436.

replace channel with call to response.cancel

6a61374

cancelation to cancellation

974fbb0

address comments

46248c9

peats-bond force-pushed the apb/blackhole branch from c4f282a to 46248c9 Compare March 2, 2018 23:25

prashantv reviewed Mar 23, 2018

View reviewed changes

use waitfor, assert error code

5f0402d

prashantv approved these changes Mar 23, 2018

View reviewed changes

explain ercode msg

f4db7cc

prashantv merged commit 0cf15e0 into dev Mar 23, 2018

prashantv deleted the apb/blackhole branch March 23, 2018 21:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow blackholing requests #681

Allow blackholing requests #681

peats-bond commented Feb 16, 2018 •

edited

Loading

codecov bot commented Feb 16, 2018 •

edited

Loading

prashantv left a comment

peats-bond commented Feb 22, 2018

HelloGrayson left a comment

HelloGrayson Feb 23, 2018

kriskowal Feb 23, 2018

HelloGrayson Feb 23, 2018

prashantv Feb 23, 2018

prashantv Feb 23, 2018

peats-bond Mar 2, 2018

prashantv Feb 23, 2018

prashantv Feb 23, 2018

prashantv Feb 23, 2018

peats-bond commented Mar 2, 2018

prashantv Mar 23, 2018

prashantv Mar 23, 2018

prashantv left a comment

prashantv Mar 23, 2018

peats-bond Mar 23, 2018

Allow blackholing requests #681

Allow blackholing requests #681

Conversation

peats-bond commented Feb 16, 2018 • edited Loading

codecov bot commented Feb 16, 2018 • edited Loading

Codecov Report

prashantv left a comment

Choose a reason for hiding this comment

peats-bond commented Feb 22, 2018

HelloGrayson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peats-bond commented Mar 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prashantv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peats-bond commented Feb 16, 2018 •

edited

Loading

codecov bot commented Feb 16, 2018 •

edited

Loading