1.2.0: TestServerInterceptorSuite failures

[onlyjob]

As reported in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=902466
TestServerInterceptorSuite sometimes fails as follows:

--- FAIL: TestServerInterceptorSuite (0.73s)
    --- PASS: TestServerInterceptorSuite/TestRegisterPresetsStuff (0.18s)
    --- FAIL: TestServerInterceptorSuite/TestStreamingIncrementsHandled (0.08s)
    	server_test.go:219: 
    			Error Trace:	server_test.go:219
    			Error:      	Not equal: 
    			            	expected: 1
    			            	actual  : 0
    			Test:       	TestServerInterceptorSuite/TestStreamingIncrementsHandled
    			Messages:   	grpc_server_handled_total should be incremented for PingList FailedPrecondition
    --- FAIL: TestServerInterceptorSuite/TestStreamingIncrementsHistograms (0.11s)
    	server_test.go:194: 
    			Error Trace:	server_test.go:194
    			Error:      	Not equal: 
    			            	expected: 4
    			            	actual  : 3
    			Test:       	TestServerInterceptorSuite/TestStreamingIncrementsHistograms
    			Messages:   	grpc_server_handling_seconds_count should be incremented for PingList FailedPrecondition
    --- PASS: TestServerInterceptorSuite/TestStreamingIncrementsMessageCounts (0.08s)
    --- PASS: TestServerInterceptorSuite/TestStreamingIncrementsStarted (0.03s)
    --- PASS: TestServerInterceptorSuite/TestUnaryIncrementsHandled (0.07s)
    --- PASS: TestServerInterceptorSuite/TestUnaryIncrementsHistograms (0.06s)
    --- PASS: TestServerInterceptorSuite/TestUnaryIncrementsStarted (0.07s)
	server_test.go:87: stopped grpc.Server at: 127.0.0.1:33609
FAIL

https://tests.reproducible-builds.org/debian/rb-pkg/buster/amd64/golang-github-grpc-ecosystem-go-grpc-prometheus.html

[onlyjob]

Problem is easy enough to reproduce by running tests repeatedly. Usually it only takes 2...15 tries before this test fail.

I could not reproduce after applying the following patch:

--- a/server_test.go
+++ b/server_test.go
@@ -77,9 +77,9 @@
 }

 func (s *ServerInterceptorTestSuite) SetupTest() {
        // Make all RPC calls last at most 2 sec, meaning all async issues or deadlock will not kill tests.
-       s.ctx, _ = context.WithTimeout(context.TODO(), 2*time.Second)
+       s.ctx, _ = context.WithTimeout(context.TODO(), 11*time.Second)
 }

 func (s *ServerInterceptorTestSuite) TearDownSuite() {
        if s.serverListener != nil {

[brancz]

I'm fine with bumping the timeout in the tests as there are actual servers being started here, so there could be various things that could cause some delay. What do you think @Bplotka?

[onlyjob]

We've got some feedback from autobuilders and unfortunately changing this timeout turned out to be ineffective... Even though adjusting timeout changed outcome in my environment, on autobuilders this test still fails reliably and reproducibly... No ideas why... :(

[brancz]

In that case and since it’s relatively easy to reproduce I think we should record cpu and mutex/lock profiles and see if there is anything we can do. (If it’s syscalls then there is little we can so, but it’s worth trying I think)

[bwplotka]

Yea, we cannot bump timeouts indefinitely. And CI tends to be slow. We need some other solution to this.

[piotrkowalczuk]

Alternative approach would be to add -count=20 into test_all.sh

[bwplotka]

How running same test 20 time halps for this issue?