Flaky test TestServerGracefulExit

[yurishkuro]

=== RUN   TestServerGracefulExit
    server_test.go:677: 
        	Error Trace:	server_test.go:677
        	Error:      	Should be true
        	Test:       	TestServerGracefulExit
        	Messages:   	Error log found on server exit: {{error 2021-05-21 19:22:10.814114204 +0000 UTC m=+2.124395084  Could not start HTTP server undefined } [{error 26 0  mux: server closed}]}
--- FAIL: TestServerGracefulExit (1.00s)

Source: ./cmd/query/app/server_test.go:651:func TestServerGracefulExit(t *testing.T)

[albertteoh]

Why is this flaky error showing up now?

• The mux: server closed error was introduced in cmux PR #69
• cmux PR #69 was brought into our repo from the cmux version bump from 0.1.4 -> 0.1.5 by dependabot 4 days ago, hence why this flaky test has only manifested itself starting from the last few days.
• After cmux PR #69, the Accept call returns mux: server closed in our TestServerGracefulExit test.

Why is it flaky?

• I think the flakiness comes down to two causes:
  • The call to Close will randomly select either case. The first case returning ErrListenerClosed is considered normal behaviour and won't fail the test. The second case returning ErrServerClosed is not recognized as normal behaviour and so propagates the error, failing the test.
  • The other reason for flakiness is a race condition between when the error is logged and when it is "observed" in the test.

Why does this only relate to the HTTP server?

• The grpcServer maintains a "connection closed" state, preventing it from propagating the error.
• cmux server will return "use of closed network connection" when the connection is closed, which is considered normal behaviour.