Client library tests taking too long to complete

[hannahrogers-google]

The client library tests are taking a long time to complete - a Kokoro Integration Test timed out when running the presubmit checks before submitting a PR. The tests did not always take this long to complete, we should look into this.

[hannahrogers-google]

I manually ran the tests, and it took 5 minutes, 27 seconds for the test suite to complete. Most of the tests took less than a second, while the PublisherImplTest took over 5 minutes.

[hannahrogers-google]

In the PublisherImplTest, after calling shutdown on the publisher, we wait up to a minute for the publisher to terminate. It looks like we always wait the full 60 seconds and the publisher never terminates, which is why the test is taking so long.

[hannahrogers-google]

It looks like the executor is not being shutdown properly - We are using a FakeScheduledExecutor for most of the tests, and it looks like we have outstanding scheduled fixed executor calls that never complete because we do not 'advance time'. We actually do not schedule anything at a fixed rate on the publisher executor inside our own client library - this is likely coming form the gax/grpc layer.

[hannahrogers-google]

We are also not handling channel shutdowns properly - the following is being logged during the publisher tests:

SEVERE: ~* Channel ManagedChannelImpl{logId=340, target=directaddress:///test-server} was not shutdown properly!!! *~
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.

[hannahrogers-google]

Gax implements a Watchdog (https://github.com/googleapis/gax-java/blob/master/gax/src/main/java/com/google/api/gax/rpc/Watchdog.java) that periodically garbage collects idle streams. The check is scheduled at a fixed rate every 10 seconds.

In our publisher test, after we shutdown the executor we should advance time by 10 seconds to ensure all scheduled callables complete.

[hannahrogers-google]

The pending fix reduces test time from >5 minutes, to <30 seconds.