[SPARK-27021][CORE] Cleanup of Netty event loop group for shuffle chunk fetch requests

[attilapiros]

What changes were proposed in this pull request?

Creating an Netty EventLoopGroup leads to creating a new Thread pool for handling the events. For stopping the threads of the pool the event loop group should be shut down which is properly done for transport servers and clients by calling for example the shutdownGracefully() method (for details see the close() method of TransportClientFactory and TransportServer). But there is a separate event loop group for shuffle chunk fetch requests which is in pipeline for handling fetch request (shared between the client and server) and owned by the TransportContext and this was never shut down.

How was this patch tested?

With existing unittest.

This leak is in the production system too but its effect is spiking in the unittest.

Checking the core unittest logs before the PR:

$ grep "LEAK IN SUITE" unit-tests.log | grep -o shuffle-chunk-fetch-handler | wc -l
381

And after the PR without whitelisting in thread audit and with an extra await after the
chunkFetchWorkers.shutdownGracefully():

$ grep "LEAK IN SUITE" unit-tests.log | grep -o shuffle-chunk-fetch-handler | wc -l
0

[SparkQA]

Test build #102920 has finished for PR 23930 at commit 3f3567d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• public class TransportContext implements Closeable

[vanzin]

looks good pending tests.

[SparkQA]

Test build #103053 has finished for PR 23930 at commit 3b469d7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Merging to master.