MSE Brokers Cold Start Errors

[satwik-pachigolla]

Long initial class loading and networking times on the query path, for example Calcite class loading during compilation, can lead to errors if the first MSE queries received on the broker have tight timeout constraints (~1-2 seconds).

We can reliably reproduce these Timed out while planning query errors on the first few MSE queries, all with 1s timeouts, received on broker startup:

ERROR [PinotClientRequest] [jersey-server-managed-async-executor-9:210] Query processing exceptions: {400=BrokerTimeoutError: Timed out while planning query}