Expose metadata for table freshness through Pinot's query response

[dang-stripe]

We want to be able to tell how fresh the overall realtime table is as part of Pinot's query response.

minConsumingFreshnessTimeMs exists on the query response today but seems to have some limitations:

• If consuming segments are pruned or not queried, the timestamp may not represent the overall freshness of the table
• If the upstream topic data is sparse, the metric may not reflect how caught up the table actually is

cc @Jackie-Jiang

[Jackie-Jiang]

I can see multiple potential solutions:

• Server maintaining freshness status, and put that in the query response when it is queried
  • Limitation: We can only gather the freshness status for servers queried. If other server is lagging, this information won't be gathered
• Broker periodically ask servers for the freshness status, and returns it along with the queries
  • Limitation: Freshness status won't be absolute accurate because it is fetched periodically
• Provide a Rest API on controller to gather freshness status from servers
  • Limitation: Freshness status won't be returned along with the query response. User needs to call the Rest API in order to get it.

Let me know which solution fits your requirement the best