Web Socket Endpoints for querying in Pinot Broker

[eaugene]

Currently, we have REST API endpoints in pinot broker for querying. However, adding a WebSocket endpoint helps with having low latency & high throughput queries once the connection is established between the pinot broker and the client service.

Use Case:
Using WebSocket to connect to a Pinot table that is being populated from a Kafka topic / some streaming source. By not directly connecting to the Kafka topic, we can take advantage of data manipulation and efficient filtering, and aggregation features in Pinot. This approach provides more control and flexibility over the data flow and analysis process.

label: feature

cc : @atris @kishoreg @Aravind-Suresh

Related Slack Discussion : https://apache-pinot.slack.com/archives/CDRCA57FC/p1687701635253309

[kishoreg]

thanks for creating this proposal. Can we add some additional details..

• will this be a standing query..
• will the connection return something only when there is a change in response?
• or is it periodic
• is it going to incremental? That return only the changes from previous response? For e.g it's a filter query and the client is only interested in new rows that match the filter criteria.

[eaugene]

@kishoreg Thanks for the questions.

1. The initial version of WebSocket Endpoint won't be for the standing queries. Web Socket would serve as one additional interface ( than existing REST API way) for an external system to connect and query in Pinot, here we save the connection establishment time for each HTTP request . The client is expected to give any queries as we currently have with REST API.
2. The additional use cases of making it support standing queries to period response/return after a change in data ( with a periodic check at pinot ) can be added once the underlying web socket interface is there. This would be a nice feature to support at Pinot as it removes the work at the client system ( particularly data viz. applications ) to periodically call Pinot.
3. For incremental results bounded to standing queries, we can this support once point 2 is made. Here we can utilise the time field of real-time data flowing in & a WebSocket connection at the pinot broker end can maintain the state of the last timestamp it served.

@atris Please add if I am missing any points.

[kishoreg]

Like the phased approach.