[Feature] Support Pre-Aggregation in Data Nodes

[hanahmily]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently, in BanyanDB, the liaison node acts as the aggregator, collecting raw data from data nodes and applying aggregation functions to produce the final result. This approach has significant drawbacks:

1. High Bandwidth Consumption: Transferring large volumes of raw data from data nodes to the liaison node consumes excessive network bandwidth.
2. Pressure on Liaison Node: The liaison node bears a heavy computational load due to aggregating raw data from multiple data nodes, which can become a bottleneck as the system scales.

To address these issues, we propose adding support for pre-aggregation in data nodes. With pre-aggregation, data nodes would perform local aggregation on their data before transferring the intermediate results to the liaison node. The liaison node would then only need to perform a final aggregation on these intermediate results, significantly reducing both bandwidth usage and computational pressure on the liaison node.

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct