[Feature]: Adding a generic virtual resource allocator to orchestrate usage of memory, CPU, and disk, across different components

[tedxu]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

There are several issues caused by poor resource management:

1. In a standalone configuration, multiple system roles, such as Data Nodes and Query Nodes, may compete for memory resources. A critical conflict may end up with OOM error.
2. Within query nodes, a similar problem can arise with disk resources. The concurrent execution of memory-mapped (mmap) operations and the maintenance of disk-based indexes can lead to a race for disk space. If not properly managed, this can lead to system panic due to perceived disk volume exhaustion.

Describe the solution you'd like.

A singleton Allocator is designed to manage and coordinate resource allocations among various system components. Each time a component is poised to execute a resource-intensive operation, it must solicit an allocation from the Allocator. The interaction with the Allocator is facilitated through the following method signature:

Allocator#Allocate(id string, r *Resource) (allocated bool, short *Resource)

If the Allocator decides not to approve the request, it returns allocated=false. Upon receiving this response, the component that made the request should cease further attempts to execute the operation, thereby preventing potential resource contention or exhaustion.

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response