enhancement(buffers, metrics): move data counting to the BufferUsage

[vparfonov]

Summary

This PR refactors the buffer usage metrics system to eliminate the global DashMap (BUFFER_COUNTERS) and centralize the state management logic. The new design localizes all gauge state within the BufferUsage instance and its dedicated reporting task.

The global static DashMap and the associated update_buffer_gauge function have been completely removed.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Related: fix(buffers): buffer_id handling in buffer usage metrics reporting #23507

[bruceg]

I am finding this hard to review at anything more than a surface level because it seems to be changing a lot of aspects in this area at once with little understanding of how it got here. Since this is a very subtle area that has had serious issues in the past (and arguably currently), I would prefer seeing an approach that was much more incremental, where each change preserves the desired behavior while making a single change. That may end up with the same result as what we see here, but it would help understanding how it got to this point by breaking down the steps in separate commits or even separate PRs.

[vparfonov]

@bruceg Please take a look, i try to break down the changes in separate commits

[bruceg]

@vparfonov I followed up on the previous work on this area in #23542 and I would appreciate your review on it as well. As mentioned, I tried to work very incrementally to avoid changing anything accidentally, but the result doesn't entirely make sense to me.

[pront]

@vparfonov thank you for your efforts in this area and for contributing to Vector. Much appreciated!

[vparfonov]

@vparfonov thank you for your efforts in this area and for contributing to Vector. Much appreciated!

Thank you for your help!

[vparfonov]

Close it in favor of #23561