fix(buffers): Optimize buffer usage metric tracking

[bruceg]

Summary

The buffer usage metrics, in particular the value of the current utilization levels, were tracked using an atomic u64 which was updated using a fetch_update mechanism in order to protect against underflowing. This same mechanism was extended to all of the atomics as well for consistency. The problem with that is that fetch_udpate internally uses a loop around a compare-and-exchange operation which is very expensive, particularly when contended. In comparison, the base fetch_add is typically a single locked instruction which completes in many fewer cycles.

This change returns these atomics to only ever use fetch_add and then calculate the current level by subtracting the count of increments from the count of decrements.

Vector configuration

N/A

How did you test this PR?

Unit tests

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

#24058

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

[bruceg]

Regression test run indicates no change

[pront]

Did we validate if this PR fixes the regression?

Ref #24911 (comment)

[pznamensky]

Thanks for the effort, @bruceg!
I'd be happy to try this out in our production env with real workload if needed.
However I have some problems building proper docker image. So if you could trigger a docker build job, I could use that image to use in our setup.

[pront]

Thanks for the effort, @bruceg! I'd be happy to try this out in our production env with real workload if needed. However I have some problems building proper docker image. So if you could trigger a docker build job, I could use that image to use in our setup.

Hi @pznamensky I kicked off a custom build: https://github.com/vectordotdev/vector/actions/runs/23146056109

You can use those builds once they are published. Looking forward to hearing back from after you test this 🤞

[pznamensky]

@pront, thank you for preparing the images.
Bad news is that in our case Vector from this PR uses CPU on the same level as v0.54.0.
Average CPU usage in our cluster:

So it looks like the original issue might be not in metrics.