[data streams] Track high watermark offsets

[piochelepiotr]

What does this PR do?

Track high watermark offsets along with produce and commit offsets.
This information can be used to determine Kafka lag of consumers.
So we can now get the Kafka lag by only instrumenting the consumer service, with no instrumentation on the producer side.
This also makes the Kafka lag more reliable, since we don't rely on stitching commit & produce offsets later on, but are doing it directly in the application.

Motivation

The main motivation is improving the accuracy and reliability of the data_streams.kafka.lag_seconds metric.
Right now, the metric has edge cases in which it doesn't work great. For example when multiple Kafka topics have the same name.

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

For Datadog employees:

• If this PR touches code that handles credentials of any kind, such as Datadog API keys, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.

Unsure? Have a question? Request a review!

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-01-25 16:23:41

Comparing candidate commit cab3120 in PR branch piotr-wolski/test-high-watermark with baseline commit c03c56b in branch main.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 38 metrics, 2 unstable metrics.

scenario:BenchmarkTracerAddSpans-24

• 🟩 execution_time [-147.693ns; -111.107ns] or [-3.578%; -2.691%]

[dianashevchenko]

Hi, please fill out the PR description. We rely on that information during release process 🙌