tracer: pre-size payload buffer to optimize memory allocations

[ahmed-mez]

What does this PR do?

Pre-size the payload buffer before encoding payloads.

Motivation

Performance improvements.

goos: darwin
goarch: arm64
pkg: gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer
                        │   old.txt   │              new.txt               │
                        │   sec/op    │   sec/op     vs base               │
BigTraces/Big_traces-10   115.1m ± 1%   114.0m ± 1%  -0.99% (p=0.019 n=10)

                        │     old.txt      │              new.txt               │
                        │ avgHeapInUse(Mb) │ avgHeapInUse(Mb)  vs base          │
BigTraces/Big_traces-10         78.71 ± 4%        77.80 ± 10%  ~ (p=0.436 n=10)

                        │   old.txt    │               new.txt                │
                        │     B/op     │     B/op      vs base                │
BigTraces/Big_traces-10   326.6Mi ± 0%   286.8Mi ± 0%  -12.18% (p=0.000 n=10)

                        │   old.txt   │              new.txt               │
                        │  allocs/op  │  allocs/op   vs base               │
BigTraces/Big_traces-10   2.983M ± 0%   2.981M ± 0%  -0.05% (p=0.043 n=10)

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.

For Datadog employees:

• If this PR touches code that handles credentials of any kind, such as Datadog API keys, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.

Unsure? Have a question? Request a review!

[pr-commenter]

Benchmarks

Benchmark execution time: 2023-10-31 16:18:09

Comparing candidate commit 9cd7c86 in PR branch ahmed/pre-size-payload with baseline commit b3f6536 in branch main.

Found 4 performance improvements and 2 performance regressions! Performance is the same for 34 metrics, 1 unstable metrics.

scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-24

• 🟩 allocated_mem [-407 bytes; -399 bytes] or [-10.385%; -10.170%]
• 🟩 execution_time [-190.602ns; -139.598ns] or [-3.743%; -2.742%]

scenario:BenchmarkOTelApiWithCustomTags/otel_api-24

• 🟩 allocated_mem [-404 bytes; -394 bytes] or [-9.703%; -9.450%]

scenario:BenchmarkPartialFlushing/Disabled-24

• 🟩 allocated_mem [-39.756MB; -39.653MB] or [-11.586%; -11.555%]

scenario:BenchmarkPartialFlushing/Enabled-24

• 🟥 allocated_mem [+24.132MB; +27.845MB] or [+6.726%; +7.761%]
• 🟥 avgHeapInUse(Mb) [+11.601MB; +15.793MB] or [+23.251%; +31.652%]

[felixge]

Thanks for looking into this! How do y'all feel about the scenario:BenchmarkPartialFlushing/Enabled-24 regression? What's causing it? Can we have the cake and eat it too here?

[ajgajg1134]

I took a look at this locally and it appears this is likely because the benchmark for partial flushing is using a somewhat small number (500) to trigger a flush, so the new line here where we expand the payload buffer is causing slightly more memory pressure than before. However compared to with partial flushing OFF there are still significant memory benefits (ran this locally to confirm), given that there's still an improvement and partial flushing is pretty niche I would say this is safe to merge!

(And if we ran the benchmark with a larger partial flush span number like 1000 or 2000 this memory pressure would be even further reduced)