ddtrace/tracer: create debug mode for old, open spans

[hannahkm]

What does this PR do?

This PR introduces a debugging mode that the user can configure for finding and logging old, unfinished spans. Traces/Spans older than some configured X amount of time will be printed to the debug logs. When the tracer stops, it will also print all remaining open spans. For example, the debug mode can output log lines like the following:

2023/08/14 16:55:45 Datadog Tracer v1.55.0-dev WARN: 4 abandoned spans:
2023/08/14 17:15:38 Datadog Tracer v1.55.0-dev WARN: [name: newSpan, span_id: 7578134955455927291, trace_id: 7578134955455927291, age: 84 sec],[name: http.request, span_id: 5293323209049443106, trace_id: 5293323209049443106, age: 84 sec],[name: newSpan, span_id: 284037345165039393, trace_id: 284037345165039393, age: 84 sec],[name: http.request, span_id: 1770903217116959580, trace_id: 1770903217116959580, age: 84 sec],

Motivation

Traces with unfinished spans may never finish but never error, which causes confusion when data goes missing.

Fixes #2064

Describe how to test/QA your changes

New and old unit tests should pass. For manual testing, turn on the debug mode with WithDebugSpansMode(true) or by setting enviornment variable DD_TRACE_DEBUG_OPEN_SPANS=true. Create spans older than the configured amount of time (default value: 1 second, configure using environment variable DD_TRACE_OPEN_SPAN_TIMEOUT) and check that they are printed out in the tracer logs.

Call defer tracer.PrintOpenSpans() to check that a list of open spans is logged when the application ends.

Reviewer's Checklist

• Changed code has unit tests for its functionality.
• If this interacts with the agent in a new way, a system test has been added.

[pr-commenter]

Benchmarks

Benchmark execution time: 2023-08-18 16:32:29

Comparing candidate commit 7c0f935 in PR branch hannahkm/debug-open-spans with baseline commit e9060fc in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 39 metrics, 2 unstable metrics.

[katiehockman]

Thanks Hannah! Some first comments. I haven't finished reviewing it yet, but wanted to give a chance to look over these first.
Mostly, I'd like to understand the context around the data structure decision. Why a linked list of generic linked lists, instead of something similar (e.g. a slice of span pointers)?

[hannahkm]

Mostly, I'd like to understand the context around the data structure decision. Why a linked list of generic linked lists, instead of something similar (e.g. a slice of span pointers)?

Re the above: The main concern I had with creating the new data structure in the tracer was the overhead we'd introduce from adding/removing spans as they are started/finished. A linked list is great for this purpose (especially when removing spans from the middle of the structure), and the outer linked list is for the purposes of bucketing spans together. Spans that have a similar start time are placed in the same "bucket" so the new goroutine hopefully has less to scan through when checking for abandoned spans!

[ajgajg1134]

Looks pretty good! A few questions / changes!

[ajgajg1134]

Thanks for being on the team!