New transform: log to trace

[bruceg]

With the introduction of the trace data type, Vector has the opportunity to generate traces from other data sources. In particular, a suggestion that was discussed today was the collection of multiple logs into one trace that described the flow of events related to some larger task. Vector should have such a transform, similar to the reduce transform but able to emit traces.

[kamalmarhubi]

A more basic version of this that I'd love to have: simply generating spans from logs.

If my logs have trace_id and span_id on them, then I can just spit out spans and have some other part of the pipeline (possibly within vector, possibly not) group them by span for tail based sampling, or stuff like the sql_query_count example in #12710.

Specific use cases for this simpler version:

• generating spans from GCP load balancer logs; the LogEntry object includes trace info in a GCP-specific format*—a bit of VRL could pluck those out
• if you instrument sql clients to insert trace context in a SQL comment, you can generate server spans from database server logs with a bit of remap to parse out the trace context from the comment

This might not even by the same transform, but it's a huge missing piece of our nascent pipeline. For load balancer logs, we currently have a cloud function doing the conversion and exporting to our opentelemetry collector. It's a lot more operational overhead than I'd like, and a simple way to turn logs into spans would tidy it all away.

---

‌* .spanId as hex, .trace as projects/<project_id>/traces/<trace_id_hex>

[kamalmarhubi]

@bruceg do you think I should create a new issue for log-to-span?

[bruceg]

Sure, go for it, it could be discussed in more detail there if needed.