Add sampled flag to Span type

[kbrockhoff]

Lack of a sampled flag in the Span type results in ambiguity. It implies that non-sampled traces are dropped at the instrumentation level and not forwarded through the system. If using tail-sampling decisions, this means all traces need to be marked sampled up front. In addition, it leaves no flexibility for different routing of traces. For example, routing all traces to a logging exporter and only sampled ones to a full-scale backend.

In addition, other protocols such as Zipkin have a sampled flag. The lack of this data makes it impossible to preserve full fidelity when converting between protocols.

[Oberon00]

Isn't an unsampled but exported span an oxymoron? How could this happen in practice?

[kbrockhoff]

There is work going on to do tail-based sampling in the collector. For this to work, all spans need to be exported to the collector.

[bogdandrutu]

I am +1 on this

[anuraaga]

@kbrockhoff Do you mind describing more concretely the use case you have in mind? I think with pure tail sampling, we would expect the SDK to do 100% sampling and that wouldn't need to be exported. Is the goal to support multiple sampling decisions, e.g., not sampled for trace, but sampled for logs or for secondary sampling of a subset of the trace? I guess instead of a sampling flag, we'd want a list of SamplingDecision, each can be namespaced to a backend or system by a user.

Also I think this would need to be propagated too for certain types of secondary sampling?

[kbrockhoff]

There are several use cases for having a sampled flag.

1. Using tail-based sampling where the team controlling the tail-based sampling and monitoring does not have control over upfront sampling configuration. This can occur in a supply chain with multiple organizations or even a siloed single organization. In this use case having a list of SamplingDecision as suggested by @anuraaga is an even better idea.

2. Using protobuf as a general storage mechanism for tracing data. An example would be compact storage of a test data set.

In either case, adequate information can potentially be stored in the TraceState field but would then need to be parsed. I can be talked into either this approach or the SamplingDecision approach.

[andrewhsu]

@arminru from the bulk move of P3 required-for-ga issues to after-ga earlier today during the issue triage mtg with TC, i noticed this issue was skipped. should the labels for tracking this issue be updated to something else?

[arminru]

This can be added after-ga, if still desired, yes.
@bogdandrutu you were +1 on ths, WDYT?