Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propose sampling API changes #24

Merged
merged 13 commits into from Aug 26, 2019
@@ -0,0 +1,337 @@
# Sampling API

*Status: proposed*

## TL;DR
This section tries to summarize all the changes proposed in this RFC:
1. Move the `Sampler` interface from the API to SDK package. Apply some minor changes to the
`Sampler` API.
1. Add a new `SamplerHint` concept to the API package.
1. Add capability to record `Attributes` that can be used for sampling decision during the `Span`
This conversation was marked as resolved by bogdandrutu

This comment has been minimized.

Copy link
@jmacd

jmacd Aug 19, 2019

Contributor

I don't understand why we would call these synthetic things "Attributes". They're not given by the programmer, they're generated by the SDK. Can we record these key:values in a way that marks them clearly as having been injected by the implementation? They're not the user's attributes.

This comment has been minimized.

Copy link
@lizthegrey

lizthegrey Aug 19, 2019

Member

They are the user's attributes, and can/should be supplied by the programmer before being used for sampling.

This comment has been minimized.

Copy link
@jmacd

jmacd Aug 19, 2019

Contributor

Oh, that wasn't clear to me. I thought we already had such a requirement, since the OpenTracing start method includes a list of attributes. I would state this simply as a requirement that the user be allowed to specify attributes at start time, practically unrelated to sampling. Thanks.

creation time.
1. Remove `addLink` APIs from the `Span` interface, and allow recording links only during the span
This conversation was marked as resolved by bogdandrutu

This comment has been minimized.

Copy link
@jmacd

jmacd Aug 19, 2019

Contributor

Is this required? If I wanted to record my children in the parent span, a very natural way to do that would be via adding a link. In the case of attributes, we now have an API that lets you provide them at start time or during the span's lifecycle. Could we do the same with links? This change doesn't feel like a requirement, considering how attributes are being treated.

This comment has been minimized.

Copy link
@reyang

reyang Aug 19, 2019

Contributor

I think we need to revisit links. Currently it has a lot of unclear stuff:

  1. What's the maximum allowed number of links?
  2. Do we in general keep links to all the children in a logical parent span, or we put logical parent as a link in the child span? (or both?)
  3. Do we allow links to be added postmortem (after both source and target spans are exported)?

We should track this in a separate PR instead of blocking this one.

This comment has been minimized.

Copy link
@bogdandrutu

bogdandrutu Aug 19, 2019

Author Member

@jmacd I applied a simple rule when I suggested this: adding a new API is considered backwards compatible so delaying this decision seems to be fine. this is why I proposed this :)

Also if we do apply the links later is going to be a discussion if they can influence the sampling/recording decision.

This comment has been minimized.

Copy link
@bogdandrutu

bogdandrutu Aug 22, 2019

Author Member

Closing this conversation, please re-open if needed.

construction time.

## Motivation

Different users of OpenTelemetry, ranging from library developers, packaged infrastructure binary
developers, application developers, operators, and telemetry system owners, have separate use cases
for OpenTelemetry that have gotten muddled in the design of the original Sampling API. Thus, we need
to clarify what APIs each should be able to depend upon, and how they will configure sampling and
OpenTelemetry according to their needs.
This conversation was marked as resolved by bogdandrutu

This comment has been minimized.

Copy link
@c24t

c24t Aug 16, 2019

The personas are helpful to motivate the changes in this RFC. It would be great to have these in another doc (in specs or elsewhere) separate from the sampler changes.

This comment has been minimized.

Copy link
@bogdandrutu

bogdandrutu Aug 16, 2019

Author Member

I will do that and use this as a starting point.


```
+----------+ +-----------+
grpc | Library | | |
Django | Devs +---------->| OTel API |
Express | | +------>| |
+----------+ | +--->+-----------+ +---------+
| | ^ | OTel |
| | | +->| Proxy +---+
| | | | | | |
+----------+ | | +-----+-----+------------+ | +---------+ |
| | | | | | OTel Wire | | |
Hbase | Infra | | | | | Export |+-+ v
Envoy | Binary +---+ | | OTel | | | +----v-----+
| Devs | | | SDK +------------+ | | |
+----------+---------->| | | +---------->| Backend |
+------>| | Custom | +---------->| |
| | | | Export | | +----------+
+----------+ | | | | |+-+ ^
| +---+ | +-----------+------------+ |
| App +------+ ^ ^ |
| Devs + | | +------------+-+
| | | | | |
+----------+ +---+----+ +----------+ Telemetry |
| SRE | | Owner |
| | | |
+--------+ +--------------+
Lightstep
Honeycomb
```
## Explanation

We outline five different use cases (who may be overlapping sets of people), and how they should
interact with OpenTelemetry:

### Library developer
Examples: gRPC, Express, Django developers.

* They must only depend upon the OpenTelemetry API and not upon the SDK.
This conversation was marked as resolved by bogdandrutu

This comment has been minimized.

Copy link
@songy23

songy23 Aug 19, 2019

Member

I think it's fine to have tests depend on SDK.

This comment has been minimized.

Copy link
@bogdandrutu

bogdandrutu Aug 20, 2019

Author Member

Done.

* For testing only they may depend on the SDK with InMemoryExporter.
* They are shipping source code that will be linked into others' applications.
* They have no explicit runtime control over the application.
* They know some signal about what traces may be interesting (e.g. unusual control plane requests)
or uninteresting (e.g. health-checks), but have to write fully generically.

**Solution:**

* On the start Span operation, the OpenTelemetry API will allow marking a span with one of three
choices for the [SamplingHint](#samplinghint).

### Infrastructure package/binary developer
Examples: HBase, Envoy developers.

* They are shipping self-contained binaries that may accept YAML or similar run-time configuration,
but are not expected to support extensibility/plugins beyond the default OpenTelemetry SDK,
OpenTelemetry SDKTracer, and OpenTelemetry wire format exporter.
* They may have their own recommendations for sampling rates, but don't run the binaries in
production, only provide packaged binaries. So their sampling rate configs, and sampling strategies
need to a finite "built in" set from OpenTelemetry's SDK.
* They need to deal with upstream sampling decisions made by services calling them.

**Solution:**
* Allow different sampling strategies by default in OpenTelemetry SDK, all configurable easily via
YAML or feature flags. See [default samplers](#default-samplers).

### Application developer
These are the folks we've been thinking the most about for OpenTelemetry in general.

* They have full control over the OpenTelemetry implementation or SDK configuration. When using the
SDK they can configure custom exporters, custom code/samplers, etc.
* They can choose to implement runtime configuration via a variety of means (e.g. baking in feature
flags, reading YAML files, etc.), or even configure the library in code.
* They make heavy usage of OpenTelemetry for instrumenting application-specific behavior, beyond
what may be provided by the libraries they use such as gRPC, Django, etc.

**Solution:**
* Allow application developers to link in custom samplers or write their own when using the
official SDK.
* These might include dynamic per-field sampling to achieve a target rate
(e.g. https://github.com/honeycombio/dynsampler-go)
* Sampling decisions are made within the start Span operation, after attributes relevant to the
span have been added to the Span start operation but before a concrete Span object exists (so that
either a NoOpSpan can be made, or an actual Span instance can be produced depending upon the
sampler's decision).
This conversation was marked as resolved by bogdandrutu

This comment has been minimized.

Copy link
@yurishkuro

yurishkuro Aug 20, 2019

We're currently adding support for ad-hoc sampling rules (configured in the Jaeger backend and pushed to the SDKs). We have specific user requirements where sampling is based on span name and/or tags and neither of those are available at span creation time.

This comment has been minimized.

Copy link
@lizthegrey

lizthegrey Aug 20, 2019

Member

ad-hoc sampling rules seem like they're ultimately a property of the collection exporter/backend rather than of the OTel SDK... e.g. I wouldn't insist on making the OTel SDK exactly replicate what Honeycomb does post-collection...

This comment has been minimized.

Copy link
@bogdandrutu

bogdandrutu Aug 20, 2019

Author Member

@yurishkuro can you clarify if the sampling decision still happens on the client side? If that is true can you clarify when is the limit when that needs to happen. One possibility that I covered in the "when does sampling happen" is that we can delay until the moment the SpanContext is accessed: to create a child, to serialize it, or for example to do log correlation.

This comment has been minimized.

Copy link
@yurishkuro

yurishkuro Aug 21, 2019

Yes, sampling decision still happens in the client.

The limit depends on the type of sampling policies provided to the client. Consider these policies:

  • A: static: sample with probability p=0.1
  • B: if span.name = "/userProfile" then sample with rate limit r=10/sec
  • C: tag based: if tag X = Y, then sample with probability p=0.9

If the tracer is configured with policies [A], then sampling decision is finalized immediately on span creation, due to the nature of the sole policy A which is independent of any later span data.

If the tracer is configured with policies [B, A], then sampling decision can happen on span creation if A says Yes or B says Yes, or after span.setName() (assumption is that the code is only going to call setName once, at least as far as sampling decision is concerned).

If tracer is configured with all 3 policies, then recording of events may never stop if none of the policies match and/or say Yes. It's more expensive but it's the user's choice.

we can delay until the moment the SpanContext is accessed

That was how we initially implemented the handling of late span.SetName() binding. However, there's no particular reason why the sampling on the parent span cannot remain "undecided" even if child spans are being created. Yes, it will result in some spans of the trace not captured, but that's just the nature of the combination of the chosen sampling policies. We're also thinking that we will use a shared sampling state across all spans for the trace in memory, so that some of those sampling policies might fire on a child span and trigger sampling of the parent.

Disclaimer: we just brainstormed these scenarios and will prototype them in the next week or so. There's a chance it might not work at all, but so far it looks this approach is meeting our requirements (which definitely include late sampling decision based on a specific tag).

This comment has been minimized.

Copy link
@lizthegrey

lizthegrey Aug 21, 2019

Member

With respect, "we will prototype 'undecided' states in the next week" feels like "add to v0.0.2 but omit for v0.0.1" given Sep 1 spec freeze. Believe me, I'm all for late-binding sampling decisions, but we need to be explicit about when there is the possibility of that happening, and allowing implementations to act accordingly. We can always add complexity in a methodical fashion, I'm worried about rushing and getting it wrong.

e.g. can you mock this for now by making the sampling decision in SDK 'always keep' (unless you know the answer is 'no') and then deciding down the road in the exporter what you want to transmit onwards?

This comment has been minimized.

Copy link
@yurishkuro

yurishkuro Aug 21, 2019

The solution with late binding of span name alone has been in Jaeger in production for >2yrs, implemented as Bogdan implied - events are recorded until sampling decision is finalized, which happens either on span.SetName() or a child span being created. I don't particularly like it, but we had some Node.js frameworks where span name was only known later. The approach I described above is meant to generalize and improve it.

I don't have a problem with not supporting any form of late binding in v1, but this RFC does provide for it via "late Builder", which feels even worse to me.

This comment has been minimized.

Copy link
@lizthegrey

lizthegrey Aug 21, 2019

Member

Ah, I think I understand. Hm. I often know the attributes "at span creation", but I want the time to compute the attributes to be included in the span duration, if that makes sense? That's why we settled on the builder pattern as it was easier for application programmers to work with...

Is there some middle ground here that allows very-short-delayed binding without requiring it all upfront at span creation and adjusting the times afterward?

This comment has been minimized.

Copy link
@yurishkuro

yurishkuro Aug 21, 2019

I guess the question is how big of a deal it would be if v1 didn't support any form of late binding and punted it on v2. In Jaeger, only Node.js client supports late span name binding that impacts sampling, nobody complained in other languages (it only matters when using sampling policies sensitive to span name).

This comment has been minimized.

Copy link
@bogdandrutu

bogdandrutu Aug 22, 2019

Author Member

I am fine to remove the delayed sampling for the moment. But I will still use as an argument in some of the decisions that we may support this, for example the reason we have a different isRecordingEvents is because we may implement something like isDelayed that you suggested so we don't want people to play with the trace flags to know if they should record or not something on the Span.

* Span.IsRecording() needs to be present to allow costly span attribute/log computation to be
skipped if the span is a NoOp span.

### Application operator
Often the same people as the application developers, but not necessarily

* They care about adjusting sampling rates and strategies to meet operational needs, debugging,
and cost.

**Solution:**
* Use config files or feature flags written by the application developers to control the
application sampling logic.
* Use the config files to configure libraries and infrastructure package behavior.

### Telemetry infrastructure owner
They are the people who provide an implementation for the OpenTelemetry API by using the SDK with
custom `Exporter`s, `Sampler`s, hooks, etc. or by writing a custom implementation, as well as
running the infrastructure for collecting exported traces.

* They care about a variety of things, including efficiency, cost effectiveness, and being able to
gather spans in a way that makes sense for them.

**Solution:**
* Infrastructure owners receive information attached to the span, after sampling hooks have already
been run.

## Internal details
In Dapper based systems (or systems without a deferred sampling decision) all exported spans are
stored to the backend, thus some of these systems usually don't scale to a high volume of traces,
or the cost to store all the Spans may be too high. In order to support this use-case and to
ensure the quality of the data we send, OpenTelemetry needs to natively support sampling with some
requirements:
* Send as many complete traces as possible. Sending just a subset of the spans from a trace is
less useful because in this case the interaction between the spans may miss.
* Allow application operator to configure the sampling frequency.

For new modern systems that need to collect all the Spans and later may or may not do a deferred
sampling decision, OpenTelemetry needs to natively support a way to configure the library to
collect and export all the Spans. This is possible even though OpenTelemetry supports sampling by
setting a default config to always collect all the spans.

### Sampling flags
OpenTelemetry API has two flags/properties:
* `RecordEvents`
* This property is exposed in the `Span` interface (e.g. `Span.isRecordingEvents()`).
* If `true` the current `Span` records tracing events (attributes, events, status, etc.),
otherwise all tracing events are dropped.
* Users can use this property to determine if expensive trace events can be avoided.
* `SampledFlag`
* This flag is propagated via the `TraceOptions` to the child Spans (e.g.
`TraceOptions.isSampled()`). For more details see the w3c definition [here][trace-flags].
* In Dapper based systems this is equivalent to `Span` being `sampled` and exported.

The flag combination `SampledFlag == false` and `RecordEvents == true` means that the current `Span`
does record tracing events, but most likely the child `Span` will not. This combination is
necessary because:
* Allow users to control recording for individual Spans.
* OpenCensus has this to support z-pages, so we need to keep backwards compatibility.

The flag combination `SampledFlag == true` and `RecordEvents == false` can cause gaps in the
distributed trace, and because of this OpenTelemetry API should NOT allow this combination.

It is safe to assume that users of the API should only access the `RecordEvents` property when
instrumenting code and never access `SampledFlag` unless used in context propagators.

### SamplingHint

This comment has been minimized.

Copy link
@yurishkuro

yurishkuro Aug 20, 2019

I suggest adding clarification / examples when the user code would want to provide one of the 4 values.

This is a new concept added in the OpenTelemetry API that allows to suggest sampling hints to the
implementation of the API:
* `NOT_RECORD`
* Suggest to not `RecordEvents = false` and not propagate `SampledFlag = false`.
* `RECORD`
* Suggest `RecordEvents = true` and `SampledFlag = false`.
* `RECORD_AND_PROPAGATE`
* Suggest to `RecordEvents = true` and propagate `SampledFlag = true`.

The default option for the span creation is to not have any suggestion (or suggestion is not
specified). This can be implemented by using `null` as the default option or any language specific
mechanism to achieve the same result.

### Sampler interface
The interface for the Sampler class that is available only in the OpenTelemetry SDK:
* `TraceID`
* `SpanID`
* Parent `SpanContext` if any
* `SamplerHint`
* `Links`
This conversation was marked as resolved by bogdandrutu

This comment has been minimized.

Copy link
@reyang

reyang Aug 17, 2019

Contributor

Does this mean all links should be provided during span construction?

This comment has been minimized.

Copy link
@bogdandrutu

bogdandrutu Aug 17, 2019

Author Member

I think this is a good practice, we should start with this approach and later add support if needed to record links on the Span. I also added a section to motivate why to remove them from the Span.

* Span name
* `SpanKind`
* Initial set of `Attributes` for the `Span` being constructed

It produces as an output called `SamplingResult`:
* A `SamplingDecision` enum [`NOT_RECORD`, `RECORD`, `RECORD_AND_PROPAGATE`].
* A set of span Attributes that will also be added to the `Span`.
* These attributes will be added after the initial set of `Attributes`.
* (under discussion in separate RFC) the SamplingRate float.
This conversation was marked as resolved by bogdandrutu

This comment has been minimized.

Copy link
@songy23

songy23 Aug 19, 2019

Member

How about span name?

This comment has been minimized.

Copy link
@bogdandrutu

bogdandrutu Aug 20, 2019

Author Member

I don't think the Sampler should change the Span name.

This comment has been minimized.

Copy link
@songy23

songy23 Aug 20, 2019

Member

Apologies I commented on the wrong line - I mean the Sample interface may need to accept span names as input parameter in order to allow sampling on particular namespaces. /cc @rghetia brought this up before.

This comment has been minimized.

Copy link
@bogdandrutu

bogdandrutu Aug 20, 2019

Author Member

Good catch, I forgot about that.

Done.


### Default Samplers
These are the default samplers implemented in the OpenTelemetry SDK:
* ALWAYS_ON
* Ignores all values in SamplingHint.
* ALWAYS_OFF
* Ignores all values in SamplingHint.
* ALWAYS_PARENT
* Ignores all values in SamplingHint.
* Trust parent sampling decision (trusting and propagating parent `SampledFlag`).
* For root Spans (no parent available) returns `NOT_RECORD`.
* Probability
* Allows users to configure to ignore or not the SamplingHint for every value different than
`UNSPECIFIED`.
* Default is to NOT ignore `NOT_RECORD` and `RECORD_AND_PROPAGATE` but ignores `RECORD`.
* Allows users to configure to ignore the parent `SampledFlag`.
* Allows users to configure if probability applies only for "root spans", "root spans and remote
parent", or "all spans".
* Default is to apply only for "root spans and remote parent".
* Remote parent property should be added to the SpanContext see specs [PR/216][specs-pr-216]
* Sample with 1/N probability

**Root Span Decision:**

|Sampler|RecordEvents|SampledFlag|
|---|---|---|
|ALWAYS_ON|`True`|`True`|
|ALWAYS_OFF|`False`|`False`|
|ALWAYS_PARENT|`False`|`False`|
|Probability|`SamplingHint==RECORD OR SampledFlag`|`SamplingHint==RECORD_AND_PROPAGATE OR Probability`|

**Child Span Decision:**

|Sampler|RecordEvents|SampledFlag|
|---|---|---|
|ALWAYS_ON|`True`|`True`|
|ALWAYS_OFF|`False`|`False`|
|ALWAYS_PARENT|`ParentSampledFlag`|`ParentSampledFlag`|
|Probability|`SamplingHint==RECORD OR SampledFlag`|`ParentSampledFlag OR SamplingHint==RECORD_AND_PROPAGATE OR Probability`|

### Links
This RFC proposes that Links will be recorded only during the start `Span` operation, because:
* Link's `SampledFlag` can be used in the sampling decision.
* OpenTracing supports adding references only during the `Span` creation.
* OpenCensus supports adding links at any moment, but this was mostly used to record child Links
which are not supported in OpenTelemetry.
* Allowing links to be recorded after the sampling decision is made will cause samplers to not
work correctly and unexpected behaviors for sampling.

### When does sampling happen?
The sampling decision will happen before a real `Span` object is returned to the user, because:
* If child spans are created they need to know the 'SampledFlag'.
* If `SpanContext` is propagated on the wire the 'SampledFlag' needs to be set.
* If user records any tracing event the `Span` object needs to know if the data are kept or not.
It may be possible to always collect all the events until the sampling decision is made but this is
an important optimization.

There are two important use-cases to be considered:
* All information that may be used for sampling decisions are available at the moment when the
logical `Span` operation should start. This is the most common case.
* Some information that may be used for sampling decision are NOT available at the moment when the
logical `Span` operation should start (e.g. `http.route` may be determine later).

The current [span creation logic][span-creation] facilitates very well the first use-case, but
the second use-case requires users to record the logical `start_time` and collect all the
information necessarily to start the `Span` in custom objects, then when all the properties are
available call the span creation API.

The RFC proposes that we keep the current [span creation logic][span-creation] as it is and we will
address the delayed sampling in a different RFC when that becomes a high priority.

The SDK must call the `Sampler` every time a `Span` is created during the start span operation.

**Alternatives considerations:**
* We considered, to offer a delayed span construction mechanism:
* For languages where a `Builder` pattern is used to construct a `Span`, to allow users to
create a `Builder` where the start time of the Span is considered when the `Builder` is created.
* For languages where no intermediate object is used to construct a `Span`, to allow users maybe
via a `StartSpanOption` object to start a `Span`. The `StartSpanOption` allows users to set all
the start `Span` properties.
* Pros:
* Would resolve the second use-case posted above.
* Cons:
* We could not identify too many real case examples for the second use-case and decided to
postpone the decision to avoid premature decisions.
* We considered, instead of requiring that sampling decision happens before the `Span` is
created to add an explicit `MakeSamplingDecision(SamplingHint)` on the `Span`. Attempts to create
a child `Span`, or to access the `SpanContext` would fail if `MakeSamplingDecision()` had not yet
been run.
* Pros:
* Simplifies the case when all the attributes that may be used for sampling are not available
when the logical `Span` operation should start.
* Cons:
* The most common case would have required an extra API call.
* Error prone, users may forget to call the extra API.
* Unexpected and hard to find errors if user tries to create a child `Span` before calling
MakeSamplingDecision().
* We considered allowing the sampling decision to be arbitrarily delayed, but guaranteed before
any child `Span` is created, or `SpanContext` is accessed, or before `Span.end()` finished.
* Pros:
* Similar and smaller API that supports both use-cases defined ahead.
* Cons:
* If `SamplingHint` needs to also be delayed recorded then an extra API on Span is required
to set this.
* Does not allow optimization to not record tracing events, all tracing events MUST be
recorded before the sampling decision is made.
This conversation was marked as resolved by bogdandrutu

This comment has been minimized.

Copy link
@yurishkuro

yurishkuro Aug 20, 2019

if sampling decisions are based on a set of policies, it can be an inherent attribute of a policy whether it depends on a delayed datum (such as late tag or span name) and thus requires recording events prior to sampling decision being made. If user picks a policy that does not depend on such delayed data (e.g. user picks a basic probabilistic policy), then the tracer would know that no delayed sampling is going to happen and can short-circuit event recording once the decision is No. This is the approach we're taking in Jaeger SDKs.

This comment has been minimized.

Copy link
@lizthegrey

lizthegrey Aug 20, 2019

Member

We deliberately chose to defer the "make decision later" option until later, as it adds more complexity and we can always add it later.

This comment has been minimized.

Copy link
@bogdandrutu

bogdandrutu Aug 20, 2019

Author Member

We need to define when is the limit when the sampling decision should happen. I like the idea to have on the sampler a property that says is delayed sampling so we keep all the events until the sampling happens, but when is the limit/point that guarantees that sampling happened.

This comment has been minimized.

Copy link
@yurishkuro

This comment has been minimized.

Copy link
@bogdandrutu

bogdandrutu Aug 22, 2019

Author Member

This is in a "alternatives" section and we actually reverted the proposed change. So I would keep it here just for the record and we will discuss in more details when we do make a decision about this.


## Prior art and alternatives
Prior art for Zipkin, and other Dapper based systems: all client-side sampling decisions are made at
head. Thus, we need to retain compatibility with this.

## Open questions
This RFC does not necessarily resolve the question of how to propagate sampling rate values between
different spans and processes. A separate RFC will be opened to cover this case.

## Future possibilities
In the future, we propose that library developers may be able to defer the decision on whether to
recommend the trace be sampled or not sampled until mid-way through execution;

## Related Issues
* [opentelemetry-specification/189](https://github.com/open-telemetry/opentelemetry-specification/issues/189)
* [opentelemetry-specification/187](https://github.com/open-telemetry/opentelemetry-specification/issues/187)
* [opentelemetry-specification/164](https://github.com/open-telemetry/opentelemetry-specification/issues/164)
* [opentelemetry-specification/125](https://github.com/open-telemetry/opentelemetry-specification/issues/125)
* [opentelemetry-specification/87](https://github.com/open-telemetry/opentelemetry-specification/issues/87)
* [opentelemetry-specification/66](https://github.com/open-telemetry/opentelemetry-specification/issues/66)
* [opentelemetry-specification/65](https://github.com/open-telemetry/opentelemetry-specification/issues/65)
* [opentelemetry-specification/53](https://github.com/open-telemetry/opentelemetry-specification/issues/53)
* [opentelemetry-specification/33](https://github.com/open-telemetry/opentelemetry-specification/issues/33)
* [opentelemetry-specification/32](https://github.com/open-telemetry/opentelemetry-specification/issues/32)
* [opentelemetry-specification/31](https://github.com/open-telemetry/opentelemetry-specification/issues/31)

[trace-flags]: https://github.com/w3c/trace-context/blob/master/spec/20-http_header_format.md#trace-flags
[specs-pr-216]: https://github.com/open-telemetry/opentelemetry-specification/pull/216
[span-creation]: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/api-tracing.md#span-creation
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.