Adding propagators API and b3 SDK implementation (#51, #52) #78

toumorokoshi · 2019-08-08T15:26:42Z

Specification: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/api-propagators.md.

…pen-telemetry#52) Specification: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/api-propagators.md.

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py

opentelemetry-sdk/src/opentelemetry/sdk/context/propagation/b3_format.py

reyang · 2019-08-08T16:03:40Z

A general question - do we expect the API package to provide implementations for:

Binary, which is going to be used for multi-processing.
W3C TraceContext, which is going to be used by HTTP.

The reason for this ask is that we have a scenario that users only have API package installed, and the application should work (which means if the application received W3C TraceContext headers, it should follow the W3C spec and process the headers).

c24t

Minor comments, but this LGTM pending the docstring fixes. You may want to include the tracer methods to get the propagators in this PR, or add them in a follow up PR.

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py

opentelemetry-sdk/tests/context/propagation/test_b3_format.py

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py

opentelemetry-sdk/src/opentelemetry/sdk/context/propagation/b3_format.py

c24t · 2019-08-09T01:26:15Z

A general question - do we expect the API package to provide implementations

That's my understanding, the API should pass the headers through if there's no SDK loaded. It's also what the java client does.

I think it's fine to add the implementation (in the API) in another PR since it will just implement these abstract classes, not replace them.

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py

toumorokoshi · 2019-08-10T03:56:44Z

A general question - do we expect the API package to provide implementations

That's my understanding, the API should pass the headers through if there's no SDK loaded. It's also what the java client does.

I think it's fine to add the implementation (in the API) in another PR since it will just implement these abstract classes, not replace them.

Sounds good. I'm not strongly opinionated on where to place them, although I will note that moving implementations to SDK pretty much guarantees that opentelemetry-sdk will always be consumed, as I imagine most distributed trace implementations will propagate one of binary / w3c tracestate / b3 headers anyway. If we move all propagators to the API then there's a chance nothing from SDK will be necessary (although Resource's implementation is in SDK as well).

toumorokoshi · 2019-08-11T04:34:46Z

I've submitted / contributed to 3 Issues in the specification around some of the design decisions here:

open-telemetry/opentelemetry-specification#112
open-telemetry/opentelemetry-specification#210
open-telemetry/opentelemetry-specification#211

Guidance from Ted on the gitter is to create a python example of how some of these improvements will work. I'll focus on cleaning up the code in this PR, and add a future PR around how to use these propagators that will answer questions like where the registry lies.

https://gitter.im/open-telemetry/opentelemetry-specification?at=5d4f492153490e334dccf927

incorporating unit tests and a more lenient implementation of the b3 propagator. Fixing a bug with b3 propagators consuming and producing integers rather than hex-encoded values.

Oberon00 · 2019-08-12T06:24:35Z

opentelemetry-api/src/opentelemetry/context/propagation/binaryformat.py

+from opentelemetry.trace import SpanContext
+
+
+class BinaryFormat(abc.ABC):


I think this class could use some "performance-oriented" APIs: Since creating a slice of bytes copies it, I suggest from_bytes(byte_representation:bytes, offset: int = 0, length: int = -1), and maybe append_bytes(dst: bytearray).

I can see the value here, but I may wait on adding these until I add an implementation (e.g. the standard binary format). That might better illustrate the right API, unless you already have an example in mind.

Alternatively it looks like memoryview would be a great way to read values:

https://docs.python.org/3/library/stdtypes.html#memoryview

Although I could always construct a memoryview from the bytes object I received. For to_bytes I could use bytearray internally to construct the buffer.

In case it's helpful, OC uses memoryview in the tags (read: distributed context) propagator: https://github.com/census-instrumentation/opencensus-python/blob/577fb61ab26513e3b448b6696432eefb0b11fa73/opencensus/tags/propagation/binary_serializer.py#L38.

opentelemetry-sdk/src/opentelemetry/sdk/context/propagation/b3_format.py

opentelemetry-api/src/opentelemetry/context/propagation/binaryformat.py

opentelemetry-sdk/tests/context/propagation/test_b3_format.py

refactoring formatting of trace / spans for b3 as it's used frequently. renaming trace_id / span_id in b3 unit tests for clarity. removing unneeded int casts.

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py

reyang · 2019-08-13T19:55:26Z

I have a general question which more related to the spec.
Do we expect the propagator to take care of SpanContext (W3C TraceContext) plus the DistributedContext (W3C CorrelationContext) or just the former?

bogdandrutu · 2019-08-14T01:30:48Z

opentelemetry-api/src/opentelemetry/context/propagation/binaryformat.py

+from opentelemetry.trace import SpanContext
+
+
+class BinaryFormat(abc.ABC):


In order to not duplicate the same class for correlation-context (distributedcontext) I would suggest to make this a template and have BinaryFormat<SpanContext> and BinaryFormat<DistributedContext>

Thanks! Mypy typing does support generics, but I'm planning of filing another PR that aligns closer to the tickets I've mentioned above. This will include creating a unified composed object for SpanContext and DistributedContext and using the composed object as the context for propagators, thereby eliminating the need for the generic.

From an offline conversation with @bogdandrutu: a bigger problem is that referencing SpanContext here would make context depend on trace.

This creates a circular dependency, and even if we remove the propagators from the tracer trace will depend on context in other places. If we think of the context package as representing the base context propagation layer it's surprising to see it depend on a layer higher in the stack.

This isn't a blocking comment for this PR, just something to keep in mind when you start composing context objects.

HTTP Headers can contains multiple values for the same key. This is important to support for formats such as w3c tracestate.

reyang · 2019-08-14T04:44:16Z

opentelemetry-api/src/opentelemetry/context/propagation/binaryformat.py

+    @staticmethod
+    @abc.abstractmethod
+    def from_bytes(byte_representation: bytes) -> typing.Optional[SpanContext]:
+        """Return a SpanContext that was represented by bytes.


As we discussed today, we might want to use the same propagator for both SpanContext and DistributedContext.
No need to be blocked though, we can address this in another PR.

yes, coming up in the next PR. want to get this merged in first.

Roger that.

reyang · 2019-08-14T04:52:46Z

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py

+                must be paired with an appropriate get_from_carrier
+                which understands how to extract a value from it.
+        Returns:
+            A SpanContext with configuration found in the carrier.


If the carrier doesn't provide a SpanContext, do we expect to generate one or return None?
This could become tricky since we might receive something between valid and invalid. For example
https://github.com/w3c/trace-context/blob/b145878f5618fccbf4926bc181ecb25b709b111d/test/test.py#L536.

According to the the Specification (https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/api-propagators.md#frombytes):

If the value could not be parsed, the underlying implementation SHOULD decide to return ether an empty value, an invalid value, or a valid value.

In Java we do not return null ever for this (makes me wonder if we should tune the Specification for this), and we instead will be returning SpanContext.getInvalid() ;)

The current organization of the API makes the ramifications of this choice and how to handle it a decision of whoever would handle handing the SpanContext back to the context. I.E. error handling and how to deal with that is up to whoever calls the extract / inject method.

I think as a convention the formats should return either a valid value or nothing at all. But honestly I think we'll see changes here as we discover more use cases around things like needing to support multiple propagators.

carlosalberto · 2019-08-15T14:23:02Z

opentelemetry-sdk/src/opentelemetry/sdk/context/propagation/b3_format.py

+            elif len(fields) == 4:
+                trace_id, span_id, sampled, _parent_span_id = fields
+            else:
+                return trace.INVALID_SPAN_CONTEXT


toumorokoshi · 2019-08-15T17:00:32Z

@reyang @c24t @carlosalberto @Oberon00 if all looks good, can I get an approval? Have a proposal PR waiting on merging this one.

reyang

The code is good, there are some open design questions that we need to figure out. We can merge this then explore the open questions in separate PRs.

Where to put propagator (not likely to be Tracer, very likely to be with integrations)?
How to decouple formatter and propagator?
Do we handle SpanContext and DistributedContext in the same propagator (most likely yes!)
How many DistributedContext do we want to have (e.g. is each Tracer having its own version)?

toumorokoshi · 2019-08-15T17:12:22Z

Yes! Great points.

I have a PR that has a proposal for 3. I'll put 1,2 on my list to get proposals for.

4 has far-reaching impacts so I imagine we'll start considering that case after our initial release date.

c24t

Only minor comments, this LGTM pending @reyang and @carlosalberto's approval.

Thanks for taking on the propagators! This PR has unearthed a lot of assumptions about context prop.

c24t · 2019-08-15T17:33:10Z

opentelemetry-api/src/opentelemetry/context/propagation/binaryformat.py

+from opentelemetry.trace import SpanContext
+
+
+class BinaryFormat(abc.ABC):


From an offline conversation with @bogdandrutu: a bigger problem is that referencing SpanContext here would make context depend on trace.

This creates a circular dependency, and even if we remove the propagators from the tracer trace will depend on context in other places. If we think of the context package as representing the base context propagation layer it's surprising to see it depend on a layer higher in the stack.

This isn't a blocking comment for this PR, just something to keep in mind when you start composing context objects.

c24t · 2019-08-15T17:36:00Z

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py

+        PROPAGATOR = HTTPTextFormat()
+
+
+


Why all the blank lines?

c24t · 2019-08-15T17:36:32Z

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py

+            get_from_carrier: a function that can retrieve zero
+                or more values from the carrier. In the case that
+                the value does not exist, return an empty list.
+            carrier: and object which contains values that are


Suggested change

carrier: and object which contains values that are

carrier: an object which contains values that are

c24t · 2019-08-15T17:37:21Z

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py

+        """Create a SpanContext from values in the carrier.
+
+        The extract function should retrieve values from the carrier
+        object using get_from_carrier, and use values to populate a


Suggested change

object using get_from_carrier, and use values to populate a

object using `get_from_carrier`, and use values to populate a

c24t · 2019-08-15T17:44:03Z

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py

+
+        The extract function should retrieve values from the carrier
+        object using get_from_carrier, and use values to populate a
+        SpanContext value and return it.


Suggested change

SpanContext value and return it.

`SpanContext` value and return it.

And the same changes to other docstrings. This is to make the sphinx docs render nicely, but feel free to omit the sphinx markup where it hurts code readability.

I haven't been including sphinx markup in the first line of the docstrings since we need them to fit on a single line, but since we changed the sphinx default role most markdown is just backticks now, and it may be worth the extra characters to do this consistently.

Adding propagators API and b3 SDK implementation (open-telemetry#51, o…

f62eeac

…pen-telemetry#52) Specification: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/api-propagators.md.

toumorokoshi requested review from c24t, carlosalberto, Oberon00 and reyang as code owners August 8, 2019 15:26

Addressing lint errors

4a3d617

reyang reviewed Aug 8, 2019

View reviewed changes

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py Outdated Show resolved Hide resolved

reyang reviewed Aug 8, 2019

View reviewed changes

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py Outdated Show resolved Hide resolved

reyang reviewed Aug 8, 2019

View reviewed changes

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py Outdated Show resolved Hide resolved

reyang reviewed Aug 8, 2019

View reviewed changes

opentelemetry-sdk/src/opentelemetry/sdk/context/propagation/b3_format.py Show resolved Hide resolved

c24t reviewed Aug 9, 2019

View reviewed changes

alban reviewed Aug 9, 2019

View reviewed changes

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py Show resolved Hide resolved

Adding more flexible b3 format, fiixing linting

ce65a73

incorporating unit tests and a more lenient implementation of the b3 propagator. Fixing a bug with b3 propagators consuming and producing integers rather than hex-encoded values.

Oberon00 reviewed Aug 12, 2019

View reviewed changes

Addressing comments

90080f2

refactoring formatting of trace / spans for b3 as it's used frequently. renaming trace_id / span_id in b3 unit tests for clarity. removing unneeded int casts.

reyang reviewed Aug 13, 2019

View reviewed changes

opentelemetry-api/src/opentelemetry/context/propagation/httptextformat.py Show resolved Hide resolved

songy23 mentioned this pull request Aug 14, 2019

Remove BinaryFormat and HttpTextFormat from Tracer open-telemetry/opentelemetry-specification#214

Closed

bogdandrutu reviewed Aug 14, 2019

View reviewed changes

Modifying HTTPTextFormat getter to retrieve multiple values

529e184

HTTP Headers can contains multiple values for the same key. This is important to support for formats such as w3c tracestate.

reyang reviewed Aug 14, 2019

View reviewed changes

carlosalberto reviewed Aug 15, 2019

View reviewed changes

reyang approved these changes Aug 15, 2019

View reviewed changes

toumorokoshi merged commit 74a9534 into open-telemetry:master Aug 15, 2019

c24t approved these changes Aug 15, 2019

View reviewed changes

toumorokoshi mentioned this pull request Aug 16, 2019

unifying propagator formatters #89

Closed

srikanthccv pushed a commit to srikanthccv/opentelemetry-python that referenced this pull request Nov 1, 2020

Add Event interface (open-telemetry#78)

93db59e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding propagators API and b3 SDK implementation (#51, #52) #78

Adding propagators API and b3 SDK implementation (#51, #52) #78

toumorokoshi commented Aug 8, 2019

reyang commented Aug 8, 2019

c24t left a comment

c24t commented Aug 9, 2019

toumorokoshi commented Aug 10, 2019

toumorokoshi commented Aug 11, 2019

Oberon00 Aug 12, 2019

toumorokoshi Aug 13, 2019

c24t Aug 15, 2019

reyang commented Aug 13, 2019

bogdandrutu Aug 14, 2019 •

edited

Loading

toumorokoshi Aug 14, 2019

c24t Aug 15, 2019

reyang Aug 14, 2019 •

edited

Loading

toumorokoshi Aug 15, 2019

reyang Aug 15, 2019

reyang Aug 14, 2019

carlosalberto Aug 15, 2019

toumorokoshi Aug 15, 2019

carlosalberto Aug 15, 2019

toumorokoshi commented Aug 15, 2019

reyang left a comment

toumorokoshi commented Aug 15, 2019

c24t left a comment

c24t Aug 15, 2019

c24t Aug 15, 2019

c24t Aug 15, 2019

c24t Aug 15, 2019

c24t Aug 15, 2019

		from opentelemetry.trace import SpanContext


		class BinaryFormat(abc.ABC):

	carrier: and object which contains values that are
	carrier: an object which contains values that are

	object using get_from_carrier, and use values to populate a
	object using `get_from_carrier`, and use values to populate a

	SpanContext value and return it.
	`SpanContext` value and return it.

Adding propagators API and b3 SDK implementation (#51, #52) #78

Adding propagators API and b3 SDK implementation (#51, #52) #78

Conversation

toumorokoshi commented Aug 8, 2019

reyang commented Aug 8, 2019

c24t left a comment

Choose a reason for hiding this comment

c24t commented Aug 9, 2019

toumorokoshi commented Aug 10, 2019

toumorokoshi commented Aug 11, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reyang commented Aug 13, 2019

bogdandrutu Aug 14, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reyang Aug 14, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toumorokoshi commented Aug 15, 2019

reyang left a comment

Choose a reason for hiding this comment

toumorokoshi commented Aug 15, 2019

c24t left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bogdandrutu Aug 14, 2019 •

edited

Loading

reyang Aug 14, 2019 •

edited

Loading