Moving contextvars and threadlocal context implementations to the API #419

codeboten · 2020-02-14T00:12:55Z

While keeping the flexibility of the interface to support additional implementations of the RuntimeContext, moving the ContextVarsRuntimeContext and ThreadLocalRuntimeContext into the API ensures a useful context implementation for users without requiring additional configuration. This changes checks the version number to determine which implementation to use, unless it is overridden by the OPENTELEMETRY_CONTEXT environment variable. Removing the DefaultRuntimeContext as part of this change.

The discussion suggesting this change can be found here

While keeping the flexibility of the interface to support additional implementations of the RuntimeContext, moving the ContextVarsRuntimeContext and ThreadLocalRuntimeContext into the API ensures a useful context implementation for users without requiring additional configuration. This changes checks the version number to determine which implementation to use, unless it is overridden by the OPENTELEMETRY_CONTEXT environment variable. Removing the DefaultRuntimeContext as part of this change. Signed-off-by: Alex Boten <aboten@lightstep.com>

opentelemetry-api/setup.py

Oberon00 · 2020-02-14T14:15:09Z

I think we should follow what Java does in open-telemetry/opentelemetry-java#720 and partially has already done (https://github.com/open-telemetry/opentelemetry-java/tree/master/context_prop): Move the context to its own distribution that depends on nothing and on which API depends.

codeboten · 2020-02-14T15:58:48Z

@Oberon00 I like this, were you thinking another top level directory for this?

Signed-off-by: Alex Boten <aboten@lightstep.com>

opentelemetry-api/setup.py

ocelotl · 2020-02-14T21:38:24Z

I think we should follow what Java does in open-telemetry/opentelemetry-java#720 and partially has already done (https://github.com/open-telemetry/opentelemetry-java/tree/master/context_prop): Move the context to its own distribution that depends on nothing and on which API depends.

@Oberon00 are you talking about moving the abstract RuntimeContext class into its own distribution or about moving the ContextVarsRuntimeContext and ThreadLocalRuntimeContext implementations or the three of them?

Signed-off-by: Alex Boten <aboten@lightstep.com>

codecov-io · 2020-02-14T22:34:54Z

Codecov Report

Merging #419 into master will decrease coverage by 0.1%.
The diff coverage is 90%.

@@            Coverage Diff             @@
##           master     #419      +/-   ##
==========================================
- Coverage    88.3%   88.19%   -0.11%     
==========================================
  Files          42       41       -1     
  Lines        2060     2042      -18     
  Branches      236      234       -2     
==========================================
- Hits         1819     1801      -18     
  Misses        170      170              
  Partials       71       71

Impacted Files	Coverage Δ
...elemetry-api/src/opentelemetry/context/__init__.py	`67.39% <100%> (+3.1%)`	⬆️
...i/src/opentelemetry/context/threadlocal_context.py	`100% <100%> (ø)`
...i/src/opentelemetry/context/contextvars_context.py	`87.5% <66.66%> (ø)`
...etry-sdk/src/opentelemetry/sdk/metrics/__init__.py	`98.29% <0%> (-0.18%)`	⬇️
...elemetry-api/src/opentelemetry/metrics/__init__.py	`95.58% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 72862c9...57b214e. Read the comment docs.

c24t

Looks great!

c24t · 2020-02-15T00:48:43Z

opentelemetry-api/setup.py

-    install_requires=["typing; python_version<'3.5'"],
+    install_requires=[
+        "typing; python_version<'3.5'",
+        "aiocontextvars; python_version<'3.7' and python_version>='3.5'",


Good call on the conditional requirements.

opentelemetry-api/src/opentelemetry/context/threadlocal_context.py

c24t · 2020-02-15T01:05:32Z

opentelemetry-api/tests/context/test_contextvars_context.py

+        context.set_current(self.previous_context)
+
+    @patch(
+        "opentelemetry.context._RUNTIME_CONTEXT", ContextVarsRuntimeContext()  # type: ignore


I'm not sure why the CLI black isn't wrapping these. FWIW black.format_file_contents via the vim plugin does.

It could be line length isn't being honored. I see it in the pyproject.toml, but IIRC my editor wasn't always honoring that configuration either.

c24t · 2020-02-15T01:06:38Z

@Oberon00 are you talking about moving the abstract RuntimeContext class into its own distribution or about moving the ContextVarsRuntimeContext and ThreadLocalRuntimeContext implementations or the three of them?

I'd assume all three. This is what we did with the opencensus-context package too.

Oberon00 · 2020-02-15T11:44:32Z

Exactly. It is illuminating to read OTEP 66, in which the context API was introduced, especially the motivation:

Separation of concerns

Cleaner package layout results in an easier to learn system. It is possible to
understand Context Propagation without needing to understand Observability.
[...]

Extensibility

A clean separation allows the context propagation mechanisms to be used on
their own [...]

-- https://github.com/open-telemetry/oteps/blob/master/text/0066-separate-context-propagation.md

So one could install opentelemetry-context without even using telemetry, to e.g. transport identity information across contexts and threads. Basically that package would contain our re-implementation/abstraction of contextvars plus the inter-process propagation.

condorcet · 2020-02-16T09:09:42Z

opentelemetry-api/src/opentelemetry/context/contextvars_context.py


 if (3, 5, 3) <= version_info < (3, 7):
-    import aiocontextvars  # type: ignore # pylint:disable=unused-import
+    import aiocontextvars  # type: ignore # pylint:disable=unused-import,import-error


Please note that without this patch in aiocontextvars fantix/aiocontextvars#66 behavior of async code in python<3.7 differs from original contextvars in python 3.7.

Hello @condorcet

What specific behavior difference are you referring to?

Hi @ocelotl !
Even with aiocontextvars call_soon and similar methods of loop don't create their own context. Hence such scheduled callbacks are "stealing" outer context.
For example:

import asyncio from contextvars import ContextVar var = ContextVar('var') def callback(name): print(f'{name} var before: {var.get()}') var.set(name) print(f'{name} var after: {var.get()}') if __name__ == '__main__': var.set('root') loop = asyncio.get_event_loop() loop.call_soon(callback, 'callback_1') loop.call_soon(callback, 'callback_2') asyncio.get_event_loop().run_forever()

In python 3.7 (proper behavior):

callback_1 var before: root callback_1 var after: callback_1 callback_2 var before: root callback_2 var after: callback_2

In python 3.6 (with aiocontextvars):

callback_1 var before: root callback_1 var after: callback_1 callback_2 var before: callback_1 callback_2 var after: callback_2

Here you can see that first callback changed context var, and next callback used modified value rather than initial.

Thanks for clarifying @condorcet, I'll open an issue to track this.

Opened #436

toumorokoshi

nice!

toumorokoshi · 2020-02-17T19:03:05Z

opentelemetry-api/src/opentelemetry/context/__init__.py

@@ -84,9 +85,14 @@ def get_current() -> Context:
    if _RUNTIME_CONTEXT is None:
        # FIXME use a better implementation of a configuration manager to avoid having
        # to get configuration values straight from environment variables
+        if version_info < (3, 5):
+            # contextvars are not supported in 3.4, use thread-local storage
+            default_context = "threadlocal_context"


does this mean that async context won't work for python3.4?

toumorokoshi · 2020-02-17T19:06:19Z

opentelemetry-api/tests/context/test_contextvars_context.py

+        context.set_current(self.previous_context)
+
+    @patch(
+        "opentelemetry.context._RUNTIME_CONTEXT", ContextVarsRuntimeContext()  # type: ignore


It could be line length isn't being honored. I see it in the pyproject.toml, but IIRC my editor wasn't always honoring that configuration either.

…t.py Co-Authored-By: Chris Kleinknecht <libc@google.com>

toumorokoshi · 2020-02-18T18:20:59Z

There was a good conversation in the Gitter around breaking this out into it's own package now.

@c24t points that there will be complications around version mismatches.

E.g. today, if someone hard pins the API / SDK, they will also have to be aware that there is a context package that must also be hard pinned.

so consumers will probably hard pin API / SDK, pick up a backwards-incompatible context change (since they're unaware of it's existence or role), then have to go figure out what went wrong.

Probably good enough motivation to not split the context package out until it becomes more stable.

Oberon00 · 2020-02-19T15:25:45Z

@toumorokoshi, @c24t: With the same argument, we shouldn't split the SDK from the API.

toumorokoshi · 2020-02-19T18:09:06Z

Yes, I think ideally, we should have kept it together as long as we could have, until we had a strong need to (e.g. someone started implementing their own SDK). But doing it from the start I think isn't the end of the world, it also sets the precedence that both SDK and API packages should be called out specifically.

I could be convinced either way, hopefully context changes are few and far between after the first cycle or two. But I don't intend to be a voice of authority on this one, I'll defer to those who are more directly contributing, and also have a need for a separate context object.

codeboten requested a review from c24t February 14, 2020 00:12

codeboten requested a review from a team as a code owner February 14, 2020 00:12

mauriciovasquezbernal reviewed Feb 14, 2020

View reviewed changes

opentelemetry-api/setup.py Outdated Show resolved Hide resolved

adding python_version

253a969

Signed-off-by: Alex Boten <aboten@lightstep.com>

mauriciovasquezbernal reviewed Feb 14, 2020

View reviewed changes

opentelemetry-api/setup.py Outdated Show resolved Hide resolved

fix lint, update required version

39e0a19

Signed-off-by: Alex Boten <aboten@lightstep.com>

c24t approved these changes Feb 15, 2020

View reviewed changes

condorcet reviewed Feb 16, 2020

View reviewed changes

toumorokoshi approved these changes Feb 17, 2020

View reviewed changes

Update opentelemetry-api/src/opentelemetry/context/threadlocal_contex…

57b214e

…t.py Co-Authored-By: Chris Kleinknecht <libc@google.com>

toumorokoshi merged commit d824f19 into open-telemetry:master Feb 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Moving contextvars and threadlocal context implementations to the API #419

Moving contextvars and threadlocal context implementations to the API #419

codeboten commented Feb 14, 2020 •

edited

Loading

Oberon00 commented Feb 14, 2020

codeboten commented Feb 14, 2020

ocelotl commented Feb 14, 2020

codecov-io commented Feb 14, 2020 •

edited

Loading

c24t left a comment

c24t Feb 15, 2020

c24t Feb 15, 2020

toumorokoshi Feb 17, 2020

c24t commented Feb 15, 2020

Oberon00 commented Feb 15, 2020 •

edited

Loading

condorcet Feb 16, 2020

ocelotl Feb 20, 2020

condorcet Feb 20, 2020

codeboten Feb 20, 2020

codeboten Feb 20, 2020

toumorokoshi left a comment

toumorokoshi Feb 17, 2020

toumorokoshi Feb 17, 2020

toumorokoshi commented Feb 18, 2020

Oberon00 commented Feb 19, 2020

toumorokoshi commented Feb 19, 2020

Moving contextvars and threadlocal context implementations to the API #419

Moving contextvars and threadlocal context implementations to the API #419

Conversation

codeboten commented Feb 14, 2020 • edited Loading

Oberon00 commented Feb 14, 2020

codeboten commented Feb 14, 2020

ocelotl commented Feb 14, 2020

codecov-io commented Feb 14, 2020 • edited Loading

Codecov Report

c24t left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c24t commented Feb 15, 2020

Oberon00 commented Feb 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toumorokoshi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toumorokoshi commented Feb 18, 2020

Oberon00 commented Feb 19, 2020

toumorokoshi commented Feb 19, 2020

codeboten commented Feb 14, 2020 •

edited

Loading

codecov-io commented Feb 14, 2020 •

edited

Loading

Oberon00 commented Feb 15, 2020 •

edited

Loading