Update trace propagation format for botocore direct invocations #2639

agocs · 2021-07-12T18:33:31Z

Description

Updates the trace propagation format for botocore direct lambda invocations. The old way was:

clientContext: {
  custom:{
    _datadog: {
      x-datadog-trace-id: "",
      x-datadog-parent-id: "",
      ...
    }
  }
}

New way is:

clientContext: {
  custom: {
      x-datadog-trace-id: "",
      x-datadog-parent-id: "",
      ...
  }
}

https://docs.google.com/document/d/1pruSeesAr2bAVMtz_rUlsiBZKFoFWdB4nSq5Vuym2l8/edit?usp=sharing

Reasoning

The Go and Java runtimes only support ClientContext.custom fields of type map[string]string. C.f. :

In order to pass trace contexts to directly invoked Go and Java lambda functions, we can write the trace context key:values to ClientContext.custom rather than creating a ClientContext.custom._datadog object like we've been doing.

Backwards compatibility

This is, strictly speaking, a breaking change in how we add trace context to direct invocations. The Python and JS repos have already been updated to accept the new format:

They both check for the old and new formats.

Checklist

Added to the correct milestone.
Tests provided or description of manual testing performed is included in the code or PR.
Library documentation is updated.
Corp site documentation is updated (link to the PR).

Kyle-Verhoog

I'm thinking we should leave the old method behind a configuration option since a bunch of users will not be on the latest version of the serverless client(s) 🤔

Thoughts @agocs?

agocs · 2021-07-12T20:17:34Z

@Kyle-Verhoog I can get behind that. I considered it, but initially decided against it as this is such a niche change. Definitely happy to add that parameter though.

Which way do you think the default should go? I was thinking it should favor the new, working trace propagation format.

Oh, and is this the extent of the configuration, or is there a config file hiding somewhere I don't know about? https://docs.datadoghq.com/tracing/setup_overview/setup/python/?tab=otherenvironments#configuration

Kyle-Verhoog · 2021-07-12T20:32:27Z

Which way do you think the default should go? I was thinking it should favor the new, working trace propagation format.

Yup - agreed 👍

The configuration option can be added here:

dd-trace-py/ddtrace/contrib/botocore/patch.py

Lines 35 to 41 in 7e1d82a

    
           # Botocore default settings 
        
           config._add( 
        
               "botocore", 
        
               { 
        
                   "distributed_tracing": get_env("botocore", "distributed_tracing", default=True), 
        
               }, 
        
           )

. We should offer an environment variable too

and then documented here

dd-trace-py/ddtrace/contrib/botocore/__init__.py

Lines 16 to 20 in 7e1d82a

    
           Configuration 
        
           ~~~~~~~~~~~~~ 
        
           .. py:data:: ddtrace.config.botocore['distributed_tracing']

(this will show up here: https://ddtrace.readthedocs.io/en/stable/integrations.html#botocore)

Also an upgrade release note entry would be required here which can be created with reno:

$ reno new botocore

agocs · 2021-07-12T20:34:25Z

Awesome! I appreciate it!

…ion_trace_propagation_format

agocs · 2021-07-14T20:15:04Z

@Kyle-Verhoog Changes made! Thanks. Should I drop a link to this in the apm-python channel?

Kyle-Verhoog

👍 solid tests - thanks for also keeping a test with the old setting! Just some suggestions around the user-facing aspects 🙂

Kyle-Verhoog · 2021-07-15T13:27:38Z

ddtrace/contrib/botocore/patch.py

@@ -37,6 +37,9 @@
    "botocore",
    {
        "distributed_tracing": get_env("botocore", "distributed_tracing", default=True),
+        "clientcontext_custom_add_datadog_object": get_env(
+            "botocore", "clientcontext_custom_add_datadog_object", default=False


Suggested change

"botocore", "clientcontext_custom_add_datadog_object", default=False

"botocore", "legacy_context", default=False

Not 100% about this suggesting.. aiming toward something less verbose - I don't think the name has to divulge the implementation details here... unless we're planning in the future to change this implementation again 😬

@DataDog/apm-python any ideas for naming here?

invoke_with_legacy_context ?

Kyle-Verhoog · 2021-07-15T13:31:15Z

ddtrace/contrib/botocore/__init__.py

@@ -24,6 +24,25 @@

   Default: ``True``

+.. py:data:: ddtrace.config.botocore['clientcontext_custom_add_datadog_object']
+
+    When a tracing a directly invoked lambda function, the trace context is added to the


This description is awesome but probably a bit too verbose for users who might get lost in the details. The important part is the version support listed below. WDYT about trimming it down?

ddtrace/contrib/botocore/__init__.py

ddtrace/contrib/botocore/patch.py

Mutate client_context_object instead of returning Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

…pagation_format' of github.com:DataDog/dd-trace-py into chris.agocs/update_botocore_direct_invocation_trace_propagation_format

…ion_trace_propagation_format

Kyle-Verhoog

nice! just some last concerns about the release note but otherwise lgtm!

Kyle-Verhoog · 2021-07-19T14:28:07Z

releasenotes/notes/botocore-46f5bf04f64777b4.yaml

@@ -0,0 +1,4 @@
+---
+features:


this should be an upgrade-level note mentioning the change and the backwards compatibility option as users using older python/node lambda client versions will have a breaking change

Kyle-Verhoog · 2021-07-19T14:30:10Z

ddtrace/contrib/botocore/__init__.py

@@ -24,6 +24,19 @@

   Default: ``True``

+.. py:data:: ddtrace.config.botocore['invoke_with_legacy_context']


👍 I think I'm good with this name.

@DataDog/apm-python thoughts?

Kyle-Verhoog · 2021-07-19T14:34:35Z

releasenotes/notes/botocore-46f5bf04f64777b4.yaml

+---
+features:
+  - |
+    botocore: added `invoke_with_legacy_context` configuration setting, which is disabled by default. This configuration setting is to preserve a legacy behavior in which, when tracing a direct lambda invocation, trace context was propagated by storing it in a `_datadog` object in `context.clientContext.custom`. This must be set to True if you are invoking Lambda functions instrumented with datadog-lambda-python < v41 or datadog-lambda-js < v3.57.0. However, if set to True, invocations to Go or Java Lambda functions may fail.


I'd probably trim this down as well. From the user standpoint they just need to know that context management support has been added for newer lambda libraries and that there is a legacy configuration option in-case they wish to run an older version.

👍 for mentioning the specific versions of each library.

Would also be good to explicitly show the configuration options, something like:

Legacy support for older libraries is available with ddtrace.config.botocore.invoke_with_legacy_context = True or by setting the environment variable DD_BOTOCORE_INVOKE_WITH_LEGACY_CONTEXT=true

I'd like to see it explicitly mentioned that the propagation will break for users using datadog-lambda-python < v41 or datadog-lambda-js < v3.57.0 first and then mention the introduction of the legacy setting that can be used to support these older versions.

…ion_trace_propagation_format

…pagation_format' of github.com:DataDog/dd-trace-py into chris.agocs/update_botocore_direct_invocation_trace_propagation_format

Kyle-Verhoog

Just one more thought on the release note!

Kyle-Verhoog · 2021-07-21T15:32:29Z

releasenotes/notes/botocore-46f5bf04f64777b4.yaml

+---
+features:
+  - |
+    botocore: added `invoke_with_legacy_context` configuration setting, which is disabled by default. This configuration setting is to preserve a legacy behavior in which, when tracing a direct lambda invocation, trace context was propagated by storing it in a `_datadog` object in `context.clientContext.custom`. This must be set to True if you are invoking Lambda functions instrumented with datadog-lambda-python < v41 or datadog-lambda-js < v3.57.0. However, if set to True, invocations to Go or Java Lambda functions may fail.


I'd like to see it explicitly mentioned that the propagation will break for users using datadog-lambda-python < v41 or datadog-lambda-js < v3.57.0 first and then mention the introduction of the legacy setting that can be used to support these older versions.

…ion_trace_propagation_format

…pagation_format' of github.com:DataDog/dd-trace-py into chris.agocs/update_botocore_direct_invocation_trace_propagation_format

…ion_trace_propagation_format

Kyle-Verhoog

👏 thanks for bearing with my nitty documentation feedback @agocs!

* Update trace propagation format for botocore direct invocations * Update client context format in one other place * Add config for add_datadog_object * Refactor modify_client_context and inject_trace_to_client_context * black * default false * Add a test for the config flag * Document the config variable * Add release notes * add a bunch of newlines for flake8 * Defaut -> Default * add js and clientContext to wordlist * Update ddtrace/contrib/botocore/patch.py Mutate client_context_object instead of returning Co-authored-by: Kyle Verhoog <kyle@verhoog.ca> * Update ddtrace/contrib/botocore/patch.py Co-authored-by: Kyle Verhoog <kyle@verhoog.ca> * Trimmed down on excess documentation * rename config value to BOTOCORE_INVOKE_WITH_LEGACY_CONTEXT * run black * Remove clientContext form wordlist * Improve documentation * upgrades -> upgrade * re-write release notes * fix spelling of fnuctions Co-authored-by: Kyle Verhoog <kyle@verhoog.ca> Co-authored-by: Brett Langdon <brett.langdon@datadoghq.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Update trace propagation format for botocore direct invocations

20949bb

agocs added the do-not-merge/WIP label Jul 12, 2021

agocs requested a review from a team as a code owner July 12, 2021 18:33

agocs marked this pull request as draft July 12, 2021 18:33

Kyle-Verhoog reviewed Jul 12, 2021

View reviewed changes

Update client context format in one other place

3aca873

agocs added 10 commits July 12, 2021 16:58

Add config for add_datadog_object

90e07ff

Refactor modify_client_context and inject_trace_to_client_context

aac4b4f

black

e27caa5

default false

bde46be

Add a test for the config flag

7ca2a40

Document the config variable

4cbfa86

Add release notes

8c9ac10

add a bunch of newlines for flake8

537ca55

Defaut -> Default

81ffcfc

add js and clientContext to wordlist

0ac17e1

agocs marked this pull request as ready for review July 14, 2021 12:04

agocs removed the do-not-merge/WIP label Jul 14, 2021

Merge branch 'master' into chris.agocs/update_botocore_direct_invocat…

8deb3b6

…ion_trace_propagation_format

Kyle-Verhoog requested changes Jul 15, 2021

View reviewed changes

agocs and others added 6 commits July 15, 2021 11:12

Update ddtrace/contrib/botocore/patch.py

6dfb19f

Mutate client_context_object instead of returning Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

Update ddtrace/contrib/botocore/patch.py

b487f40

Co-authored-by: Kyle Verhoog <kyle@verhoog.ca>

Trimmed down on excess documentation

27bc779

rename config value to BOTOCORE_INVOKE_WITH_LEGACY_CONTEXT

12115cb

Merge branch 'chris.agocs/update_botocore_direct_invocation_trace_pro…

98477b6

…pagation_format' of github.com:DataDog/dd-trace-py into chris.agocs/update_botocore_direct_invocation_trace_propagation_format

run black

0306cfd

Merge branch 'master' into chris.agocs/update_botocore_direct_invocat…

7118727

…ion_trace_propagation_format

agocs requested a review from Kyle-Verhoog July 16, 2021 15:17

Remove clientContext form wordlist

39e3197

Kyle-Verhoog reviewed Jul 19, 2021

View reviewed changes

Kyle-Verhoog added this to the 0.51.0 milestone Jul 19, 2021

agocs and others added 4 commits July 21, 2021 11:13

Improve documentation

688bf70

Merge branch 'master' into chris.agocs/update_botocore_direct_invocat…

21dba2d

…ion_trace_propagation_format

upgrades -> upgrade

90f18a3

Merge branch 'chris.agocs/update_botocore_direct_invocation_trace_pro…

f0af564

…pagation_format' of github.com:DataDog/dd-trace-py into chris.agocs/update_botocore_direct_invocation_trace_propagation_format

Kyle-Verhoog requested changes Jul 21, 2021

View reviewed changes

agocs and others added 4 commits July 21, 2021 13:21

re-write release notes

16b85e8

Merge branch 'master' into chris.agocs/update_botocore_direct_invocat…

9efdea9

…ion_trace_propagation_format

fix spelling of fnuctions

fcc11bf

Merge branch 'chris.agocs/update_botocore_direct_invocation_trace_pro…

7eca5fb

…pagation_format' of github.com:DataDog/dd-trace-py into chris.agocs/update_botocore_direct_invocation_trace_propagation_format

brettlangdon approved these changes Jul 21, 2021

View reviewed changes

mergify bot added 2 commits July 21, 2021 21:52

Merge branch 'master' into chris.agocs/update_botocore_direct_invocat…

4e8b3fa

…ion_trace_propagation_format

Merge branch 'master' into chris.agocs/update_botocore_direct_invocat…

4830cdc

…ion_trace_propagation_format

Kyle-Verhoog approved these changes Jul 22, 2021

View reviewed changes

mergify bot merged commit f9bab03 into master Jul 22, 2021

mergify bot deleted the chris.agocs/update_botocore_direct_invocation_trace_propagation_format branch July 22, 2021 12:15

Kyle-Verhoog mentioned this pull request Jul 27, 2021

Incompatible with aws-lambda-java-libs #1934

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update trace propagation format for botocore direct invocations #2639

Update trace propagation format for botocore direct invocations #2639

agocs commented Jul 12, 2021

Kyle-Verhoog left a comment

agocs commented Jul 12, 2021 •

edited

Kyle-Verhoog commented Jul 12, 2021

agocs commented Jul 12, 2021

agocs commented Jul 14, 2021

Kyle-Verhoog left a comment

Kyle-Verhoog Jul 15, 2021

agocs Jul 15, 2021

Kyle-Verhoog Jul 15, 2021

agocs Jul 15, 2021

Kyle-Verhoog left a comment

Kyle-Verhoog Jul 19, 2021

Kyle-Verhoog Jul 19, 2021

brettlangdon Jul 21, 2021

Kyle-Verhoog Jul 19, 2021

Kyle-Verhoog Jul 21, 2021

Kyle-Verhoog left a comment

Kyle-Verhoog Jul 21, 2021

Kyle-Verhoog left a comment

	"botocore", "clientcontext_custom_add_datadog_object", default=False
	"botocore", "legacy_context", default=False

		@@ -24,6 +24,19 @@

		Default: ``True``

		.. py:data:: ddtrace.config.botocore['invoke_with_legacy_context']

		@@ -0,0 +1,4 @@
		---
		features:

Update trace propagation format for botocore direct invocations #2639

Update trace propagation format for botocore direct invocations #2639

Conversation

agocs commented Jul 12, 2021

Description

Reasoning

Backwards compatibility

Checklist

Kyle-Verhoog left a comment

Choose a reason for hiding this comment

agocs commented Jul 12, 2021 • edited

Kyle-Verhoog commented Jul 12, 2021

agocs commented Jul 12, 2021

agocs commented Jul 14, 2021

Kyle-Verhoog left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kyle-Verhoog left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kyle-Verhoog left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kyle-Verhoog left a comment

Choose a reason for hiding this comment

agocs commented Jul 12, 2021 •

edited