feature: add distributed tracing (inferred spans) [SVLS-3154] #80

duncanista · 2023-06-26T15:04:54Z

What does this PR do?

Infer spans through the Datadog Lambda Extension.

The inferred services are:

API Gateway

Lambda URL (not working due to Extension serialization issue)
SQS

SNS

Kinesis

EventBridge (not working due to Extension serialization issue)
S3

DynamoDB

SNS-SQS

HTTP Propagation:

Motivation

Ruby is the only language where we don't have support for this (yet).

Testing Guidelines

Added unit tests for some methods, since most of the span generation is done through the extension.
Tested manually to see infer span generation.

Additional Notes

Trace propagation needs to be done at the Tracer level.
There are multiple issues in the Datadog Extension around serialization, which affect: Lambda URL, EventBridge, and Trace parenting.
Datadog Extension is not parenting properly due to how ruby sends the trace headers. This will be addressed in the Extension.
Also upgraded dd-trace-rb to v1.14.0 to be able to get an internal constant defined on latest versions.
SVLS-3959

Types of changes

Bug fix
New feature
Breaking change
Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

This PR's description is comprehensive
This PR contains breaking changes that are documented in the description
This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
This PR impacts documentation, and it has been updated (or a ticket has been logged)
This PR's changes are covered by the automated tests
This PR collects user input/sensitive content into Datadog

previously, the hello path in the extension would signal that the tracer would do every calculation, this is not intended, since we want the extension to do inferred spans

…o jordan.gonzalez/support-inferred-spans

… to `true`

added some debug methods to test on lambda

duncanista · 2023-08-08T01:45:37Z

integration_tests/snapshots/logs/async-metrics_ruby27.log

 Processed APIGateway request
 Processed SNS request
 Processed SQS request
 START
 START
 START
+W, [XXXX]  WARN XXXX[ddtrace] Unable to patch Datadog::Tracing::Contrib::Aws::Integration (Available?: false, Loaded? false, Compatible? false, Patchable? false)


Issue where AWS Lambda + Layer can't find AWS SDK gems.

Support case with AWS is in progress.

This will have to stay for now.

lib/datadog/lambda.rb

…nning

…o jordan.gonzalez/support-inferred-spans

also removed unnecessary tests

lib/datadog/lambda/trace/constants.rb

because if this is changed, there is no way for us to know

needed for the dependency to skip tracing an http request with metadata tag

lib/datadog/lambda.rb

lib/datadog/lambda/trace/listener.rb

lib/datadog/lambda/utils/extension.rb

astuyve

one note, rubocop shouldn't be an issue anymore. Otherwise LGTM!

lib/datadog/lambda/trace/listener.rb

duncanista added 23 commits June 22, 2023 17:04

add DD_SPAN_ID_HEADER

1f6eed9

update on_start and on_end to create and finish trace

e223f3d

update listener on_start

e03e522

update self.wrap

d332025

hit start and end routes in extension for inferred span generation

0a02f79

add check to see if extension is running

add3e5a

change extension_running logic

d0c42bd

previously, the hello path in the extension would signal that the tracer would do every calculation, this is not intended, since we want the extension to do inferred spans

update extension URIs

793b3cb

Merge branch 'main' of ssh://github.com/DataDog/datadog-lambda-rb int…

50ad69c

…o jordan.gonzalez/support-inferred-spans

enable aws sdk instrumentation by default

f4b965e

update redundant begin-rescue

d64ae7b

revert extension url change

9c6de7e

update trace listener to only initialized if DD_TRACE_ENABLED is se…

620242a

… to `true`

add method to get trace context as headers

6fd78de

clean extension.rb

078ee73

update how context is sent in headers

76c60bc

typo

98633d5

return headers on context to headers method

ce48e00

added some debug methods to test on lambda

remove debug logs

b703a47

update snapshots

579218f

lint

af6a619

fix usesless assignment

e9f14bc

remove on_wrap method

8f837de

duncanista commented Aug 8, 2023

View reviewed changes

lib/datadog/lambda.rb Outdated Show resolved Hide resolved

duncanista added 5 commits August 8, 2023 17:31

make start and end request to return early if the extension is not ru…

321ebff

…nning

Merge branch 'main' of ssh://github.com/DataDog/datadog-lambda-rb int…

e4a4da3

…o jordan.gonzalez/support-inferred-spans

Merge branch 'main' of ssh://github.com/DataDog/datadog-lambda-rb int…

c2b95eb

…o jordan.gonzalez/support-inferred-spans

allow AWS managed services tracing to be disabled

c22ea4b

update trace context from headers

f1a62c6

duncanista added 13 commits September 28, 2023 16:00

add header to not trace start/end requests

704c4ba

delegate injection/extraction to tracer

87f350b

fix header

d2061cc

move header to not be initheader

4b4fd24

update start request with host and port

7acb052

reset to manual injection/extraction and change resource name

ba81145

set resource name to the skippable resource

78f1b6e

debug logs

72e0d80

update resource name

65b87dc

make continue only if trace digest is not nil

0120a14

delegate extraction and injection to tracer properly

6b3d073

also removed unnecessary tests

lint

75fcdd2

typo

4885016

duncanista commented Oct 2, 2023

View reviewed changes

lib/datadog/lambda/trace/constants.rb Outdated Show resolved Hide resolved

duncanista added 5 commits October 2, 2023 15:51

add skip tracing headers directly from dd-trace-rb

b5d9c36

because if this is changed, there is no way for us to know

upgrade version of dd-trace-rb

f200a97

needed for the dependency to skip tracing an http request with metadata tag

upgrade version in Dockerfile for build

4e3b102

update snapshots

40521ef

update correct snapshots

a5a5c95