Squash nested sinatra/rack request span #2217

TonyCTHsu · 2022-08-12T08:42:32Z

Motivation

For Sinatra/Rack instrumentation, the middleware stack architecture renders nested flamegraph for span. Although it is technically correct, the user experience with such UI is suboptimal.

What does this PR do?

Instead of instrumenting nested middleware stack of Sinatra or Rack, the implementation changes to only instrument the top of the stack. This means the UI would display only one rack.request and sinatra.request.

Some changes followed after skipping the subsequent middlewares:

sinatra.app.name tag is removed from the sinatra.request span, but still present for sinatra.route span. Since the app name only make sense when a route is matched by its individual app.
When no route matched, sinatra.request span would no longer tagged with sinatra.route.path, since none of the route matched(sinatra.route span is absent as well).

Other improvement: Configuration is all it needs to instrument Sinatra, no longer need to register for each Sinatra app.

For Classic Sinatra app

For Modular Sinatra app with Sinatra::Router

class Acme < Sinatra::Base
  # # Use Sinatra::Router to mount different modular Sinatra applications
  use Sinatra::Router do
    mount ::Health
    mount ::Basic
  end

For Modular Sinatra app chaining with use

class Acme < Sinatra::Base
  # # Use Sinatra App as middleware
  use Health
  use Basic

lib/datadog/tracing/contrib/sinatra/patcher.rb

TonyCTHsu · 2022-08-12T12:48:08Z

lib/datadog/tracing/contrib/sinatra/ext.rb

-          RACK_ENV_MIDDLEWARE_START_TIME = 'datadog.sinatra_middleware_start_time'.freeze
-          RACK_ENV_MIDDLEWARE_TRACED = 'datadog.sinatra_middleware_traced'.freeze


Should be careful renaming/removing constants like this. This may constitute a breaking change. Might be safer to flag this as deprecated and keep the existing constants for now.

lib/datadog/tracing/contrib/sinatra/env.rb

lib/datadog/tracing/contrib/sinatra/tracer.rb

TonyCTHsu · 2022-08-12T12:49:45Z

lib/datadog/tracing/contrib/sinatra/env.rb

-            request_span = env[Ext::RACK_ENV_REQUEST_SPAN]
-            request_span && request_span[app]
+          def datadog_span(env)
+            env[Ext::RACK_ENV_SINATRA_REQUEST_SPAN]


We no longer stored multiple spans, but one.

lib/datadog/tracing/contrib/sinatra/tracer_middleware.rb

TonyCTHsu · 2022-08-12T12:51:30Z

spec/datadog/tracing/contrib/sinatra/tracer_spec.rb

@@ -52,18 +52,6 @@
        erb :msg, locals: { msg: 'hello' }
      end

-      get '/erb_manual_injection' do


While reviewing #2217 where we fixed the sinatra documentation not to mention the outdated `use`, I decided to make a quick check if we had any more such leftovers, and it turns out we did. This PR fixes the two leftover mentions of `use` that I found that should be replaced by `instrument`. I did not change the sinatra docs to avoid a conflict with #2217.

ivoanjo

Left a few notes from my first pass at the PR

integration/apps/sinatra2-modular/app/basic.rb

lib/datadog/tracing/contrib/rack/middlewares.rb

integration/apps/sinatra2-modular/app/parent.rb

ivoanjo · 2022-08-22T13:33:24Z

lib/datadog/tracing/contrib/sinatra/env.rb

+            if use_script_names
+              env[::Rack::SCRIPT_NAME].to_s + path


I'm curious about this change -- we previously used request.script_name but now we directly parse it from the env. Could you add some context on why this option is needed?

This does not change the behaviour of our instrumentation.

Sinatra::Request is a children of Rack::Request, while #script_name is still delegate to rack env with SCRIPT_NAME. I changed the implementation depends on what kind of object I could access within the current scope.

This option is introduced here.

Personally, I think this option should move to rack instrumentation and enabled(true) by default.

Personally, I think this option should move to rack instrumentation

Actually, would it impact rack? I thought pure-rack spans get named based on the verb + status code (e.g. "GET 200") so how would we represent the script name there?

and enabled(true) by default.

Fully agree on this. Perhaps worth looking into for 1.4.0 or 1.5.0?

This would be a breaking change(resource change), so probably plan for the next major release.

Well, I wouldn't call it a breaking change. As the famous xkcd comic states:

...every change has the potential to break someone. So if we applied it in the extreme every change is a potential breaking change. New integration? There's more spans now -- it's different. Fixed a typo in a resource? Someone relied on the typo -- it's different.

So I wouldn't consider that a breaking change, or at least one that would need waiting until the next major version.

But... this is just a suggestion, e.g. you're welcome to disagree and since tracing is your domain, you have more say in it than I do ;)

lib/datadog/tracing/contrib/sinatra/patcher.rb

delner

Some nuances to this change.

Biggest concern is it may break AppSec (which had some special use for Rack). Let's verify with @lloeki.

2nd concern is: should we always short-circuit Rack instrumentation if a Rack span is already present? This seems unduly restrictive, possibly undesirable, even if its good for Sinatra. Let's clarify this is what we want.

delner · 2022-08-23T18:33:16Z

lib/datadog/tracing/contrib/rack/middlewares.rb

@@ -54,16 +54,18 @@ def call(env)
            # Find out if this is rack within rack
            previous_request_span = env[Ext::RACK_ENV_REQUEST_SPAN]

+            return @app.call(env) if previous_request_span


Does this work okay with AppSec? We're now short circuiting Rack instrumentation whenever there was a Rack span before. My understanding was AppSec produces its own Rack span as well... would it get skipped? CC @lloeki

Oh my I think this is going to be a problem. In fact I think the whole PR might be.

After discussion, it should be fine with a passing test suite.

lib/datadog/tracing/contrib/sinatra/env.rb

delner · 2022-08-23T18:37:14Z

lib/datadog/tracing/contrib/sinatra/ext.rb

-          RACK_ENV_MIDDLEWARE_START_TIME = 'datadog.sinatra_middleware_start_time'.freeze
-          RACK_ENV_MIDDLEWARE_TRACED = 'datadog.sinatra_middleware_traced'.freeze


Should be careful renaming/removing constants like this. This may constitute a breaking change. Might be safer to flag this as deprecated and keep the existing constants for now.

lib/datadog/tracing/contrib/sinatra/tracer.rb

lib/datadog/tracing/contrib/sinatra/tracer_middleware.rb

marcotc · 2022-10-24T19:42:33Z

This PR returns the ddtrace Sinatra instrumentation to a similar visibility granularity to ddtrace 0.x series.

Sinatra traces will look like this after the changes, as explained by @TonyCTHsu:

This is the best representation we are able to create of a Sinatra request, given all the possible internal complex interactions between Sinatra and Rack, and it looks as close to the user application as possible.

This looks very good to me.

As far as I understand, this seems to cause issues to AppSec, is that right @lloeki? If so, what are the issue we are worried about here? We still have Rack and Sinatra spans, plus a nested sinatra.route span. I'm not sure I see any missing information here that would prevent us from correctly implementing AppSec here.

marcotc · 2022-11-07T18:21:57Z

integration/apps/sinatra2-modular/app/acme.rb

@@ -1,6 +1,7 @@
 require 'sinatra/base'
 require 'sinatra/router'
 require 'ddtrace'
+require 'pry'


There a few pry references left in this PR. Should be be kept?

marcotc · 2022-11-07T18:22:20Z

lib/datadog/tracing/contrib/sinatra/tracer.rb

-              # TODO: `route` should *only* be populated if @datadog_route is defined.
-              # TODO: If @datadog_route is not defined, then this Sinatra app is not responsible
-              # TODO: for handling this request.
-              # TODO:
-              # TODO: This change would be BREAKING for any Sinatra app (classic or modular),
-              # TODO: as it affects the `resource` value for requests not handled by the Sinatra app.
-              # TODO: Currently we use "#{method} #{path}" in such aces, but `path` is the raw,
-              # TODO: high-cardinality HTTP path, and can contain PII.
-              # TODO:
-              # TODO: The value we should use as the `resource` when the Sinatra app is not
-              # TODO: responsible for the request is a tricky subject.
-              # TODO: The best option is a value that clearly communicates that this app did not
-              # TODO: handle this request. It's important to keep in mind that an unhandled request
-              # TODO: by this Sinatra app might still be handled by another Rack middleware (which can
-              # TODO: be a Sinatra app itself) or it might just 404 if not handled at all.
-              # TODO:
-              # TODO: A possible value for `resource` could set a high level description, e.g.
-              # TODO: `request.request_method`, given we don't have the response object available yet.


Yes! Nice removal!

marcotc

🚀

TonyCTHsu added 3 commits August 12, 2022 10:06

Skip nested rack/sinatra middleware instrumentation

4a765ff

Refactor

4d57dac

Remove unuse comment

1d88b01

TonyCTHsu self-assigned this Aug 12, 2022

TonyCTHsu added 2 commits August 12, 2022 12:58

Skip sinatra.request route_path is none app matched

504c0b1

Fix loading tracer

6ec77d6

TonyCTHsu force-pushed the feature/sinatra-rack branch from 97de807 to 6ec77d6 Compare August 12, 2022 11:15

Change documentation

25fd552

TonyCTHsu force-pushed the feature/sinatra-rack branch from 41a4603 to 25fd552 Compare August 12, 2022 12:07

TonyCTHsu marked this pull request as ready for review August 12, 2022 12:46

TonyCTHsu requested a review from a team August 12, 2022 12:46