Trace Sidekiq server internal events (heartbeat, job fetch, and scheduled push) #1685

agrobbin · 2021-09-15T00:20:11Z

These 3 areas of the open source version of Sidekiq are not wrapped in high-level spans, resulting in a lot of one-off Redis spans that are not related to one another in any way.

The most important change to call out here is the bump in the supported Sidekiq version. There were some significant underlying code changes between Sidekiq v3 and v4, and with Sidekiq 3.5.4 being over 5 years old (released 2016-01-13), that seemed to be a reasonable decision.

That said, I'm open to keeping v3 support, with this new internal tracing focused only on v4+.

Closes #1673.

agrobbin · 2021-09-15T20:08:13Z

Sorry for all of the CI failure noise here folks, it took some time to get things all squared away across various Sidekiq (and Ruby) versions!

codecov-commenter · 2021-09-15T20:08:18Z

Codecov Report

Merging #1685 (b01ea9c) into master (ecfc3f3) will decrease coverage by 0.10%.
The diff coverage is 66.44%.

@@            Coverage Diff             @@
##           master    #1685      +/-   ##
==========================================
- Coverage   98.31%   98.20%   -0.11%     
==========================================
  Files         927      933       +6     
  Lines       44662    44804     +142     
==========================================
+ Hits        43909    44000      +91     
- Misses        753      804      +51

Impacted Files	Coverage Δ
...b/sidekiq/server_internal_tracer/scheduled_push.rb	`46.15% <46.15%> (ø)`
...ontrib/sidekiq/server_internal_tracer/heartbeat.rb	`50.00% <50.00%> (ø)`
...ontrib/sidekiq/server_internal_tracer/job_fetch.rb	`50.00% <50.00%> (ø)`
...b/sidekiq/server_internal_tracer/heartbeat_spec.rb	`62.50% <62.50%> (ø)`
...b/sidekiq/server_internal_tracer/job_fetch_spec.rb	`62.50% <62.50%> (ø)`
spec/ddtrace/contrib/sidekiq/support/helper.rb	`75.55% <65.62%> (-24.45%)`	⬇️
...ekiq/server_internal_tracer/scheduled_push_spec.rb	`71.42% <71.42%> (ø)`
lib/ddtrace/contrib/sidekiq/ext.rb	`100.00% <100.00%> (ø)`
lib/ddtrace/contrib/sidekiq/integration.rb	`100.00% <100.00%> (ø)`
lib/ddtrace/contrib/sidekiq/patcher.rb	`100.00% <100.00%> (ø)`
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ecfc3f3...b01ea9c. Read the comment docs.

marcotc · 2021-09-16T21:39:40Z

Thank you for the PR, @agrobbin!

Sorry for all of the CI failure noise here folks

No need to apologize! This is what CI is here for :) Feel free to push as many times as you'd like.

marcotc

Very nice work, @agrobbin! I've left a few comments for you to take a look.

marcotc · 2021-09-16T22:08:10Z

lib/ddtrace/contrib/sidekiq/patcher.rb

+        def patch_server_internals
+          # Sidekiq < 5.2.4 (before mperham/sidekiq@ddb0c8b3) doesn't require this until
+          # too late in the process (after `CLI#run` has been called)
+          require 'sidekiq/launcher'


We normally avoid requiring external gem code in ddtrace, as to not affect gem initialization order in the host application. Some gems are picky with initialization order.

Also, if I'm reading this correctly, 'sidekiq/launcher' is only needed when running Sidekiq jobs, but not when simply enqueuing them. This means that many applications that only enqueue jobs would actually never need to require 'sidekiq/launcher'.

With that in mind, what do you think of removing the require here and instead making

patch_server_heartbeat patch_server_job_fetch patch_server_scheduled_push

conditional on having sidekiq/launcher loaded? It seems to me that this would work for users of Sidekiq >= 5.2.4, which is over 2 years old, but not force us to explicitly load it for all users of ddtrace.

I'm more than happy to only do this for Sidekiq >= 5.2.4, but just to be clear, this won't attempt to load sidekiq/launcher for all users of dd-trace, only those who call use :sidekiq in their configuration. Does that change your opinion at all @marcotc?

only those who call use :sidekiq in their configuration

For web applications, for example, they will have use :sidekiq because they want to capture MyJob.perform_async calls. But these applications will never require sidekiq/launcher normally. Does that make sense?

Sure, but we were only ever requiring sidekiq/launcher inside a Sidekiq.configure_server block, so it shouldn't have had a significant impact. Either way though, I've removed it!

lib/ddtrace/contrib/sidekiq/server_internal_tracer/heartbeat.rb

lib/ddtrace/contrib/sidekiq/server_internal_tracer/job_fetch.rb

lib/ddtrace/contrib/sidekiq/server_internal_tracer/scheduled_push.rb

marcotc · 2021-09-16T22:28:57Z

Appraisals

-    gem 'sidekiq', '~> 3.5.4'
+    gem 'sidekiq', '~> 4.0.0'


Our test suite runs for all major Sidekiq versions, from 3.5 to 6.2.

I noticed that you needed to bump up only Ruby 2.1, which uses Sidekiq 3.5.4.

What's the impact of leaving this 3.5.4 support for Sidekiq? Is it possible to keep support at 3.5.4, while dynamically skipping new features introduced in this PR if necessary?

We can definitely do that! Out of curiosity, is there a particular rule of thumb DataDog follows for backwards compatibility? That'd be great to know for any future contributions.

We always try to not break existing behaviour: if it was working in version X, should work in version X+1.

If we want to improve things, but these improvements only work with newer versions of the integrated library (Sidekiq in our case), it's reasonable to have separate code paths: one for old behaviour, another for new and improved behaviour. We normally only do this for brand new features, not for two versions of an existing feature (unless strictly technically required).

So, for example, adding a few additional spans or additional tags for only more recent versions of Sidekiq is ok. But having, for example, the span.resource for the main Sidekiq span changing depending on Sidekiq version is not great, as people will upgrade Sidekiq and all of the sudden their instrumentation has changed as well, breaking dashboards and monitors.

marcotc · 2021-09-16T22:30:44Z

spec/ddtrace/contrib/sidekiq/support/helper.rb

-      include Sidekiq::Worker
-      def perform; end
-    end)
+module SidekiqServerExpectations


This is super cool!

Thanks! It took a number of iterations to finally find the right combination of things that make this work, so hopefully it can be useful going forward!

spec/ddtrace/contrib/sidekiq/support/helper.rb

agrobbin · 2021-09-17T00:23:07Z

@marcotc I just pushed up an update that limits these server internal traces to SIdekiq v5.2.4+. Let me know what you think of my approach.

marcotc · 2021-09-17T22:26:28Z

Everything looks great, @agrobbin!

One last thing: did you get a chance to try this in your development/staging environment, to see if the new traces look good to you as a Sidekiq user?

…uled push) These 3 areas of the open source version of Sidekiq are not wrapped in high-level spans, resulting in a *lot* of one-off Redis spans that are not related to one another in any way.

agrobbin · 2021-09-18T02:37:10Z

@marcotc I just deployed it to our staging environment, and things look good! I've attached some screenshots of the traces:

marcotc

Thank you so much, @agrobbin!

agrobbin · 2021-10-04T23:14:12Z

Awesome @marcotc, looking forward to seeing this released!

marcotc · 2021-10-06T21:36:46Z

@agrobbin, thank you again for all your contributions. 0.53.0 was just released, including the changes in this PR!

agrobbin requested a review from a team September 15, 2021 00:20

agrobbin force-pushed the sidekiq-internal-traces branch 11 times, most recently from 65b0f56 to fc2a28f Compare September 15, 2021 19:59

marcotc reviewed Sep 16, 2021

View reviewed changes

marcotc self-assigned this Sep 16, 2021

agrobbin force-pushed the sidekiq-internal-traces branch 2 times, most recently from 930447a to 38a1310 Compare September 17, 2021 00:22

agrobbin force-pushed the sidekiq-internal-traces branch from 38a1310 to 6752b5d Compare September 17, 2021 00:30

trace Sidekiq server internal events (heartbeat, job fetch, and sched…

b01ea9c

…uled push) These 3 areas of the open source version of Sidekiq are not wrapped in high-level spans, resulting in a *lot* of one-off Redis spans that are not related to one another in any way.

agrobbin force-pushed the sidekiq-internal-traces branch from 6752b5d to b01ea9c Compare September 18, 2021 02:13

marcotc approved these changes Oct 4, 2021

View reviewed changes

marcotc merged commit 13651e2 into DataDog:master Oct 4, 2021

marcotc added community Was opened by a community member integrations Involves tracing integrations labels Oct 4, 2021

github-actions bot added this to the 0.53.0 milestone Oct 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trace Sidekiq server internal events (heartbeat, job fetch, and scheduled push) #1685

Trace Sidekiq server internal events (heartbeat, job fetch, and scheduled push) #1685

agrobbin commented Sep 15, 2021

agrobbin commented Sep 15, 2021

codecov-commenter commented Sep 15, 2021 •

edited

Loading

marcotc commented Sep 16, 2021

marcotc left a comment

marcotc Sep 16, 2021

agrobbin Sep 16, 2021

marcotc Sep 17, 2021

agrobbin Sep 18, 2021

marcotc Sep 16, 2021

agrobbin Sep 16, 2021

marcotc Sep 17, 2021

marcotc Sep 16, 2021

agrobbin Sep 16, 2021

agrobbin commented Sep 17, 2021

marcotc commented Sep 17, 2021

agrobbin commented Sep 18, 2021

marcotc left a comment

agrobbin commented Oct 4, 2021

marcotc commented Oct 6, 2021

Trace Sidekiq server internal events (heartbeat, job fetch, and scheduled push) #1685

Trace Sidekiq server internal events (heartbeat, job fetch, and scheduled push) #1685

Conversation

agrobbin commented Sep 15, 2021

agrobbin commented Sep 15, 2021

codecov-commenter commented Sep 15, 2021 • edited Loading

Codecov Report

marcotc commented Sep 16, 2021

marcotc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agrobbin commented Sep 17, 2021

marcotc commented Sep 17, 2021

agrobbin commented Sep 18, 2021

marcotc left a comment

Choose a reason for hiding this comment

agrobbin commented Oct 4, 2021

marcotc commented Oct 6, 2021

codecov-commenter commented Sep 15, 2021 •

edited

Loading