Profiling

Job profiling is an amazing superpower for those who have mastered its complexities. Sidekiq makes it as easy as possible to profile your jobs.

Profiling allows you to see the code executed by your job and how much time it took to execute. This makes it really easy to find slow spots in your job where you can optimize, e.g. you might find a slow query which is missing a database index.

Teaching you how to use a profiler is beyond the scope of this wiki page but I recommend watching this Youtube video by the author of Vernier:

Prepare

Add Vernier to your Gemfile:

gem "vernier"

Run

1. Start Sidekiq.
2. Open a console.
3. Pick a job type you wish to profile and kick off a job with the special profile token:

$ bundle exec rails console
> HardJob.set(profile: "token").perform_async("mike", 2, nil)

The purpose of token is to associate profile data to you, making it easier to find your profiles if other team members are also profiling their jobs. Typically I just use "mike" as my token. Profiling works fine with Rails and Active Jobs.

Notes

• Vernier requires Ruby 3.2.1+.
• Your profiling data is uploaded to profiler.firefox.com when you click the View button in the Web UI.
• Remember to profile in production mode. Development mode will add a lot of noise and won't reflect production exactly.

API

You can access profile data via Sidekiq::ProfileSet:

ps = Sidekiq::ProfileSet.new
ps.each do |profile|
  p profile
end

The profiled data is stored in Redis as a Hash with key "token-jid", e.g. "mike-1234567890abcdef".

#{token}-#{jid}:
{
  started_at: <timestamp>,
  token: <token>,
  data: <gzip'd JSON>,
  jid: <jid>,
  type: <jobtype>,
  sid: <storeid> # the profile's storage id at profiler.firefox.com
}

Vernier's profiler output can be large so Sidekiq uses gzip to compress it for storage. Be careful not to profile every job and quickly fill up your Redis! Profiling data expires after 24 hours.

Profiling only saves data if the job is successful. Failures will retry as normal.

Web UI

The Web UI provides a new Profiles tab with a listing of all profile recordings in Redis. You can click the View button and see your profile in the Firefox Viewer. You can click the Data button to see or download the profile's raw JSON.

Profile Data and Security

Firefox Profiler is a complex JavaScript SPA which can't be maintained or distributed by the Sidekiq project. Instead Sidekiq uses the public instance at profiler.firefox.com which is run and maintained by Mozilla. When you click the View button in the Web UI, your profile data is uploaded to p.f.c for viewing. Profiling data generally does not contain anything sensitive but it does contain the names of your application classes and methods. It should never contain any application data.

I'm not aware of any official data retention policy from Mozilla for your uploaded profiles. Your profiling data does not leave your system until you click View. You can use the Data button to access the raw JSON, should you wish to run your own Firefox Profiler instance locally.