Collect success stories

[jvns]

I'd like stories from people who are using rbspy. Have you used rbspy successfully to fix a performance problem? What kind of performance problem? How did rbspy help?

This helps with developing rbspy -- if we can see how people are using it, it's easier to figure out what other features might be useful to add!

comment on this issue if you have a story to share!

[jvns]

from twitter: https://twitter.com/palkan_tula/status/955450943939776512

I used it to profile a test run on CI: some tests suddenly became very slow; I connected to the container thru SSH, downloaded rbspy and took a couple of snapshots while tests were running; that was enough to find the cause of the problem)

[baloo]

Just debugged a puppet run on a node that was very slow when parsing a catalog. With rbspy I could see which provider was so slow, and fix it.

[segiddins]

CocoaPods/CocoaPods#7348 (comment) -- sped up pod install times on our massive app by 50% (translating to roughly 90 seconds saved)

[parkr]

@jvns We used rbspy to find a silly bug where we were doing crazy array manipulations unnecessarily! jekyll/jekyll#6730

[Iristyle]

Not to derail this thread, but was the provider yours @baloo or one included with Puppet?

[breunigs]

json_schema is ~1/3 faster and the bottlenecks were mostly identified by rbspy. Thanks!

[zetaben]

We found pretty serious performance improvement in our ruby agent partly thanks to rbspy. Having a flamegraph generated on a repeating transaction allowed us to look into the application with a smaller resolution than the advertised 10ms. Some of these optims even concerned operation that individually take less than a millisecond but were happening much to often and it many places.

Also the cheer ease of use of the tool (give it a pid, CTRL+C when done & you have your flamegraph) is completely awesome !

[andrewhampton]

I do a lot of work on a rails app that's over 10 years old. The extensive test suite is highly parallelized, and took about 14m to run in CI. However, it was up to about 1hr 50m of CPU time across all cores. I used rbspy record bundle exec rspec ... to run about 10m worth of tests to see if anything stood out.

It revealed we had not properly configured the scrypt gem for the test environment. It was taking the default 200ms to encrypt the password every time a user was created. After proper configuration, the 10m worth of plummeted to under 3m. The CPU time for the entire suite dropped from by nearly an hour, which translated to about 3m shaved off every parallelized run.

Thanks @jvns for the amazing tool!

[cheerfulstoic]

I've been using rbspy lately to debug sidekiq procesess and it works on many jobs, which has been really great. stackprof has generally been my go-to for that sort of thing, but rbspy is great, of course, because I don't need to restart the process to start profiling. But really best of all is the flamegraph. It feels like the perfect visualization for making sense of all of the data about how much time was spent in a method vs "under" a method.

I'm not sure if flamegraphs are available in other tools, but the ease and accessibility of them in rbspy brought them to me for the first time and for that I'm definitely going to keep using it. Thanks!

[thorsteneckel]

We over at Zammad had a massive performance issue in one of our customers installations. We made several optimizations by hand but never found the cause. Weeks passing by. Frustration was on a peak. The customer nearly lost the patience. However, I remembered about rbspy and decided to give it a try. Since it's production save it was easy as 1 2 3 to get it running and pointing to the exact line causing it. It was a deep recursion without proper handling for one certain Rails Model only. Less than 5 minutes of work to find it. My boss was (and still is) amazed. rbspy is now the number 1 tool in our toolbox for such cases. It helped us out multiple times since then. Thanks for your amazing work!

[shaldengeki]

We've used rbspy twice now at Grand Rounds to successfully resolve production incidents. In both cases, we used it to inspect the runtime behaviour of a misbehaving Rails webserver process in production by generating an icicle graph, which immediately identified the problem, saving us what would've likely been hours of downtime spent trying to reproduce the issue elsewhere or blindly applying patches.

This is an amazing tool and we'll continue to use it here; thank you for making this possible!

[nathanhinchey]

We have a very old Rails app that was taking over 10 seconds to load, and my boss heard about rbspy in a podcast. So then I listened to @jvns on the FLOSS Weekly podcast, and tried out rbspy. Turns out the entire problem was in one extremely inefficient function. With rbspy it took five minute to diagnosis the problem, which only took half an hour to fix.

Bonus, I got to learn about profilers!

[kdole]

rbspy is awesome! We occasionally have performance issues that only show up after a process has been running in production for an hour or more. rbspy just allowed us to easily diagnose one of those.

BTW, if you are getting a "No stack counts found" error, you may just need to run rbspy as root.

[petethepig]

Some things I used rbspy for:

• static performance analysis. Sometimes it's some function you thought was quick and turns our it's not, sometimes it's some db call that's taking more time than you would expect.
• debugging performance issues on live servers: webservers, background jobs, CI workers.

I love how versatile rbspy is. I can both get a very high level summary of what's going on, but also if I need to I can easily figure out exactly which parts of the code are being slow. I guess to be fair, this is mostly thanks to flamegraphs. But it's really nice to have a tool that you can generate flamegraphs with so easily.

Also, speaking of static analysis, we use this little snippet to run rbspy from within our ruby scripts. It's loaded automatically in production console environment:

def rbspy(local_filepath)
  Dir.mktmpdir do |tmpdir|
    svg_file_path = File.join(tmpdir, "rbspy-#{SecureRandom.hex}")
    cmd = "rbspy record --silent --format flamegraph --file #{svg_file_path} --pid #{Process.pid}"
    pid = Kernel.spawn(cmd, :out => "/dev/null", :err => "/dev/null")
    begin
      yield
    ensure
      Process.kill(:INT, pid)
      Process.waitpid2(pid)
      save_file(local_filepath, File.binread(svg_file_path))
    end
  end
end

You can wrap your slow code in a block and get nice flamegraphs:

rbspy("slow-code.svg") do
  slow_code()
end

[wodin]

I've used rbspy to troubleshoot performance issues in a big, old, hairy Rails app.
I've also used rack-mini-profiler before, but I find the flamegraphs from rbspy easier to use.

[ric2b]

I've used it after running bundle exec rspec --profile 10 (tells you the 10 slowest tests in your test suite) to find out why the slowest tests were so slow.
Turns out 9 of those 10 were testing very similar things, I profiled one of them, identified a step that was done over and over (parsing a specific file) that could instead be cached.
Result: our test suite is ~30% faster now!

[ilyazub]

rbspy was a starting point to find performance issues in Nokogiri which led to a 2-4x speed up between v1.10.0 and v1.11.0.

I've described the whole story in a blog post.

[davebarrau]

Used rbspy to improve performance in Middleman, a static site generator. Also blogged about it.