Performance improvements for better editor integration

[bbatsov]

This article/research on the state of Ruby formatters was shared with me earlier today - https://github.com/gurgeous/ruby-formatters

There are some interesting data points there - like how much slower we are then the competition when it comes to formatting a single file (I think that's mostly because RuboCop's bootstrapping is much more complex than that of the average formatter and I'm assuming that the editors just keep starting a fresh RuboCop process). Probably we should come up with some improvements for editor integration (e.g. a daemon mode) that would prevent RuboCop from constantly going through the painful process of processing the configuration and finding all the target files. If someone from the @rubocop-hq/rubocop-core would like to do some benchmarks and float some performance improvement ideas that'd be appreciated.

[jonas054]

Here's a simple benchmark to establish at least one fact. I made this change in exe/rubocop:

diff --git a/exe/rubocop b/exe/rubocop
index 3dacabebe..d1f381711 100755
--- a/exe/rubocop
+++ b/exe/rubocop
@@ -3,8 +3,10 @@
 
 $LOAD_PATH.unshift("#{__dir__}/../lib")
 
-require 'rubocop'
 require 'benchmark'
+require_time = Benchmark.realtime do
+  require 'rubocop'
+end
 
 cli = RuboCop::CLI.new
 result = 0
@@ -13,5 +15,5 @@ time = Benchmark.realtime do
   result = cli.run
 end
 
-puts "Finished in #{time} seconds" if cli.options[:debug] || cli.options[:display_time]
+puts "Finished in #{require_time} + #{time} seconds" if cli.options[:debug] || cli.options[:display_time]
 exit result

Result:

$ time ./exe/rubocop --display-time Gemfile
Inspecting 1 file
.

1 file inspected, no offenses detected
Finished in 0.6445538369989663 + 0.3408129759991425 seconds

real	0m1,081s
user	0m1,008s
sys	0m0,072s

We see here that around two thirds of the time is spent in require when inspecting a single file. So a solution involving some kind of daemon seems almost necessary. The linked article warns about the complexity of that kind of set-up. Is there some way we could make it easier to use by integrating the daemon functionality in RuboCop itself?

[marcandre]

I'm curious how much bootsnap would save. Clearly a daemon would be even faster though.

[jonas054]

My results are that a bootsnap wrapper taken from @hanachin's comment in #7082 reduces execution time by about a third. This is a smaller improvement than you get from rubocop-daemon, but it's easier to invoke.

[bbatsov]

I guess we can mention this in the docs, so at least more people are aware of this possibility. Still, that if we want to play well with editors we'll need to get much faster.

[marcandre]

Maybe we should integrate https://github.com/fohte/rubocop-daemon ?

[jonas054]

When I try out rubocop-daemon on the command line it seems to deliver what it promises, like inspection time for a single file around 0.2s vs 0.9s with just rubocop. So I'm not sure what the problems concerning editor integration are. Maybe the best way forward it to try to improve rubocop-daemon.

[bbatsov]

I'm guessing one practical problem is that most editors just shell out to RuboCop and getting users to install this daemon and editors to support it is going to be tricky. That's why I suggested looking into ways to address the problem directly with RuboCop. Obviously if we just bundle the daemon this would solve half the problem.

[bbatsov]

@gurgeous @fohte Feel free to share your thoughts as well.

[gurgeous]

Hi guys, thanks for including me. Glad you liked the writeup. rubocop-daemon is a fine tool and an excellent hack. We should pause and appreciate the work that's gone into creating it. We love it on my team. Having said that, there's room for improvement:

1. Install complexity is daunting. For example, integration with the most popular editor (vscode & vscode-ruby) requires manually creating a symlink. Updating RuboCop overwrites the symlink, oops. This is exacerbated by the opacity of the vscode-ruby plugin itself.
2. It's easy for rubocop-daemon to run the wrong version of RuboCop. If you have multiple versions of RuboCop installed and you aren't aware of RUBOCOP_DAEMON_USE_BUNDLER, you will accidentally run the wrong RuboCop half the time. This is difficult to debug.
3. What happens if you upgrade RuboCop while the daemon is running? You'll be stuck with a stale RuboCop since I don't think the daemon handles this case. Same thing can happen when upgrading Ruby. This happens every couple of months on my team.
4. Fails silently. Sudden lack of linting/formatting is surprisingly hard for the average person to notice. This has happened numerous times with my team.

Just picture what the average engineer is trying to do:

1. Is the problem vscode? Maybe try looking at dev tools. Restart vscode. Is it fixed?
2. Is it vscode-ruby? Muck with the settings. Don't forget there are two sets of settings, one for the user and one for the workspace. Try using the builtin vscode console to run RuboCop manually.
3. Is it the daemon? pkill the daemon and see if it starts working next time. Open the bash script and manually start logging things to a file in /tmp.
4. Is it RuboCop? Try running from the command line, somehow avoiding the symlink. If desperate, manually edit the RuboCop gem to see if you can figure out what broke.

Building this functionality into RuboCop would alleviate many of these issues, I think. A bit of logging would go a long way too, especially if it can be configured via rubocop.yml.

FYI we've had great luck with bootsnap too. We turned it on and basically forgot about it, but I'm confident it's dramatically sped up startup.

[koic]

I've opened #10706 to integrate rubocop-daemon.