Skip to content

If the OS allows it, build the docs (ri and rdoc) in a forked background process #352

Merged
merged 4 commits into from Jul 6, 2012

6 participants

@alexch
alexch commented Jul 6, 2012

ri and rdoc are amazingly great, but sadly, many people are impatient, so the Ruby world is filled with advice to add --no-rdoc and --no-ri everywhere, and people are then forced to use their web browsers and visit ruby-doc.org when a quick "ri titleize" would suffice, with much less risk of flow interruption.

This patch attempts to resolve this tension between install-time and dev-time impatience by forking a background process to build the docs after the gems have been installed. The only risk will be if someone tries to use ri or rdoc during the seconds or minutes immediately after the install, which is unlikely and at any rate, there are enough messages that people should realize the docs aren't quite built yet. And all that would happen is that the docs wouldn't be found; no risk of corruption or even misleading errors as far as I can tell.

The code is fairly elegant but it currently relies on the fact that the only user of the done_installing_hooks is the documentation builder. If any other code wants to add hooks then we'd have to make it a little more configurable -- possibly allowing different status messages, possibly adding a new hook array for just backgroundable tasks and restoring done_installing_hooks to just run synchronously like before.

Is there a need to add a "--no-background" option? I can't think of a real use case. If we're on Windows (or another forkless Ruby) it'll revert to synchronous. Perhaps rubygems.org needs to build docs synchronously, but if it's using the RDoc code directly this patch won't affect it.

One thing I feel strongly about is that backgrounding doc gen should be the default behavior. If we hide this behind an option then we're just perpetuating the original problem -- the perception that "gem install" is slow so you should just not install docs. (Which you actually should.)

There are no unit tests yet. It's kind of difficult to unit test fork but if you insist I could try to give it a go. Maybe we'd mock fork just to assert that it was called without actually forking anything during the test.

I just tried a clean install of rails and here are some stats:

time to install 2 dozen gems: 13sec
time to build docs for same: 238sec

So I guess it's no wonder people prefer --no-rdoc...

@travisbot

This pull request fails (merged 322fac3 into f65ea70).

@alexch
alexch commented Jul 6, 2012

Interesting. Those tests are failing because the temporary dir no longer exists by the time the background process runs. Looks like I need to worry about unit tests after all.

@travisbot

This pull request passes (merged 55df43d into f65ea70).

@luislavena
RubyGems member
@alexch
alexch commented Jul 6, 2012

Spawned process could run wildly without control of you install 30 gems in one row (like installing rails)

Nope. Only one process forks; that process generates each gem's docs in turn, just like it happens now (in the original process).

I would rather invest time in making ri lazily generate docs of newer gems
than firing a background job that will take considerable amount of time (eg
any rails gem documentation).

I don't understand. Installing new gems always generates new docs. And the fact that it takes time is entirely the point of this patch.

And this patch is really pretty simple; it took me less than an hour to write and the behavior is isolated to 2 methods (plus a config setting to allow tests to disable it).

@luislavena
RubyGems member
@alexch
alexch commented Jul 6, 2012
@zenspider

Typing 'ri' and having to wait as it parses and indexes those 30 gems is worse.

I'd rather we actually try this than reject it out of fear of lack of control. We can improve it or roll it back if it doesn't work out.

@zenspider zenspider merged commit 31f87d9 into rubygems:master Jul 6, 2012
@luislavena
RubyGems member

I'd rather we actually try this than reject it out of fear of lack of control.

Is not actually fear, generating the docs for gems forks a background process, that process will keep adding stuff to the same output (attached), so is not done, stuff is still going on.

But heck, what do I know about that...

@bhenderson bhenderson commented on the diff Jul 6, 2012
test/rubygems/test_gem_commands_install_command.rb
@@ -304,6 +304,8 @@ def test_execute_rdoc
assert_equal 0, e.exit_code
end
+ Process.wait rescue Errno::ECHILD if Process.respond_to?(:fork)
@bhenderson
bhenderson added a note Jul 6, 2012

what's the Errno::ECHILD for? It seems like this line isn't doing what it was intended to do. (same for line 277)

@alexch
alexch added a note Jul 7, 2012

If the child process has already exited, Process.wait will raise it. Note that the ri docs for Process.wait say it will raise a SystemError but actually, Errno::ECHILD < SystemCallError < StandardError (tested under 1.9.3 and 1.8.7). You can verify this by simply calling Process.wait from irb.

@alexch
alexch added a note Jul 7, 2012

To be more precise, the line is to wait for the process to exit, but not to fail if either there never was a child process forked, or if the process has already been reaped by an earlier Process.wait.

@bhenderson
bhenderson added a note Jul 7, 2012

thanks @alexch for getting back to me. What I mean is that when you rescue like that, you're describing the return value, not the Exception class to rescue.

>> Process.wait rescue Errno::ECHILD
=> Errno::ECHILD
>> Process.wait rescue 'return value'
=> "return value"

also,

>> begin Process.wait; rescue; Errno::ECHILD end
=> Errno::ECHILD
>> begin Process.wait; rescue Errno::ECHILD; end
=> nil

(I only have 1.8.7 in front of me to test, but I'm pretty sure 1.9.x does the same)

I realize this is a very minor issue. I do think it would be nice if ruby allowed for specifying the exception class in this way.

thanks for your time.

@alexch
alexch added a note Jul 7, 2012
@alexch
alexch added a note Jul 7, 2012
@bhenderson
bhenderson added a note Jul 7, 2012

great. thanks.

@zenspider
zenspider added a note Jul 7, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@raggi
raggi commented Jul 7, 2012

This won't work on JRuby.

@alexch
alexch commented Jul 8, 2012
@alexch
alexch commented Jul 8, 2012

thanks @raggi ! fixed in 6f87401

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.