rake index hangs in certain conditions #32

Closed
floere opened this Issue Oct 10, 2011 · 8 comments

Comments

Projects
None yet
2 participants
Owner

floere commented Oct 10, 2011

4 Picky users have reported that when using X indexes, X > 1, Picky hangs while doing the indexing, just after the "indexing using N processors" message.

floere was assigned Oct 10, 2011

Contributor

kschiess commented Oct 11, 2011

What Ruby version, what OS?

I've had threading-related locks in Ruby itself bite me.

Owner

floere commented Oct 11, 2011

OSX, and Ruby 1.9.2. I hope to have more info soon.

Contributor

kschiess commented Oct 11, 2011

What patch level?

Owner

floere commented Oct 11, 2011

I anticipated your question, Mr. Bond, and have prepared accordingly: 180.

Owner

floere commented Oct 11, 2011

I have further encircled the bug. Apparently, it seems to be in Enumerator#next (however, more debugging is needed). And occurs in p290, but not in p136, for example.

Owner

floere commented Oct 11, 2011

Interestingly, when I replace the incredibly elegant ;) generator pattern (element = elements.next) with a simple shift array queue (element = elements.shift) then it works in p290.

Possibly this is related to http://redmine.ruby-lang.org/issues/5003 on OS 10.7 Lion.

Contributor

kschiess commented Oct 12, 2011

In 1.8, external enumerators are intertwined with Continuations, in 1.9, with fibers. Since we're already guessing: I've had numerous instances where messing around with locks/fibers/threads/continuations and then forking left Ruby deadlocked. This almost always had to do with faulty reinit of the GIL after fork; or of unreleased locks, anyway.

The question is: Can this be made minimal enough for us to get out the big debugging iron? (dtrace and the like)

Owner

floere commented Oct 12, 2011

I'd be happy to go bug hunting soonish.

For practical reasons, I rewrote the current code to not use fork and Enumerator#next together. Forking is now only used in case there are multiple "processors" and the capability to fork (i.e. Process.respond_to?(:fork) returns true).

I will close this issue as the issue itself is resolved (until someone tells me otherwise).

floere closed this Oct 12, 2011

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment