EM-Synchrony doesn't work with Enumerator #114

Open
prepor opened this Issue Mar 11, 2012 · 8 comments

Projects

None yet

4 participants

@prepor
Contributor
prepor commented Mar 11, 2012

Hello! Simple case:

require 'em-synchrony'
EM.synchrony do
  start = Time.now
  enum = Enumerator.new do |y|
    3.times do |i|
      EM::Synchrony.sleep 100
      y.yield i
    end
  end
  puts enum.next
  puts "Time: #{Time.now - start}"
  EM.stop
end

And it will return value (nil) instantly, because Enumerator uses Fibers internally and we have yielded from its fiber in EM::Synchrony.sleep.

@igrigorik
Owner

Hmmm, wow.. I am really surprised this hasn't show up before! Nice one :-)

This seems to do the trick for me:

require 'em-synchrony'

EM.synchrony do
  start = Time.now

  enum = Enumerator.new do |y|
    3.times do |i|
        y << i
      end
  end

  enum.each do |v|
    puts v
    EM::Synchrony.sleep 1
  end

  puts "Time: #{Time.now - start}"
  EM.stop
end

# $> ruby test.rb 
# 0
# 1
# 2
# Time: 3.001568
@fl00r
fl00r commented Mar 12, 2012

The actual problem was with slow IO in Enumerator. The usecase was something like

enumerator = Enumerator.new do |enum|
  ActiveRecordModel.some.query.each do |item|
    enum.yield item.title
  end
end
EM::Synchrony::FiberIterator.new(enumerator, 10).each do |title|
  # processing
end

And in EM::Iterator next is called on enumerators, so it fails

@prepor
Contributor
prepor commented Mar 12, 2012

In fact, there are more simple trick:

require 'em-synchrony'
EM.synchrony do
  start = Time.now
  enum = Enumerator.new do |y|
    3.times do |i|
      EM::Synchrony.sleep 1
      y.yield i
    end
  end
  enum.each { |i| puts i }
  puts "Time: #{Time.now - start}"
  EM.stop
end

So, rule is sound like this: "do not use em-sync in body of enumerator with #next"

@prepor
Contributor
prepor commented Mar 12, 2012

Another way:

ROOT = Fiber.current

EM.run do
  Fiber.new do
    en = Enumerator.new do |y|
      3.times do |i|
        f = Fiber.current
        EM.add_timer(1) { f.transfer }
        ROOT.transfer
        y << i
      end
    end
    puts en.next
    EM.stop
  end.resume
end

Fiber.yield in EM.Synchrony is always "ok, we can't do anything more in this fiber and must wait callback", so always transfer root fiber (with event loop) is ok, i think.

@igrigorik
Owner

@funny-falcon's patch seems to do the trick.. but I'm having a hard time trying to figure out the actual patch in there :-)

@igrigorik
Owner

So, do we want to merge any of this into master? If not, I'll close the bug.

@prepor
Contributor
prepor commented Jun 21, 2012

But bug are exist, but it is impossible to fix it in current EM-Syncrhony correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment