Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
brain-dead simple parallel processing for ruby
Ruby HTML
branch: master

This branch is even with ahoward:master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
lib
pkg
samples
test
README
forkoff.gemspec
rakefile
readme.erb

README

NAME

  forkoff

SYNOPSIS

  brain-dead simple parallel processing for ruby

URI

  http://rubyforge.org/projects/codeforpeople
  http://github.com/ahoward/forkoff

INSTALL

  gem install forkoff

DESCRIPTION

  forkoff works for any enumerable object, iterating a code block to run in a
  child process and collecting the results.  forkoff can limit the number of
  child processes which is, by default, 2.

SAMPLES

  
  <========< samples/a.rb >========>

  ~ > cat samples/a.rb

    # forkoff makes it trivial to do parallel processing with ruby, the following
    # prints out each word in a separate process
    #
    
      require 'forkoff'
    
      %w( hey you ).forkoff!{|word| puts "#{ word } from #{ Process.pid }"}

  ~ > ruby samples/a.rb

    hey from 7907
    you from 7908


  <========< samples/b.rb >========>

  ~ > cat samples/b.rb

    # for example, this takes only 4 seconds or so to complete (8 iterations
    # running in two processes = twice as fast)
    #
    
      require 'forkoff'
    
      a = Time.now.to_f
    
      results =
        (0..7).forkoff do |i|
          sleep 1
          i ** 2
        end
    
      b = Time.now.to_f
    
      elapsed = b - a
    
      puts "elapsed: #{ elapsed }"
      puts "results: #{ results.inspect }"

  ~ > ruby samples/b.rb

    elapsed: 4.19184589385986
    results: [0, 1, 4, 9, 16, 25, 36, 49]


  <========< samples/c.rb >========>

  ~ > cat samples/c.rb

    # forkoff does *NOT* spawn processes in batches, waiting for each batch to
    # complete.  rather, it keeps a certain number of processes busy until all
    # results have been gathered.  in otherwords the following will ensure that 3
    # processes are running at all times, until the list is complete. note that
    # the following will take about 3 seconds to run (3 sets of 3 @ 1 second).
    #
    
    require 'forkoff'
    
    pid = Process.pid
    
    a = Time.now.to_f
    
    pstrees =
      %w( a b c d e f g h i ).forkoff! :processes => 3 do |letter|
        sleep 1
        { letter => ` pstree -l 2 #{ pid } ` }
      end
    
    
    b = Time.now.to_f
    
    puts
    puts "pid: #{ pid }"
    puts "elapsed: #{ b - a }"
    puts
    
    require 'yaml'
    
    pstrees.each do |pstree|
      y pstree
    end

  ~ > ruby samples/c.rb

    
    pid: 7922
    elapsed: 3.37899208068848
    
    --- 
    a: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07923 ahoward ruby -Ilib samples/c.rb
       |-+- 07924 ahoward (ruby)
       \-+- 07925 ahoward ruby -Ilib samples/c.rb
    
    --- 
    b: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07923 ahoward ruby -Ilib samples/c.rb
       |-+- 07924 ahoward ruby -Ilib samples/c.rb
       \-+- 07925 ahoward ruby -Ilib samples/c.rb
    
    --- 
    c: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07923 ahoward ruby -Ilib samples/c.rb
       |-+- 07924 ahoward (ruby)
       \-+- 07925 ahoward ruby -Ilib samples/c.rb
    
    --- 
    d: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07932 ahoward ruby -Ilib samples/c.rb
       |--- 07933 ahoward ruby -Ilib samples/c.rb
       \--- 07934 ahoward ruby -Ilib samples/c.rb
    
    --- 
    e: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |--- 07932 ahoward (ruby)
       |-+- 07933 ahoward ruby -Ilib samples/c.rb
       \-+- 07934 ahoward (ruby)
    
    --- 
    f: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |--- 07932 ahoward (ruby)
       |-+- 07933 ahoward ruby -Ilib samples/c.rb
       \-+- 07934 ahoward ruby -Ilib samples/c.rb
    
    --- 
    g: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07941 ahoward ruby -Ilib samples/c.rb
       |--- 07942 ahoward ruby -Ilib samples/c.rb
       \--- 07943 ahoward ruby -Ilib samples/c.rb
    
    --- 
    h: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07941 ahoward (ruby)
       |-+- 07942 ahoward ruby -Ilib samples/c.rb
       \--- 07943 ahoward ruby -Ilib samples/c.rb
    
    --- 
    i: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |--- 07942 ahoward (ruby)
       \-+- 07943 ahoward ruby -Ilib samples/c.rb
    


  <========< samples/d.rb >========>

  ~ > cat samples/d.rb

    # forkoff supports two strategies of reading the result from the child: via
    # pipe (the default) or via file.  you can select which to use using the
    # :strategy option.
    #
    
      require 'forkoff'
    
      %w( hey you guys ).forkoff :strategy => :file do |word|
        puts "#{ word } from #{ Process.pid }"
      end

  ~ > ruby samples/d.rb

    hey from 7953
    you from 7954
    guys from 7955



HISTORY
  1.1.0 
    - move to a model with one work queue and signals sent from consumers to
    producer to noitify ready state.  this let's smaller jobs race through a
    single process even while a larger job may have one sub-process bound up.
    incorporates a fix from http://github.com/fredrikj/forkoff which meant
    some processes would lag behind when jobs didn't have similar execution
    times.

  1.0.0
    - move to github

  0.0.4
    - code re-org
    - add :strategy option
    - default number of processes is 2, not 8

  0.0.1

    - updated to use producer threds pushing onto a SizedQueue for each consumer
      channel.  in this way the producers do not build up a massize parllel data
      structure but provide data to the consumers only as fast as they can fork
      and proccess it.  basically for a 4 process run you'll end up with 4
      channels of size 1 between 4 produces and 4 consumers, each consumer is a
      thread popping of jobs, forking, and yielding results.

    - removed use of Queue for capturing the output.  now it's simply an array
      of arrays which removed some sync overhead.

    - you can configure the number of processes globally with

        Forkoff.default['proccess'] = 4

    - you can now pass either an options hash

        forkoff( :processes => 2 ) ...

      or plain vanilla number

        forkoff( 2 ) ...

      to the forkoff call

    - default number of processes is 8, not 2
        

  0.0.0

    initial version
Something went wrong with that request. Please try again.