Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Ruby: parallel processing made simple and fast
Ruby

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
lib
spec
Gemfile
Gemfile.lock
Rakefile
Readme.md
parallel.gemspec

Readme.md

Run any code in parallel Processes(> use all CPUs) or Threads(> speedup blocking operations).
Best suited for map-reduce or e.g. parallel downloads/uploads.

Install

sudo gem install parallel

Usage

# 2 CPUs -> work in 2 processes (a,b + c)
results = Parallel.map(['a','b','c']) do |one_letter|
  expensive_calculation(letter)
end

# 3 Processes -> finished after 1 run
results = Parallel.map(['a','b','c'], :in_processes=>3){|one_letter| ... }

# 3 Threads -> finished after 1 run
results = Parallel.map(['a','b','c'], :in_threads=>3){|one_letter| ... }

Same can be done with each Parallel.each(['a','b','c']){|one_letter| ... } or each_with_index or map_with_index

Processes

  • Speedup through multiple CPUs
  • Speedup for blocking operations
  • Protects global data
  • Extra memory used ( very low on REE through copy_on_write_friendly )
  • Child processes are killed when your main process is killed through Ctrl+c or kill -2

Threads

  • Speedup for blocking operations
  • Global data can be modified
  • No extra memory used

Processes/Threads are workers, they grab the next piece of work when they finish

Tips

  • [ActiveRecord] ActiveRecord::Base.connection.reconnect! inside the parallel block prevents errors
  • [Benchmark/Test] Disable threading/forking with :in_threads => 0 or :in_processes => 0, great to test performance or to debug parallel issues

TODO

  • JRuby / Windows support <-> possible ?

Authors

Contributors

Michael Grosser
michael@grosser.it
License: MIT
Flattr

Something went wrong with that request. Please try again.