Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Ruby: parallel processing made simple and fast

This branch is 0 commits ahead and 0 commits behind master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Readme.md

Run any code in parallel Processes(> use all CPUs) or Threads(> speedup blocking operations).
Best suited for map-reduce or e.g. parallel downloads/uploads.

Install

sudo gem install parallel

Usage

# 2 CPUs -> work in 2 processes (a,b + c)
results = Parallel.map(['a','b','c']) do |one_letter|
  expensive_calculation(letter)
end

# 3 Processes -> finished after 1 run
results = Parallel.map(['a','b','c'], :in_processes=>3){|one_letter| ... }

# 3 Threads -> finished after 1 run
results = Parallel.map(['a','b','c'], :in_threads=>3){|one_letter| ... }

Same can be done with each Parallel.each(['a','b','c']){|one_letter| ... } or each_with_index or map_with_index

Processes

  • Speedup through multiple CPUs
  • Speedup for blocking operations
  • Protects global data
  • Extra memory used ( very low on REE through copy_on_write_friendly )
  • Child processes are killed when your main process is killed through Ctrl+c or kill -2

Threads

  • Speedup for blocking operations
  • Global data can be modified
  • No extra memory used

Processes/Threads are workers, they grab the next piece of work when they finish

Tips

  • [ActiveRecord] ActiveRecord::Base.connection.reconnect! inside the parallel block prevents errors

TODO

  • JRuby / Windows support <-> possible ?

Authors

Contributors

Michael Grosser
michael@grosser.it
Hereby placed under public domain, do what you want, just do not hold me accountable...

Something went wrong with that request. Please try again.