Ruby: parallel processing made simple and fast
Latest commit 3301eda Jan 19, 2017 @grosser committed on GitHub Merge pull request #192 from aeroastro/feature/simplified-array-initi…

Simplify Array initialization with slight performance improvement

Run any code in parallel Processes(> use all CPUs) or Threads(> speedup blocking operations).
Best suited for map-reduce or e.g. parallel downloads/uploads.


gem install parallel


# 2 CPUs -> work in 2 processes (a,b + c)
results =['a','b','c']) do |one_letter|

# 3 Processes -> finished after 1 run
results =['a','b','c'], in_processes: 3) { |one_letter| ... }

# 3 Threads -> finished after 1 run
results =['a','b','c'], in_threads: 3) { |one_letter| ... }

Same can be done with each

Parallel.each(['a','b','c']) { |one_letter| ... }

or each_with_index or map_with_index

Produce one item at a time with lambda (anything that responds to .call) or Queue.

items = [1,2,3]
Parallel.each( -> { items.pop || Parallel::Stop }) { |number| ... }

Processes/Threads are workers, they grab the next piece of work when they finish.


  • Speedup through multiple CPUs
  • Speedup for blocking operations
  • Variables are protected from change
  • Extra memory used
  • Child processes are killed when your main process is killed through Ctrl+c or kill -2


  • Speedup for blocking operations
  • Variables can be shared/modified
  • No extra memory used


Try any of those to get working parallel AR

# reproducibly fixes things (spec/cases/map_with_ar.rb)
Parallel.each(User.all, in_processes: 8) do |user|
  user.update_attribute(:some_attribute, some_value)

# maybe helps: explicitly use connection pool
Parallel.each(User.all, in_threads: 8) do |user|
  ActiveRecord::Base.connection_pool.with_connection do
    user.update_attribute(:some_attribute, some_value)

# maybe helps: reconnect once inside every fork
Parallel.each(User.all, in_processes: 8) do |user|
  @reconnected ||= User.connection.reconnect! || true
  user.update_attribute(:some_attribute, some_value)

Break do |user|
  raise Parallel::Break # -> stops after all current items are finished


Only use if whatever is executing in the sub-command is safe to kill at any point[1,2,3]) do |x|
  raise Parallel::Kill if x == 1# -> stop all sub-processes, killing them instantly
  sleep 100 # Do stuff

Progress / ETA

# gem install ruby-progressbar, progress: "Doing stuff") { sleep 1 }

# Doing stuff | ETA: 00:00:02 | ====================               | Time: 00:00:10

Use :finish or :start hook to get progress information.

  • :start has item and index
  • :finish has item, index, result

They are called on the main process and protected with a mutex., finish: -> (item, i, result) { ... do something ... }) { sleep 1 }

Worker number

Use Parallel.worker_number to determine the worker slot in which your task is running.

Parallel.each(1..5, :in_processes => 2) { |i| puts "Item: #{i}, Worker: #{Parallel.worker_number}" }
Item: 1, Worker: 1
Item: 2, Worker: 0
Item: 3, Worker: 1
Item: 4, Worker: 0
Item: 5, Worker: 1


Here are a few notable options.

  • [Benchmark/Test] Disable threading/forking with in_threads: 0 or in_processes: 0, great to test performance or to debug parallel issues
  • [Isolation] Do not reuse previous worker processes: isolation: true
  • [Stop all processses with an alternate interrupt signal] 'INT' (from ctrl+c) is caught by default. Catch 'TERM' (from kill) with interrupt_signal: 'TERM'


  • Replace Signal trapping with simple rescue Interrupt handler



Michael Grosser
License: MIT
Build Status