Guidance on using listen with multiple processes

[schneems]

Rails 5 is now using an evented file system listener. We were seeing a bug where controller code would not be reloaded in development. I eventually tracked it down rails/rails#24990 (comment)

What I think is happening is that the listen code is initialized in a process, let's call it PID 1. Then our two puma workers boot and fork to make PID 2 and PID 3. When a file is changed the callback gets triggered inside of PID 1 but since variable changes aren't persisted across other processes PID 2 and PID 3 never know that the file was updated and they need to reload code.

Initially I thought we could re-invoke the listen code on each process to get a notification on that process. However this doesn't work:

$stdout.sync = true

require 'fileutils'
require 'listen'

FileUtils.mkdir_p("/tmp/listen")

def change_callback(modified, added, removed)
  puts "  Changed registered in: #{ Process.pid }"
end


def boot!
  puts "Listening on process: #{ Process.pid }"
  Listen.to("/tmp/listen", &method(:change_callback)).start
end

boot!


Thread.new do
  sleep 5
  FileUtils.touch("/tmp/listen/foo")
end

fork do
  boot!
end

fork do
  boot!
end

sleep 10

I would expect to see something like:

Listening on process: 68158
Listening on process: 68160
Listening on process: 68161
  Changed registered in: 68158
  Changed registered in: 68160
  Changed registered in: 68161

However this is the output that I get

Listening on process: 68158
Listening on process: 68160
Listening on process: 68161
  Changed registered in: 68158

Only the parent process is notified even though we've "booted" a Listen instance on each fork.

So my question is this: are there any good practices with using Listen with multiple processes?

[e2]

I can't reply much right now (exhausted and falling asleep as I type), so I'll check this out tomorrow.

Listen does keep a list of global threads and I'm suspecting that's the problem - they should be process-specific.

To get a clearer picture of what's going on, run it with LISTEN_GEM_DEBUGGING=2 in the environment and that should help find what exactly isn't working.

I'll see if I can patch this tomorrow and release a fix.

Thanks for the report and example! Much appreciated!

[e2]

Listener#start doesn't sleep. You need to do that manually (see comments for changes):

$stdout.sync = true

require 'fileutils'
require 'listen'

FileUtils.mkdir_p("/tmp/listen")

def change_callback(modified, added, removed)
  puts "  Changed registered in: #{ Process.pid }"
end

def boot!
  puts "Listening on process: #{ Process.pid }"
  Listen.to("/tmp/listen", &method(:change_callback)).start
end

boot!

Thread.new do
  sleep 0.5
  FileUtils.touch("/tmp/listen/foo")
end

pids = []  # collect pids to kill and wait on

pids << fork do
  boot!
  sleep # sleep to prevent boot! returning from fork
end

pids << fork do
  boot!
  sleep # sleep to prevent boot! returning from fork
end

sleep 2

pids.each { |pid| Process.kill("TERM", pid) }
pids.each { |pid| Process.wait(pid) }

[e2]

For a clean solution, it's best to trap a special signal (INT ?), wake from the sleep and stop Listen before exiting. Killing process will still obviously free the system resources, but in the future gracefully exiting may help find/prevent other bugs.

[schneems]

Thanks for your responses here and for clarifying behavior, I saw earlier but forgot to comment. I plan on adding a small section in the documentation to remind people using forks to re-run listen code on multiple processes.

About that suggestion to trap a signal, I would recommend to anyone doing this to be careful. You want to make sure that you re-signal it as other systems may be depending on getting the event like Puma or Sidekiq. Original article: https://devcenter.heroku.com/articles/what-happens-to-ruby-apps-when-they-are-restarted#why-some-programs-won-t-die

Here's an okay-ish way to re-signal if you have to trap one: http://stackoverflow.com/questions/29568298/run-code-when-signal-is-sent-but-do-not-trap-the-signal-in-ruby