Reset ThreadPoolExecutor on fork. #501

jdantonio · 2016-01-24T23:06:10Z

When a ThreadPoolExecutor in the forked process detects that a fork has occurred it immediately takes the following actions:

Clears all pending jobs from its queue (assuming they will be handled by the forking process).
Deletes all worker threads (they will have died during the fork).
Resets all job counters (these counts will be reflected in the forking process).
Begins posting new jobs as normal.

jdantonio · 2016-01-25T14:37:00Z

This latest update actually clears the pool if it detects a fork. I've updated the test script to perform a fork using the example given here. The new implementation survives the fork and posts the new job. I ran the test script on my work computer (2.8 GHz Intel Core i7, 16 GB 1600 MHz DDR3) and the difference in performance is still minimal.

matthewd · 2016-01-25T15:23:00Z

We cannot use Thread.main to determine if our process has forked because Ruby copies all attributes of Thread.main on the new fork.

FWIW, my imagined Thread.main-using solution would look something like this: https://gist.github.com/matthewd/aeb876b9fad8e189bcff (modulo a method on Worker to avoid the ivar get, of course).

But yeah.. my results with that script match yours: they're so close I can't even get the original, clearly-less-code, version to consistently bench as the fastest... and when they do, $$ seems to be faster than the thread check anyway.

matthewd · 2016-01-25T15:23:23Z

examples/benchmark_thread_pool_executor.rb

+      @queue.clear
+      @ready.clear
+      @pool.clear
+      @ruby_pid == $$


jdantonio · 2016-01-26T15:14:30Z

@matthewd @pitr-ch @chrisseaton Please review.

matthewd · 2016-01-26T15:59:23Z

doc/thread_pools.md

+
+Some Ruby versions allow the Ruby process to be [forked](http://ruby-doc.org/core-2.3.0/Process.html#method-c-fork). Generally, mixing threading and forking is an [anti-pattern](https://en.wikipedia.org/wiki/Anti-pattern). Threading and forking are both concurrency techniques and mixing the two rarely provides a benefit. Moreover, Ruby does not copy any threads execept the main thread when forking. Thus threads created before the fork become unusable ("dead") in the forked process. This aspect of forking is a significant issue for any application or library which spawns threads, including but not limited to Concurrent Ruby. It is strongly advised that applications using `ThreadPoolExecutor` either directly or indirectly (via high-level concurrency abstractions like `Future` and `Actor`) do **not** also fork. Since Concurrent Ruby is a foundational library often used by gems which are in turn used by other applications it is impossible to prevent forking upstream. Concurrent Ruby therefore makes a few guaratnees about the behavior of `ThreadPoolExecutor` after forking.
+
+*Concurrent Ruby guarantees that any thread pools created on by the forking process before the fork remain functional on the forked process after the fork. It also guarantees that jobs post to a thread pool before a fork remain on the thread pool in the forking process after the fork.*


"created on by"

Maybe "remain on the thread pool in (only) the forking process"?

eregon · 2016-01-27T15:05:28Z

doc/thread_pools.md

+
+Some Ruby versions allow the Ruby process to be [forked](http://ruby-doc.org/core-2.3.0/Process.html#method-c-fork). Generally, mixing threading and forking is an [anti-pattern](https://en.wikipedia.org/wiki/Anti-pattern). Threading and forking are both concurrency techniques and mixing the two is rarely beneficial. Moreover, Ruby does not copy any threads execept the main thread when forking. Thus threads created before the fork become unusable ("dead") in the forked process. This aspect of forking is a significant issue for any application or library which spawns threads. It is strongly advised that applications using `ThreadPoolExecutor`, either directly or indirectly (via high-level concurrency abstractions like `Future` and `Actor`), do **not** also fork. Since Concurrent Ruby is a foundational library often used by gems which are in turn used by other applications, it is impossible to predict or prevent upstream forking. Concurrent Ruby therefore makes a few guaratnees about the behavior of `ThreadPoolExecutor` after forking.
+
+*Concurrent Ruby guarantees that all threads created by thread pools in the forking process before the fork remain only in the forking process after the fork. It also guarantees that jobs post to a thread pool before a fork remain on the thread pool in the forking process after the fork. Finally, it guarantees that thread pools copied to a forked process will continue to function normally on the forked process.*


Looks good, but the terminology here confused me a couple times.
Maybe parent/child would already help to figure which process we are talking about.
Also, the first guarantee is not a Concurrent Ruby one but one of the semantics of fork(2) at the OS level, right?

Maybe something simpler along the lines of
"Current jobs will be processed by the parent process. The child process does not inherit any job." would help.

schneems · 2016-01-27T15:09:42Z

Another angle of attack is to ask the the executor if it has a thread that is alive with Thread#alive? I benchmarked speed and it's in the same ballpark as $$ (though slightly slower), we could either loop through workers via Enumerable#detect or set up a "watchdog" thread that we would check every time to see if it is alive. I don't know if there's other occasions when threads might die unexpectedly other than via forking, or if it makes sense to turn this special failure case into a general failure case.

eregon · 2016-01-27T15:10:38Z

doc/thread_pools.md

+
+## Forking
+
+Some Ruby versions allow the Ruby process to be [forked](http://ruby-doc.org/core-2.3.0/Process.html#method-c-fork). Generally, mixing threading and forking is an [anti-pattern](https://en.wikipedia.org/wiki/Anti-pattern). Threading and forking are both concurrency techniques and mixing the two is rarely beneficial. Moreover, Ruby does not copy any threads execept the main thread when forking. Thus threads created before the fork become unusable ("dead") in the forked process. This aspect of forking is a significant issue for any application or library which spawns threads. It is strongly advised that applications using `ThreadPoolExecutor`, either directly or indirectly (via high-level concurrency abstractions like `Future` and `Actor`), do **not** also fork. Since Concurrent Ruby is a foundational library often used by gems which are in turn used by other applications, it is impossible to predict or prevent upstream forking. Concurrent Ruby therefore makes a few guaratnees about the behavior of `ThreadPoolExecutor` after forking.


"Moreover, Ruby does not copy any threads execept the main thread when forking."
s/Ruby/the Operating System/ I believe.
Also, it's not necessarily the main thread:
ruby -e 'p Thread.main; Thread.new { fork {p Thread.main; p Thread.current}}.join'

Maybe use the example of locks taken by threads (other than the one forking) as why it's an anti-pattern (the lock would remained locked forever and unlock-able in the child process).

jdantonio · 2016-01-28T12:56:17Z

Another update to the docs.

Reset ThreadPoolExecutor on fork.

schneems · 2016-02-23T16:37:49Z

Can we get a release cut with this fix? Any major issues to date?

jdantonio · 2016-02-23T16:44:16Z

I'll cut a release this week. I was just waiting to see if other bugs were reported.

schneems · 2016-02-23T16:45:03Z

Great, thanks!

jdantonio · 2016-02-27T14:12:38Z

@schneems v1.0.1 has been released. With this update you should now also be able to safely use require 'concurrent/future' and not experience problems on JRuby. Emphasis on "should." :-)

schneems · 2016-02-27T14:48:50Z

Awesome! Thanks ❤️ we need to get sprockets-rails back in the green. The threads dying on fork was causing failures

jdantonio added the in progress label Jan 24, 2016

jdantonio mentioned this pull request Jan 24, 2016

ThreadExecutor doesn't survive fork #500

Closed

matthewd reviewed Jan 25, 2016
View reviewed changes

matthewd reviewed Jan 26, 2016
View reviewed changes

jdantonio changed the title ~~[DO NOT MERGE] Testing threads and forking.~~ Reset ThreadPoolExecutor on fork. Jan 27, 2016

Testing threads and forking.

4422355

jdantonio force-pushed the fork-in-the-road branch from 104a2c7 to 72a43cd Compare January 27, 2016 12:54

eregon reviewed Jan 27, 2016
View reviewed changes

jdantonio force-pushed the fork-in-the-road branch 2 times, most recently from 6b66cca to ab2825e Compare January 28, 2016 12:50

ThreadPoolExecutor now survives a fork.

d53cf37

jdantonio force-pushed the fork-in-the-road branch from ab2825e to d53cf37 Compare January 28, 2016 12:55

jdantonio added a commit that referenced this pull request Jan 29, 2016

Merge pull request #501 from ruby-concurrency/fork-in-the-road

ffc3322

Reset ThreadPoolExecutor on fork.

jdantonio merged commit ffc3322 into master Jan 29, 2016

jdantonio removed the in progress label Jan 29, 2016

jdantonio deleted the fork-in-the-road branch January 29, 2016 13:23

jdantonio mentioned this pull request Feb 15, 2016

concurrent-edge: reading from a closed channel (timing issue) #494

Closed

nickelser mentioned this pull request Sep 13, 2016

TimerSet now survives a fork. #573

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Reset ThreadPoolExecutor on fork. #501

Reset ThreadPoolExecutor on fork. #501

Uh oh!

jdantonio commented Jan 24, 2016

Uh oh!

jdantonio commented Jan 25, 2016

Uh oh!

matthewd commented Jan 25, 2016

Uh oh!

matthewd Jan 25, 2016

Uh oh!

jdantonio Jan 25, 2016

Uh oh!

jdantonio commented Jan 26, 2016

Uh oh!

matthewd Jan 26, 2016

Uh oh!

eregon Jan 27, 2016

Uh oh!

schneems commented Jan 27, 2016

Uh oh!

eregon Jan 27, 2016

Uh oh!

jdantonio commented Jan 28, 2016

Uh oh!

schneems commented Feb 23, 2016

Uh oh!

jdantonio commented Feb 23, 2016

Uh oh!

schneems commented Feb 23, 2016

Uh oh!

jdantonio commented Feb 27, 2016

Uh oh!

schneems commented Feb 27, 2016

Uh oh!

Uh oh!


		Some Ruby versions allow the Ruby process to be [forked](http://ruby-doc.org/core-2.3.0/Process.html#method-c-fork). Generally, mixing threading and forking is an [anti-pattern](https://en.wikipedia.org/wiki/Anti-pattern). Threading and forking are both concurrency techniques and mixing the two rarely provides a benefit. Moreover, Ruby does not copy any threads execept the main thread when forking. Thus threads created before the fork become unusable ("dead") in the forked process. This aspect of forking is a significant issue for any application or library which spawns threads, including but not limited to Concurrent Ruby. It is strongly advised that applications using `ThreadPoolExecutor` either directly or indirectly (via high-level concurrency abstractions like `Future` and `Actor`) do not also fork. Since Concurrent Ruby is a foundational library often used by gems which are in turn used by other applications it is impossible to prevent forking upstream. Concurrent Ruby therefore makes a few guaratnees about the behavior of `ThreadPoolExecutor` after forking.

		Concurrent Ruby guarantees that any thread pools created on by the forking process before the fork remain functional on the forked process after the fork. It also guarantees that jobs post to a thread pool before a fork remain on the thread pool in the forking process after the fork.


		Some Ruby versions allow the Ruby process to be [forked](http://ruby-doc.org/core-2.3.0/Process.html#method-c-fork). Generally, mixing threading and forking is an [anti-pattern](https://en.wikipedia.org/wiki/Anti-pattern). Threading and forking are both concurrency techniques and mixing the two is rarely beneficial. Moreover, Ruby does not copy any threads execept the main thread when forking. Thus threads created before the fork become unusable ("dead") in the forked process. This aspect of forking is a significant issue for any application or library which spawns threads. It is strongly advised that applications using `ThreadPoolExecutor`, either directly or indirectly (via high-level concurrency abstractions like `Future` and `Actor`), do not also fork. Since Concurrent Ruby is a foundational library often used by gems which are in turn used by other applications, it is impossible to predict or prevent upstream forking. Concurrent Ruby therefore makes a few guaratnees about the behavior of `ThreadPoolExecutor` after forking.

		Concurrent Ruby guarantees that all threads created by thread pools in the forking process before the fork remain only in the forking process after the fork. It also guarantees that jobs post to a thread pool before a fork remain on the thread pool in the forking process after the fork. Finally, it guarantees that thread pools copied to a forked process will continue to function normally on the forked process.


		## Forking

		Some Ruby versions allow the Ruby process to be [forked](http://ruby-doc.org/core-2.3.0/Process.html#method-c-fork). Generally, mixing threading and forking is an [anti-pattern](https://en.wikipedia.org/wiki/Anti-pattern). Threading and forking are both concurrency techniques and mixing the two is rarely beneficial. Moreover, Ruby does not copy any threads execept the main thread when forking. Thus threads created before the fork become unusable ("dead") in the forked process. This aspect of forking is a significant issue for any application or library which spawns threads. It is strongly advised that applications using `ThreadPoolExecutor`, either directly or indirectly (via high-level concurrency abstractions like `Future` and `Actor`), do not also fork. Since Concurrent Ruby is a foundational library often used by gems which are in turn used by other applications, it is impossible to predict or prevent upstream forking. Concurrent Ruby therefore makes a few guaratnees about the behavior of `ThreadPoolExecutor` after forking.

Reset ThreadPoolExecutor on fork. #501

Reset ThreadPoolExecutor on fork. #501

Uh oh!

Conversation

jdantonio commented Jan 24, 2016

Uh oh!

jdantonio commented Jan 25, 2016

Uh oh!

matthewd commented Jan 25, 2016

Uh oh!

matthewd Jan 25, 2016

Choose a reason for hiding this comment

Uh oh!

jdantonio Jan 25, 2016

Choose a reason for hiding this comment

Uh oh!

jdantonio commented Jan 26, 2016

Uh oh!

matthewd Jan 26, 2016

Choose a reason for hiding this comment

Uh oh!

eregon Jan 27, 2016

Choose a reason for hiding this comment

Uh oh!

schneems commented Jan 27, 2016

Uh oh!

eregon Jan 27, 2016

Choose a reason for hiding this comment

Uh oh!

jdantonio commented Jan 28, 2016

Uh oh!

schneems commented Feb 23, 2016

Uh oh!

jdantonio commented Feb 23, 2016

Uh oh!

schneems commented Feb 23, 2016

Uh oh!

jdantonio commented Feb 27, 2016

Uh oh!

schneems commented Feb 27, 2016

Uh oh!

Uh oh!