doit parallel execution doesn't work on macOS
#1354
Assignees
Labels
complexity:easy
Something that requires less than a day to fix
kind:bug
Something isn't working
topic:build
Anything related to building steps
Comments
slaperche-scality
added a commit
that referenced
this issue
Jul 3, 2019
When trying to use the parallel execution feature of `doit` on Mac, we observe that the workers processes are killed by the OS and only the main one survives. The issues seems related to the fact that: - by default `doit` uses `fork` (through `multiprocessing`) to spawn its workers - since macOS 10.13 (High Sierra), Apple added a new security measure[1] that kill processes that are using a dangerous mix of threads and forks[2]) As a consequence, now instead of working most of the time (and failing in a hard way to debug) , the processes are directly killed. There are three ways to solve this problems: 1. set the environment variable `OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES.` 2. don't use `fork` 3. fix the code that uses a dangerous mix of thread and forks ① is not good as it doesn't fix the underlying issue: it only disable the security and we're back to "works most of the time, sometimes does weird things" ② is easy to do because we can tell to `doit` to uses only threads instead of forks. ③ is probably the best, but requires more troubleshooting/time/… In conclusion, this commit implements ② until ③ is done (if ever…) by detecting macOS and forcing the use of thread in that case. [1]: http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html [2]: https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/ Closes: #1354 Signed-off-by: Sylvain Laperche <sylvain.laperche@scality.com>
slaperche-scality
added
the
complexity:easy
slaperche-scality
added a commit
that referenced
this issue
Jul 3, 2019
When trying to use the parallel execution feature of `doit` on Mac, we observe that the workers processes are killed by the OS and only the main one survives. The issues seems related to the fact that: - by default `doit` uses `fork` (through `multiprocessing`) to spawn its workers - since macOS 10.13 (High Sierra), Apple added a new security measure[1] that kill processes that are using a dangerous mix of threads and forks[2]) As a consequence, now instead of working most of the time (and failing in a hard way to debug) , the processes are directly killed. There are three ways to solve this problems: 1. set the environment variable `OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES.` 2. don't use `fork` 3. fix the code that uses a dangerous mix of thread and forks ① is not good as it doesn't fix the underlying issue: it only disable the security and we're back to "works most of the time, sometimes does weird things" ② is easy to do because we can tell to `doit` to uses only threads instead of forks. ③ is probably the best, but requires more troubleshooting/time/… In conclusion, this commit implements ② until ③ is done (if ever…) by detecting macOS and forcing the use of thread in that case. [1]: http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html [2]: https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/ Closes: #1354 Signed-off-by: Sylvain Laperche <sylvain.laperche@scality.com>
slaperche-scality
added a commit
that referenced
this issue
Jul 3, 2019
When trying to use the parallel execution feature of `doit` on Mac, we observe that the worker processes are killed by the OS and only the main one survives. The issues seems related to the fact that: - by default `doit` uses `fork` (through `multiprocessing`) to spawn its workers - since macOS 10.13 (High Sierra), Apple added a new security measure[1] that kill processes that are using a dangerous mix of threads and forks[2]) As a consequence, now instead of working most of the time (and failing in a hard way to debug), the processes are directly killed. There are three ways to solve this problems: 1. set the environment variable `OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES.` 2. don't use `fork` 3. fix the code that uses a dangerous mix of thread and forks ① is not good as it doesn't fix the underlying issue: it only disable the security and we're back to "works most of the time, sometimes does weird things" ② is easy to do because we can tell to `doit` to uses only threads instead of forks. ③ is probably the best, but requires more troubleshooting/time/… In conclusion, this commit implements ② until ③ is done (if ever…) by detecting macOS and forcing the use of threads in that case. [1]: http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html [2]: https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/ Closes: #1354 Signed-off-by: Sylvain Laperche <sylvain.laperche@scality.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component:
buildchain
What happened:
When running
doitwith parallel execution enabled, the Python workers are killed by the OS:Only the main thread survive.
What was expected:
The build is executed with no issue, all the workers survive until the end.
Steps to reproduce
On macOS, run
./doit.sh -n 4and you should see some crashes (but the build will continue since at least one worker survive).Resolution proposal:
The issues seems related to the fact that:
doitusesfork(throughmultiprocessing) to spawn its workersAs a consequence, now instead of working most of the time (and failing in a hard way to debug) , the processes are directly killed.
There are three ways to solve this problems:
OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES.fork① is not good as it doesn't fix the underlying issue: it only disable the security and we're back to "works most of the time, sometimes does weird things"
② is easy to do because we can tell to
doitto uses only threads instead of forks.③ is probably the best, but requires more troubleshooting/time/…
I recommend that we implements ② until ③ is done (if ever…).
The text was updated successfully, but these errors were encountered: