Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependence on multiprocessing "fork" start-method #26

Closed
jsiwek opened this issue Sep 22, 2020 · 3 comments
Closed

Dependence on multiprocessing "fork" start-method #26

jsiwek opened this issue Sep 22, 2020 · 3 comments

Comments

@jsiwek
Copy link
Contributor

jsiwek commented Sep 22, 2020

btest is structured to only work with the multiprocessing "fork" start-method [1], but Python 3.8 has changed to default start-method from "fork" to "spawn" on macOS [2] to avoid transient/intractable issues in macOS 10.13+ [3]. Running btest with Python 3.8's new "spawn" default on macOS emits a run-time error:

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

I suggest the simplest near-term fix is to explicitly ask for the "fork" start-method since I've not, up to this point, observed any significant degree of strange/unexplained failures that make me think we actually run into the underlying macOS issue (guess nothing we're currently doing ends up calling into ObjC runtime). Longer-term, could still investigate restructuring to work with other start-methods. Such a restructuring may be required if Windows support became desirable since it uses the "spawn" start-method.

[1] https://docs.python.org/3.8/library/multiprocessing.html#contexts-and-start-methods
[2] https://bugs.python.org/issue33725
[3] https://wefearchange.org/2018/11/forkmacos.rst.html

jsiwek added a commit that referenced this issue Sep 22, 2020
The default start-method on macOS in Python 3.8+ is "spawn", but that
emits a RuntimeError with current btest structuring.
jsiwek added a commit that referenced this issue Sep 22, 2020
The default start-method on macOS in Python 3.8+ is "spawn", but that
emits a RuntimeError with current btest structuring.
jsiwek added a commit that referenced this issue Sep 24, 2020
…ng-fork-method' into master

* origin/topic/jsiwek/gh-26-multiprocessing-fork-method:
  GH-26: Explicitly use multiprocessing "fork" start-method
@rsmmr
Copy link
Member

rsmmr commented Oct 20, 2020

Merged already.

@rsmmr rsmmr closed this as completed Oct 20, 2020
@jsiwek
Copy link
Contributor Author

jsiwek commented Oct 20, 2020

@rsmmr The near-term fix of "return to traditional behavior" is merged, but the underlying reason for Python 3.8+ switching the default behavior due to "potential for transient failures on macOS" may still exist. The more complex task of switching to a different multiprocessing method would resolve that (and also likely be required if Windows support is ever desired) in case it's worth keeping this open for those reasons.

@rsmmr
Copy link
Member

rsmmr commented Oct 22, 2020

Let's keep this closed given that the immediate issue is resolved, we still have the record. I'll add a note to the Zeek-side Windows ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants