Issue 220 make plugins support multiprocessing spawn #222
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With this, the plugins are accessible from other modules without requiring an explicit import. This was the default behaviour of the previous pre-3.10 loader. Without this, an import call would re-import the plugin making the first and the second ones different from Python's perspective even if they are both the same. This breaks pickle.
load_modules() is split in two: one part imports and registers the modules while the other does the proper initialization of the extension objects (ExampleFinder, ExampleParser, ExampleRunner, ZoneDelimiter and Concern). The first part imports the (python) modules and registers them in sys.modules so they objects are pickle-able and the access to them does not require another import. This is to maintain compatibility with the form that Python pre-3.10 used to load the modules. Also, if a module cannot be loaded (typically due a syntax error), emit an error by default. Adding -vvv will print the full traceback as usual.
When multiprocessing uses another start method different of 'fork', it requires that the target function and its arguments can be pickled. In particular, multiprocessing needs to pickle them in the parent process and un-pickle them in the child process. Due how pickling works in Python, pickling a function only involves storing the information needed to reload it: the bytecode is never stored. This makes pickling particular tricky in byexample: if a we want to call a function that it is from one of the byexample modules/plugins, the pickling will fail. It will not fail when multiprocessing serialize it (dumps) but when it deserialize it (loads) because in the child process, the byexample modules/plugins will not be loaded in sys.modules and because they are not in the sys.path (in principle), Python will not be able to find them. _prepare_subprocess_call() can wrap the target function and its arguments and replacing the target by a _subprocess_trampoline function that will call the former. This _subprocess_trampoline will do all the bootstraping needed in the child process, including the (re)loading of the modules/plugins, to make the un-pickling work. Most of these details are hidden from the user (plugin developer). He/she is only required to call prepare_subprocess_call() and use the returned target/arguments in replace of his/her. The prepare_subprocess_call() is a partial bound function of _prepare_subprocess_call(). The former can be obtained optionally from the extension constructor (__init__ method of ExampleParser, ExampleFinder, ExampleRunner, ZoneDelimiter and Concern). The function is thread-safe and it will accessible both in the main byexample process and in each worker thread.
eldipa
force-pushed
the
Issue-220-Make-Plugins-Support-Multiprocessing-Spawn
branch
from
6a9f928
to
3838adc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #220