Feature: support ProcessPoolExecutor (Python >= 3.3 required) #31
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Summary
functools.partial
instead of directly wrapping therequest
method,regardless of executor used - avoid code branching
functools.partial(Session.request, self)
instead ofsuper(FuturesSession, self).request
for pickling, again no code branchingNotes / Limitations
copy_reg
to add this support to Python 2.pickle
related issues fixed in 3.5.ProcessPoolExecutor
but only when the callback also modifies__attrs__
so it gets pickled and sent to parent process (see an example for that in the tests).background_callback
MUST be a top-level (or otherwise importable) object. Additionally, since an instance ofFuturesSession
is passed to the callback (asself
), it too must be importable, otherwisepickle
will complain.Conclusions
This relatively simple implementation fixes issue #11, however due to numerous caveats, I'd suggest we recommend it be used in Python 3.5 and above and explain the limitation of earlier versions. If earlier versions (like Python 2) are really required by someone they may need to implement a manual fix to the
pickle
implementation (e.g usingcopy_reg
).Personally, this PR solves my issue but I have yet to test it on production, hopefully, will do so in the near future.