S3 + zipfile on fork-server multiprocessing #292
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/test |
|
Successfully created a job for commit dffff08: |
1 similar comment
|
Successfully created a job for commit dffff08: |
|
/test |
|
Successfully created a job for commit e77521d: |
1 similar comment
|
Successfully created a job for commit e77521d: |
|
/test |
|
Successfully created a job for commit d407139: |
1 similar comment
|
Successfully created a job for commit d407139: |
|
/test |
|
Successfully created a job for commit 612ad31: |
1 similar comment
|
Successfully created a job for commit 612ad31: |
kuenishi
approved these changes
Dec 26, 2022
kuenishi
added
the
cat:enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Current PFIO doesn't support S3 + zipfile on fork-server based multiprocessing execution model.
This model requires parameters given to Process should be pickled to move object from child to parent, who has responsibility to fork an new process.
dffff08 solves this issue by avoiding pickling of fileobj and zipobj, then after fork, it will be re-created by zip and S3's _reset() functions.