Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose multiprocessing_context for dataloading #446

Merged
merged 1 commit into from
Aug 28, 2023

Conversation

maciej-sypetkowski
Copy link
Collaborator

@maciej-sypetkowski maciej-sypetkowski commented Aug 27, 2023

Changelogs

  • Expose multiprocessing_context argument for DataModule classes (constructor & config).

Checklist:

  • Was this PR discussed in an issue? It is recommended to first discuss a new feature into a GitHub issue before opening a PR.
  • Add tests to cover the fixed bug(s) or the new introduced feature(s) (if appropriate).
  • Update the API documentation is a new function is added, or an existing one is deleted.
  • Write concise and explanatory changelogs above.
  • If possible, assign one of the following labels to the PR: feature, fix or test (or ask a maintainer to do it for you).

discussion related to that PR

Changing multiprocessing_context to spawn can sometimes give a significant data loading speedup.

@codecov
Copy link

codecov bot commented Aug 27, 2023

Codecov Report

Merging #446 (5d316bd) into main (95ece97) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #446      +/-   ##
==========================================
+ Coverage   65.37%   65.38%   +0.01%     
==========================================
  Files          92       92              
  Lines        8253     8256       +3     
==========================================
+ Hits         5395     5398       +3     
  Misses       2858     2858              
Flag Coverage Δ
unittests 65.38% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
ipu 49.14% <ø> (ø)

@DomInvivo
Copy link
Collaborator

Perhaps grouping everything into a dataloader_kwargs that takes any dict of options, so that we don't need to expose new options every time we need something new.

@maciej-sypetkowski
Copy link
Collaborator Author

Perhaps grouping everything into a dataloader_kwargs that takes any dict of options, so that we don't need to expose new options every time we need something new.

I agree, but let's tackle it in another PR in the future.

@maciej-sypetkowski maciej-sypetkowski merged commit 35b617f into main Aug 28, 2023
7 checks passed
@maciej-sypetkowski maciej-sypetkowski deleted the expose-multiprocessing-context branch August 28, 2023 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants