Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor loop.setup_data with utility functions #16918

Merged
merged 10 commits into from
Mar 9, 2023
Merged

Refactor loop.setup_data with utility functions #16918

merged 10 commits into from
Mar 9, 2023
+316 −282

Conversation

carmocca
Copy link
Contributor

@carmocca carmocca commented Mar 1, 2023 •
edited by github-actions bot
Loading

What does this PR do?

Follow-up to #16726

This PR is pure refactoring into smaller functional utilities that we share. No intended logic changes

cc @Borda @justusschock @awaelchli @carmocca

@carmocca carmocca added refactor data handling Generic data-related topic pl Generic label for PyTorch Lightning package labels Mar 1, 2023
@carmocca carmocca added this to the 2.0 milestone Mar 1, 2023
@carmocca carmocca self-assigned this Mar 1, 2023
@carmocca carmocca mentioned this pull request Mar 1, 2023
Merged
10 tasks
@carmocca carmocca changed the title Restructure setup_data into utility functions Refactor loop.setup_data with utility functions Mar 1, 2023
@github-actions github-actions bot added the fabric lightning.fabric.Fabric label Mar 1, 2023
@carmocca carmocca marked this pull request as ready for review March 1, 2023 18:14
@github-actions
Copy link
Contributor

github-actions bot commented Mar 1, 2023 •
edited
Loading

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, lightning, 3.8, 1.11) success ✅
pl-cpu (macOS-11, lightning, 3.9, 1.12) success ✅
pl-cpu (macOS-11, lightning, 3.10, 1.13) success ✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest) success ✅
pl-cpu (macOS-11, lightning, 3.9, 2.0, pre) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.11) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.12) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest) success ✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 2.0, pre) success ✅
pl-cpu (windows-2022, lightning, 3.9, 1.11) success ✅
pl-cpu (windows-2022, lightning, 3.10, 1.12) success ✅
pl-cpu (windows-2022, lightning, 3.10, 1.13) success ✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest) success ✅
pl-cpu (windows-2022, lightning, 3.9, 2.0, pre) success ✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13) success ✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13) success ✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13) success ✅

These checks are required after the changes to src/lightning/fabric/utilities/data.py, src/lightning/pytorch/loops/evaluation_loop.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/loops/prediction_loop.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/strategies/xla.py, src/lightning/pytorch/trainer/connectors/data_connector.py, src/lightning/pytorch/trainer/trainer.py, src/lightning/pytorch/utilities/data.py, tests/tests_pytorch/strategies/test_single_device_strategy.py, tests/tests_pytorch/trainer/connectors/test_data_connector.py, tests/tests_pytorch/trainer/flags/test_overfit_batches.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_dataloaders.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_combined_loader.py, tests/tests_pytorch/utilities/test_data.py.

🟢 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) success ✅

These checks are required after the changes to src/lightning/pytorch/loops/evaluation_loop.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/loops/prediction_loop.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/strategies/xla.py, src/lightning/pytorch/trainer/connectors/data_connector.py, src/lightning/pytorch/trainer/trainer.py, src/lightning/pytorch/utilities/data.py, tests/tests_pytorch/strategies/test_single_device_strategy.py, tests/tests_pytorch/trainer/connectors/test_data_connector.py, tests/tests_pytorch/trainer/flags/test_overfit_batches.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_dataloaders.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_combined_loader.py, tests/tests_pytorch/utilities/test_data.py, src/lightning/fabric/utilities/data.py.

🟢 pytorch_lightning: Azure HPU
Check ID Status
pytorch-lightning (HPUs) success ✅

These checks are required after the changes to src/lightning/fabric/utilities/data.py, src/lightning/pytorch/loops/evaluation_loop.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/loops/prediction_loop.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/strategies/xla.py, src/lightning/pytorch/trainer/connectors/data_connector.py, src/lightning/pytorch/trainer/trainer.py, src/lightning/pytorch/utilities/data.py, tests/tests_pytorch/strategies/test_single_device_strategy.py, tests/tests_pytorch/trainer/connectors/test_data_connector.py, tests/tests_pytorch/trainer/flags/test_overfit_batches.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_dataloaders.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/utilities/test_combined_loader.py, tests/tests_pytorch/utilities/test_data.py.

🟢 fabric: Docs
Check ID Status
make-doctest (fabric) success ✅
make-html (fabric) success ✅

These checks are required after the changes to src/lightning/fabric/utilities/data.py.

🟢 pytorch_lightning: Docs
Check ID Status
make-doctest (pytorch) success ✅
make-html (pytorch) success ✅

These checks are required after the changes to src/lightning/pytorch/loops/evaluation_loop.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/loops/prediction_loop.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/strategies/xla.py, src/lightning/pytorch/trainer/connectors/data_connector.py, src/lightning/pytorch/trainer/trainer.py, src/lightning/pytorch/utilities/data.py.

🟢 lightning_fabric: CPU workflow
Check ID Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11) success ✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12) success ✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13) success ✅
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest) success ✅
fabric-cpu (macOS-11, lightning, 3.9, 2.0, pre) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.11) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.12) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest) success ✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 2.0, pre) success ✅
fabric-cpu (windows-2022, lightning, 3.9, 1.11) success ✅
fabric-cpu (windows-2022, lightning, 3.10, 1.12) success ✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13) success ✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest) success ✅
fabric-cpu (windows-2022, lightning, 3.9, 2.0, pre) success ✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13) success ✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13) success ✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13) success ✅

These checks are required after the changes to src/lightning/fabric/utilities/data.py.

🟢 lightning_fabric: Azure GPU
Check ID Status
lightning-fabric (GPUs) success ✅

These checks are required after the changes to src/lightning/fabric/utilities/data.py.

🟢 mypy
Check ID Status
mypy success ✅

These checks are required after the changes to src/lightning/fabric/utilities/data.py, src/lightning/pytorch/loops/evaluation_loop.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/loops/prediction_loop.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/strategies/xla.py, src/lightning/pytorch/trainer/connectors/data_connector.py, src/lightning/pytorch/trainer/trainer.py, src/lightning/pytorch/utilities/data.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.8) success ✅
install-pkg (ubuntu-22.04, app, 3.10) success ✅
install-pkg (ubuntu-22.04, fabric, 3.8) success ✅
install-pkg (ubuntu-22.04, fabric, 3.10) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.8) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.10) success ✅
install-pkg (ubuntu-22.04, lightning, 3.8) success ✅
install-pkg (ubuntu-22.04, lightning, 3.10) success ✅
install-pkg (ubuntu-22.04, notset, 3.8) success ✅
install-pkg (ubuntu-22.04, notset, 3.10) success ✅
install-pkg (macOS-12, app, 3.8) success ✅
install-pkg (macOS-12, app, 3.10) success ✅
install-pkg (macOS-12, fabric, 3.8) success ✅
install-pkg (macOS-12, fabric, 3.10) success ✅
install-pkg (macOS-12, pytorch, 3.8) success ✅
install-pkg (macOS-12, pytorch, 3.10) success ✅
install-pkg (macOS-12, lightning, 3.8) success ✅
install-pkg (macOS-12, lightning, 3.10) success ✅
install-pkg (macOS-12, notset, 3.8) success ✅
install-pkg (macOS-12, notset, 3.10) success ✅
install-pkg (windows-2022, app, 3.8) success ✅
install-pkg (windows-2022, app, 3.10) success ✅
install-pkg (windows-2022, fabric, 3.8) success ✅
install-pkg (windows-2022, fabric, 3.10) success ✅
install-pkg (windows-2022, pytorch, 3.8) success ✅
install-pkg (windows-2022, pytorch, 3.10) success ✅
install-pkg (windows-2022, lightning, 3.8) success ✅
install-pkg (windows-2022, lightning, 3.10) success ✅
install-pkg (windows-2022, notset, 3.8) success ✅
install-pkg (windows-2022, notset, 3.10) success ✅

These checks are required after the changes to src/lightning/fabric/utilities/data.py, src/lightning/pytorch/loops/evaluation_loop.py, src/lightning/pytorch/loops/fit_loop.py, src/lightning/pytorch/loops/prediction_loop.py, src/lightning/pytorch/strategies/strategy.py, src/lightning/pytorch/strategies/xla.py, src/lightning/pytorch/trainer/connectors/data_connector.py, src/lightning/pytorch/trainer/trainer.py, src/lightning/pytorch/utilities/data.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

@carmocca carmocca requested a review from tchaton as a code owner March 2, 2023 01:33
@codecov
Copy link

codecov bot commented Mar 2, 2023 •
edited
Loading

Codecov Report

Merging #16918 (57af569) into master (d7e2ab9) will decrease coverage by 23%.
The diff coverage is 96%.

Additional details and impacted files 
@@            Coverage Diff            @@
##           master   #16918     +/-   ##
=========================================
- Coverage      82%      59%    -23%     
=========================================
  Files         437      414     -23     
  Lines       31755    31481    -274     
=========================================
- Hits        25993    18681   -7312     
- Misses       5762    12800   +7038

@mergify mergify bot added the has conflicts label Mar 2, 2023
@mergify mergify bot removed the has conflicts label Mar 3, 2023
@mergify mergify bot added the ready PRs ready to be merged label Mar 8, 2023
@carmocca carmocca enabled auto-merge (squash) March 9, 2023 00:04
@carmocca carmocca merged commit bdd9b12 into master Mar 9, 2023
@carmocca carmocca deleted the refactor/setup-data-utilities branch March 9, 2023 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@awaelchli awaelchli awaelchli approved these changes

@justusschock justusschock justusschock approved these changes

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

@tchaton tchaton Awaiting requested review from tchaton tchaton is a code owner

Assignees

@carmocca carmocca

Labels
data handling Generic data-related topic fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package ready PRs ready to be merged refactor
Projects
None yet
Milestone
2.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@carmocca @awaelchli @justusschock