Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make dataloader_idx optional for batch start/end hooks #16753

Merged
merged 6 commits into from Feb 14, 2023

Conversation

carmocca
Copy link
Member

@carmocca carmocca commented Feb 14, 2023

What does this PR do?

When you don't use multiple dataloaders, training_step does not take a dataloader_idx argument.
However, this is not the case for on_{validation,test,predict}_batch_{start,end}.

This PR makes it optional for them too.

This is a breaking change, as the users who define this argument and don't use multiple dataloaders now need to set dataloader_idx=0 in their signature, or remove it if they don't need it.

cc @justusschock @awaelchli @Borda @carmocca

@carmocca carmocca added refactor hooks Related to the hooks API pl Generic label for PyTorch Lightning package labels Feb 14, 2023
@carmocca carmocca added this to the 2.0 milestone Feb 14, 2023
@carmocca carmocca self-assigned this Feb 14, 2023
@github-actions github-actions bot added the app Generic label for Lightning App package label Feb 14, 2023
@carmocca carmocca added the breaking change Includes a breaking change label Feb 14, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Feb 14, 2023

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, lightning, 3.8, 1.11) success
pl-cpu (macOS-11, lightning, 3.9, 1.12) success
pl-cpu (macOS-11, lightning, 3.10, 1.13) success
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest) success
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.11) success
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.12) success
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest) success
pl-cpu (windows-2022, lightning, 3.9, 1.11) success
pl-cpu (windows-2022, lightning, 3.10, 1.12) success
pl-cpu (windows-2022, lightning, 3.10, 1.13) success
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest) success
pl-cpu (macOS-11, pytorch, 3.8, 1.13) success
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13) success
pl-cpu (windows-2022, pytorch, 3.8, 1.13) success

These checks are required after the changes to src/lightning/pytorch/callbacks/callback.py, src/lightning/pytorch/callbacks/device_stats_monitor.py, src/lightning/pytorch/callbacks/prediction_writer.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, src/lightning/pytorch/core/hooks.py, src/lightning/pytorch/loops/epoch/evaluation_epoch_loop.py, src/lightning/pytorch/loops/epoch/prediction_epoch_loop.py, tests/tests_pytorch/accelerators/test_hpu.py, tests/tests_pytorch/accelerators/test_ipu.py, tests/tests_pytorch/accelerators/test_tpu.py, tests/tests_pytorch/callbacks/progress/test_tqdm_progress_bar.py, tests/tests_pytorch/callbacks/test_callback_hook_outputs.py, tests/tests_pytorch/loops/test_all.py, tests/tests_pytorch/loops/test_fetchers.py, tests/tests_pytorch/loops/test_training_loop.py, tests/tests_pytorch/models/test_hooks.py, tests/tests_pytorch/models/test_restore.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/strategies/test_fsdp.py, tests/tests_pytorch/trainer/logging_/test_train_loop_logging.py, tests/tests_pytorch/trainer/optimization/test_manual_optimization.py, tests/tests_pytorch/trainer/test_dataloaders.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) success

These checks are required after the changes to src/lightning/pytorch/callbacks/callback.py, src/lightning/pytorch/callbacks/device_stats_monitor.py, src/lightning/pytorch/callbacks/prediction_writer.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, src/lightning/pytorch/core/hooks.py, src/lightning/pytorch/loops/epoch/evaluation_epoch_loop.py, src/lightning/pytorch/loops/epoch/prediction_epoch_loop.py, tests/tests_pytorch/accelerators/test_hpu.py, tests/tests_pytorch/accelerators/test_ipu.py, tests/tests_pytorch/accelerators/test_tpu.py, tests/tests_pytorch/callbacks/progress/test_tqdm_progress_bar.py, tests/tests_pytorch/callbacks/test_callback_hook_outputs.py, tests/tests_pytorch/loops/test_all.py, tests/tests_pytorch/loops/test_fetchers.py, tests/tests_pytorch/loops/test_training_loop.py, tests/tests_pytorch/models/test_hooks.py, tests/tests_pytorch/models/test_restore.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/strategies/test_fsdp.py, tests/tests_pytorch/trainer/logging_/test_train_loop_logging.py, tests/tests_pytorch/trainer/optimization/test_manual_optimization.py, tests/tests_pytorch/trainer/test_dataloaders.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure HPU
Check ID Status
pytorch-lightning (HPUs) success

These checks are required after the changes to src/lightning/pytorch/callbacks/callback.py, src/lightning/pytorch/callbacks/device_stats_monitor.py, src/lightning/pytorch/callbacks/prediction_writer.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, src/lightning/pytorch/core/hooks.py, src/lightning/pytorch/loops/epoch/evaluation_epoch_loop.py, src/lightning/pytorch/loops/epoch/prediction_epoch_loop.py, tests/tests_pytorch/accelerators/test_hpu.py, tests/tests_pytorch/accelerators/test_ipu.py, tests/tests_pytorch/accelerators/test_tpu.py, tests/tests_pytorch/callbacks/progress/test_tqdm_progress_bar.py, tests/tests_pytorch/callbacks/test_callback_hook_outputs.py, tests/tests_pytorch/loops/test_all.py, tests/tests_pytorch/loops/test_fetchers.py, tests/tests_pytorch/loops/test_training_loop.py, tests/tests_pytorch/models/test_hooks.py, tests/tests_pytorch/models/test_restore.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/strategies/test_fsdp.py, tests/tests_pytorch/trainer/logging_/test_train_loop_logging.py, tests/tests_pytorch/trainer/optimization/test_manual_optimization.py, tests/tests_pytorch/trainer/test_dataloaders.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Azure IPU
Check ID Status
pytorch-lightning (IPUs) success

These checks are required after the changes to src/lightning/pytorch/callbacks/callback.py, src/lightning/pytorch/callbacks/device_stats_monitor.py, src/lightning/pytorch/callbacks/prediction_writer.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, src/lightning/pytorch/core/hooks.py, src/lightning/pytorch/loops/epoch/evaluation_epoch_loop.py, src/lightning/pytorch/loops/epoch/prediction_epoch_loop.py, tests/tests_pytorch/accelerators/test_hpu.py, tests/tests_pytorch/accelerators/test_ipu.py, tests/tests_pytorch/accelerators/test_tpu.py, tests/tests_pytorch/callbacks/progress/test_tqdm_progress_bar.py, tests/tests_pytorch/callbacks/test_callback_hook_outputs.py, tests/tests_pytorch/loops/test_all.py, tests/tests_pytorch/loops/test_fetchers.py, tests/tests_pytorch/loops/test_training_loop.py, tests/tests_pytorch/models/test_hooks.py, tests/tests_pytorch/models/test_restore.py, tests/tests_pytorch/strategies/test_deepspeed_strategy.py, tests/tests_pytorch/strategies/test_fsdp.py, tests/tests_pytorch/trainer/logging_/test_train_loop_logging.py, tests/tests_pytorch/trainer/optimization/test_manual_optimization.py, tests/tests_pytorch/trainer/test_dataloaders.py, tests/tests_pytorch/trainer/test_trainer.py.

🟢 pytorch_lightning: Docs
Check ID Status
make-doctest (pytorch) success
make-html (pytorch) success

These checks are required after the changes to src/lightning/pytorch/callbacks/callback.py, src/lightning/pytorch/callbacks/device_stats_monitor.py, src/lightning/pytorch/callbacks/prediction_writer.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, src/lightning/pytorch/core/hooks.py, src/lightning/pytorch/loops/epoch/evaluation_epoch_loop.py, src/lightning/pytorch/loops/epoch/prediction_epoch_loop.py.

🟢 lightning_app: Tests workflow
Check ID Status
app-pytest (macOS-11, lightning, 3.8, latest) success
app-pytest (macOS-11, lightning, 3.8, oldest) success
app-pytest (macOS-11, app, 3.9, latest) success
app-pytest (ubuntu-20.04, lightning, 3.8, latest) success
app-pytest (ubuntu-20.04, lightning, 3.8, oldest) success
app-pytest (ubuntu-20.04, app, 3.9, latest) success
app-pytest (windows-2022, lightning, 3.8, latest) success
app-pytest (windows-2022, lightning, 3.8, oldest) success
app-pytest (windows-2022, app, 3.8, latest) success

These checks are required after the changes to src/lightning/app/cli/pl-app-template/core/callbacks.py.

🟢 lightning_app: Examples
Check ID Status
app-examples (macOS-11, lightning, 3.9, latest) success
app-examples (macOS-11, lightning, 3.9, oldest) success
app-examples (macOS-11, app, 3.9, latest) success
app-examples (ubuntu-20.04, lightning, 3.9, latest) success
app-examples (ubuntu-20.04, lightning, 3.9, oldest) success
app-examples (ubuntu-20.04, app, 3.9, latest) success
app-examples (windows-2022, lightning, 3.9, latest) success
app-examples (windows-2022, lightning, 3.9, oldest) success
app-examples (windows-2022, app, 3.9, latest) success

These checks are required after the changes to src/lightning/app/cli/pl-app-template/core/callbacks.py.

🟢 lightning_app: Azure
Check ID Status
App.cloud-e2e success

These checks are required after the changes to src/lightning/app/cli/pl-app-template/core/callbacks.py.

🟢 lightning_app: Docs
Check ID Status
make-doctest (lightning) success
make-html (lightning) success

These checks are required after the changes to src/lightning/app/cli/pl-app-template/core/callbacks.py.

🟢 mypy
Check ID Status
mypy success

These checks are required after the changes to src/lightning/app/cli/pl-app-template/core/callbacks.py, src/lightning/pytorch/callbacks/callback.py, src/lightning/pytorch/callbacks/device_stats_monitor.py, src/lightning/pytorch/callbacks/prediction_writer.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, src/lightning/pytorch/core/hooks.py, src/lightning/pytorch/loops/epoch/evaluation_epoch_loop.py, src/lightning/pytorch/loops/epoch/prediction_epoch_loop.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.8) success
install-pkg (ubuntu-22.04, app, 3.10) success
install-pkg (ubuntu-22.04, fabric, 3.8) success
install-pkg (ubuntu-22.04, fabric, 3.10) success
install-pkg (ubuntu-22.04, pytorch, 3.8) success
install-pkg (ubuntu-22.04, pytorch, 3.10) success
install-pkg (ubuntu-22.04, lightning, 3.8) success
install-pkg (ubuntu-22.04, lightning, 3.10) success
install-pkg (ubuntu-22.04, notset, 3.8) success
install-pkg (ubuntu-22.04, notset, 3.10) success
install-pkg (macOS-12, app, 3.8) success
install-pkg (macOS-12, app, 3.10) success
install-pkg (macOS-12, fabric, 3.8) success
install-pkg (macOS-12, fabric, 3.10) success
install-pkg (macOS-12, pytorch, 3.8) success
install-pkg (macOS-12, pytorch, 3.10) success
install-pkg (macOS-12, lightning, 3.8) success
install-pkg (macOS-12, lightning, 3.10) success
install-pkg (macOS-12, notset, 3.8) success
install-pkg (macOS-12, notset, 3.10) success
install-pkg (windows-2022, app, 3.8) success
install-pkg (windows-2022, app, 3.10) success
install-pkg (windows-2022, fabric, 3.8) success
install-pkg (windows-2022, fabric, 3.10) success
install-pkg (windows-2022, pytorch, 3.8) success
install-pkg (windows-2022, pytorch, 3.10) success
install-pkg (windows-2022, lightning, 3.8) success
install-pkg (windows-2022, lightning, 3.10) success
install-pkg (windows-2022, notset, 3.8) success
install-pkg (windows-2022, notset, 3.10) success

These checks are required after the changes to src/lightning/app/cli/pl-app-template/core/callbacks.py, src/lightning/pytorch/callbacks/callback.py, src/lightning/pytorch/callbacks/device_stats_monitor.py, src/lightning/pytorch/callbacks/prediction_writer.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, src/lightning/pytorch/core/hooks.py, src/lightning/pytorch/loops/epoch/evaluation_epoch_loop.py, src/lightning/pytorch/loops/epoch/prediction_epoch_loop.py.

🟢 link-check
Check ID Status
markdown-link-check success

These checks are required after the changes to src/lightning/pytorch/CHANGELOG.md.


Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

@awaelchli
Copy link
Member

awaelchli commented Feb 14, 2023

Does this close the age old #4206 as well?

@mergify mergify bot added the ready PRs ready to be merged label Feb 14, 2023
@carmocca
Copy link
Member Author

Does this close the age old #4206 as well?

No it doesn't.

@codecov
Copy link

codecov bot commented Feb 14, 2023

Codecov Report

Merging #16753 (2d79218) into master (104290e) will decrease coverage by 3%.
The diff coverage is 100%.

Additional details and impacted files
@@            Coverage Diff            @@
##           master   #16753     +/-   ##
=========================================
- Coverage      81%      78%     -3%     
=========================================
  Files         446      426     -20     
  Lines       32270    32057    -213     
=========================================
- Hits        26247    25073   -1174     
- Misses       6023     6984    +961     

@awaelchli
Copy link
Member

awaelchli commented Feb 14, 2023

Why not, the error there was

TypeError: validation_step() takes 3 positional arguments but 4 were given

which you would get when missing the arg in the method override. But you are making that optional now.

@carmocca
Copy link
Member Author

The linked issue references validation_step, which this PR does not touch. dataloader_idx is already optional for that method. As I understand, that PR suggest that we add validation logic at the beginning of the run to check whether *_step methods have this argument to avoid the Python exception. With the changes in this PR, the on_{validation,test,predict}_batch_{start,end} hooks would need this validation too.

@carmocca carmocca merged commit fbbbbf6 into master Feb 14, 2023
@carmocca carmocca deleted the refactor/batch_start_end_dataloader_idx branch February 14, 2023 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
app Generic label for Lightning App package breaking change Includes a breaking change hooks Related to the hooks API pl Generic label for PyTorch Lightning package ready PRs ready to be merged refactor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants