Downgrade torchmetrics to 0.6.0 #19

dxoigmn · 2022-12-09T20:24:51Z

What does this PR do?

torchmetrics > 0.6.0 is slow and this still has not been fixed in newer versions: Lightning-AI/torchmetrics#1024.

This should improve the performance of object detection training from 3 hours to 1 hour.

Type of change

Please check all relevant options.

Improvement (non-breaking)
Bug fix (non-breaking)
New feature (non-breaking)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Testing

I trained an object detector for 50 iterations:

$ time CUDA_VISIBLE_DEVICES=1 python -m mart experiment=ArmoryCarlaOverObjDet_TorchvisionFasterRCNN trainer=gpu trainer.precision=16 fit=True trainer.max_steps=50 ++paths.data_dir=/raid/datasets

With be5dbcb, it takes:

244.83s user 23.34s system 129% cpu 3:26.39 total

With 6542ab0, it takes:

77.40s user 21.39s system 222% cpu 44.404 total

Before submitting

The title is self-explanatory and the description concisely explains the PR
My PR does only one thing, instead of bundling different changes together
I list all the breaking changes introduced by this pull request
I have commented my code
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I have run pre-commit hooks with pre-commit run -a command without errors

Did you have fun?

Make sure you had fun coding 🙃

mzweilin

LGTM. Thanks Cory for eliminating the performance bottleneck.

lumurillo · 2022-12-09T21:18:32Z

Running the training with the two version of torchmetrics in a GPU I was able to reproduce this improvement.

Also, I ran a profiler in the CPU (seems that the profiler times are more reliable in CPU than GPU). Here are the results:

Without the torchmetrics downgrade:

FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                    	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                     	|  -              	|  2254           	|  274.74         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                        	|  261.76         	|  1              	|  261.76         	|  95.276         	|
|  run_training_batch                                                                                                                                                                                        	|  5.0274         	|  50             	|  251.37         	|  91.493         	|
|  [LightningModule]LitModular.optimizer_step                                                                                                                                                                	|  5.0254         	|  50             	|  251.27         	|  91.456         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                   	|  2.8408         	|  50             	|  142.04         	|  51.7           	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                              	|  2.0117         	|  50             	|  100.58         	|  36.61          	|
|  [LightningModule]LitModular.training_step_end                                                                                                                                                             	|  0.14344        	|  50             	|  7.1719         	|  2.6104         	|
|  [LightningModule]LitModular.training_epoch_end                                                                                                                                                            	|  7.1119         	|  1              	|  7.1119         	|  2.5886         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                            	|  2.0532         	|  2              	|  4.1065         	|  1.4947         	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.53921        	|  1              	|  0.53921        	|  0.19626        	|
|  [LightningModule]LitModular.validation_step_end                                                                                                                                                           	|  0.18442        	|  2              	|  0.36884        	|  0.13425        	|
|  [LightningModule]LitModular.validation_epoch_end                                                                                                                                                          	|  0.36056        	|  1              	|  0.36056        	|  0.13124        	|
|  [LightningModule]LitModular.optimizer_zero_grad                                                                                                                                                           	|  0.0051853      	|  50             	|  0.25926        	|  0.094366       	|
|  on_train_batch_end                                                                                                                                                                                        	|  0.00089714     	|  50             	|  0.044857       	|  0.016327       	|
|  [LightningModule]LitModular.configure_optimizers                                                                                                                                                          	|  0.0077106      	|  1              	|  0.0077106      	|  0.0028065      	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                            	|  0.00012016     	|  52             	|  0.0062481      	|  0.0022742      	|
|  on_train_batch_start                                                                                                                                                                                      	|  9.7736e-05     	|  50             	|  0.0048868      	|  0.0017787      	|
|  [LightningModule]LitModular.lr_scheduler_step                                                                                                                                                             	|  4.7928e-05     	|  50             	|  0.0023964      	|  0.00087224     	|
|  [Callback]ModelSummary.on_fit_start                                                                                                                                                                       	|  0.0022871      	|  1              	|  0.0022871      	|  0.00083244     	|
|  [LightningModule]LitModular.on_validation_model_eval                                                                                                                                                      	|  0.0014501      	|  1              	|  0.0014501      	|  0.00052782     	|
|  [Callback]TQDMProgressBar.on_sanity_check_start                                                                                                                                                           	|  0.0013166      	|  1              	|  0.0013166      	|  0.00047922     	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                         	|  0.00034436     	|  2              	|  0.00068871     	|  0.00025068     	|
|  [LightningModule]LitModular.on_validation_model_train                                                                                                                                                     	|  0.0006001      	|  1              	|  0.0006001      	|  0.00021842     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                              	|  0.00045786     	|  1              	|  0.00045786     	|  0.00016665     	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                               	|  6.96e-06       	|  50             	|  0.000348       	|  0.00012666     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                       	|  0.00016532     	|  2              	|  0.00033065     	|  0.00012035     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                    	|  0.00023245     	|  1              	|  0.00023245     	|  8.4605e-05     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                            	|  0.00020508     	|  1              	|  0.00020508     	|  7.4644e-05     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                               	|  0.00018397     	|  1              	|  0.00018397     	|  6.6959e-05     	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                             	|  3.5624e-06     	|  50             	|  0.00017812     	|  6.4832e-05     	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                  	|  3.3192e-06     	|  50             	|  0.00016596     	|  6.0406e-05     	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                              	|  2.8127e-06     	|  50             	|  0.00014064     	|  5.1189e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                  	|  0.00013955     	|  1              	|  0.00013955     	|  5.0792e-05     	|
|  [Callback]ModelSummary.on_after_backward                                                                                                                                                                  	|  2.6837e-06     	|  50             	|  0.00013418     	|  4.884e-05      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                    	|  2.6142e-06     	|  50             	|  0.00013071     	|  4.7576e-05     	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                        	|  2.4605e-06     	|  50             	|  0.00012303     	|  4.4779e-05     	|
|  [LightningModule]LitModular.on_train_batch_end                                                                                                                                                            	|  2.0711e-06     	|  50             	|  0.00010356     	|  3.7693e-05     	|
|  [Callback]ModelSummary.on_batch_start                                                                                                                                                                     	|  2.0322e-06     	|  50             	|  0.00010161     	|  3.6984e-05     	|
|  [Callback]ModelSummary.on_batch_end                                                                                                                                                                       	|  1.9703e-06     	|  50             	|  9.8514e-05     	|  3.5857e-05     	|
|  [Callback]ModelSummary.on_before_zero_grad                                                                                                                                                                	|  1.965e-06      	|  50             	|  9.8252e-05     	|  3.5761e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  1.8836e-06     	|  50             	|  9.4179e-05     	|  3.4279e-05     	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                 	|  1.875e-06      	|  50             	|  9.375e-05      	|  3.4123e-05     	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                          	|  1.8219e-06     	|  50             	|  9.1095e-05     	|  3.3157e-05     	|
|  [Callback]ModelSummary.on_before_optimizer_step                                                                                                                                                           	|  1.8162e-06     	|  50             	|  9.0811e-05     	|  3.3053e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  1.8046e-06     	|  50             	|  9.0228e-05     	|  3.2841e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  1.7743e-06     	|  50             	|  8.8713e-05     	|  3.2289e-05     	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                    	|  1.7712e-06     	|  50             	|  8.8562e-05     	|  3.2235e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  1.7585e-06     	|  50             	|  8.7925e-05     	|  3.2003e-05     	|
|  [Callback]ModelSummary.on_before_backward                                                                                                                                                                 	|  1.7294e-06     	|  50             	|  8.6471e-05     	|  3.1473e-05     	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                      	|  1.7289e-06     	|  50             	|  8.6447e-05     	|  3.1465e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  1.7018e-06     	|  50             	|  8.5091e-05     	|  3.0971e-05     	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                          	|  1.6841e-06     	|  50             	|  8.4203e-05     	|  3.0648e-05     	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                               	|  1.65e-06       	|  50             	|  8.2498e-05     	|  3.0028e-05     	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                	|  1.6439e-06     	|  50             	|  8.2195e-05     	|  2.9917e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  1.641e-06      	|  50             	|  8.205e-05      	|  2.9864e-05     	|
|  [LightningModule]LitModular.on_before_zero_grad                                                                                                                                                           	|  1.5642e-06     	|  50             	|  7.8209e-05     	|  2.8466e-05     	|
|  [LightningModule]LitModular.on_after_backward                                                                                                                                                             	|  1.5376e-06     	|  50             	|  7.688e-05      	|  2.7983e-05     	|
|  [LightningModule]LitModular.on_train_batch_start                                                                                                                                                          	|  1.5155e-06     	|  50             	|  7.5777e-05     	|  2.7581e-05     	|
|  [LightningModule]LitModular.on_before_backward                                                                                                                                                            	|  1.4201e-06     	|  50             	|  7.1006e-05     	|  2.5845e-05     	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                       	|  1.1487e-06     	|  50             	|  5.7437e-05     	|  2.0906e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.setup                              	|  5.1267e-05     	|  1              	|  5.1267e-05     	|  1.866e-05      	|
|  [LightningModule]LitModular.on_before_optimizer_step                                                                                                                                                      	|  9.62e-07       	|  50             	|  4.81e-05       	|  1.7507e-05     	|
|  [Callback]ModelSummary.on_sanity_check_start                                                                                                                                                              	|  1.8382e-05     	|  1              	|  1.8382e-05     	|  6.6906e-06     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                             	|  1.5665e-05     	|  1              	|  1.5665e-05     	|  5.7017e-06     	|
|  [Callback]ModelSummary.on_validation_batch_end                                                                                                                                                            	|  5.8427e-06     	|  2              	|  1.1685e-05     	|  4.2532e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  1.0884e-05     	|  1              	|  1.0884e-05     	|  3.9615e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                  	|  4.1758e-06     	|  2              	|  8.3516e-06     	|  3.0398e-06     	|
|  [Callback]ModelSummary.on_validation_batch_start                                                                                                                                                          	|  3.9036e-06     	|  2              	|  7.8073e-06     	|  2.8417e-06     	|
|  [Callback]GradientAccumulationScheduler.on_sanity_check_start                                                                                                                                             	|  7.6639e-06     	|  1              	|  7.6639e-06     	|  2.7895e-06     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                      	|  7.4888e-06     	|  1              	|  7.4888e-06     	|  2.7257e-06     	|
|  [Callback]ModelSummary.on_epoch_start                                                                                                                                                                     	|  3.7174e-06     	|  2              	|  7.4347e-06     	|  2.7061e-06     	|
|  [Callback]ModelSummary.on_validation_start                                                                                                                                                                	|  7.397e-06      	|  1              	|  7.397e-06      	|  2.6924e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  3.6741e-06     	|  2              	|  7.3481e-06     	|  2.6746e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                    	|  3.567e-06      	|  2              	|  7.1339e-06     	|  2.5966e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                               	|  7.0138e-06     	|  1              	|  7.0138e-06     	|  2.5529e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_sanity_check_start              	|  6.4541e-06     	|  1              	|  6.4541e-06     	|  2.3491e-06     	|
|  [Callback]ModelSummary.on_train_epoch_end                                                                                                                                                                 	|  6.3577e-06     	|  1              	|  6.3577e-06     	|  2.3141e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  6.3139e-06     	|  1              	|  6.3139e-06     	|  2.2981e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                       	|  5.838e-06      	|  1              	|  5.838e-06      	|  2.1249e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                         	|  5.6401e-06     	|  1              	|  5.6401e-06     	|  2.0529e-06     	|
|  [Callback]ModelSummary.on_validation_epoch_start                                                                                                                                                          	|  5.5702e-06     	|  1              	|  5.5702e-06     	|  2.0274e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  5.452e-06      	|  1              	|  5.452e-06      	|  1.9844e-06     	|
|  [Callback]TQDMProgressBar.on_sanity_check_end                                                                                                                                                             	|  5.3332e-06     	|  1              	|  5.3332e-06     	|  1.9412e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                           	|  5.051e-06      	|  1              	|  5.051e-06      	|  1.8385e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                    	|  2.5239e-06     	|  2              	|  5.0478e-06     	|  1.8373e-06     	|
|  [Callback]ModelSummary.on_train_end                                                                                                                                                                       	|  5.0003e-06     	|  1              	|  5.0003e-06     	|  1.82e-06       	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                           	|  2.2838e-06     	|  2              	|  4.5677e-06     	|  1.6625e-06     	|
|  [Callback]ModelSummary.on_validation_end                                                                                                                                                                  	|  4.3302e-06     	|  1              	|  4.3302e-06     	|  1.5761e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                         	|  2.1106e-06     	|  2              	|  4.2212e-06     	|  1.5364e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.0117e-06     	|  2              	|  4.0233e-06     	|  1.4644e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  1.976e-06      	|  2              	|  3.9521e-06     	|  1.4385e-06     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                      	|  3.8771e-06     	|  1              	|  3.8771e-06     	|  1.4112e-06     	|
|  [Callback]ModelSummary.on_train_start                                                                                                                                                                     	|  3.702e-06      	|  1              	|  3.702e-06      	|  1.3474e-06     	|
|  [Callback]TQDMProgressBar.on_save_checkpoint                                                                                                                                                              	|  3.533e-06      	|  1              	|  3.533e-06      	|  1.2859e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                              	|  3.4883e-06     	|  1              	|  3.4883e-06     	|  1.2697e-06     	|
|  [Callback]ModelSummary.on_train_epoch_start                                                                                                                                                               	|  3.3882e-06     	|  1              	|  3.3882e-06     	|  1.2332e-06     	|
|  [Callback]ModelSummary.on_epoch_end                                                                                                                                                                       	|  1.6845e-06     	|  2              	|  3.3691e-06     	|  1.2263e-06     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                    	|  3.2829e-06     	|  1              	|  3.2829e-06     	|  1.1949e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                         	|  3.2731e-06     	|  1              	|  3.2731e-06     	|  1.1913e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                      	|  1.6293e-06     	|  2              	|  3.2587e-06     	|  1.1861e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  1.5816e-06     	|  2              	|  3.1632e-06     	|  1.1513e-06     	|
|  [LightningModule]LitModular.on_val_dataloader                                                                                                                                                             	|  2.9132e-06     	|  1              	|  2.9132e-06     	|  1.0603e-06     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                       	|  2.8401e-06     	|  1              	|  2.8401e-06     	|  1.0337e-06     	|
|  [LightningModule]LitModular.on_epoch_start                                                                                                                                                                	|  1.3877e-06     	|  2              	|  2.7753e-06     	|  1.0102e-06     	|
|  [LightningModule]LitModular.on_validation_start                                                                                                                                                           	|  2.7311e-06     	|  1              	|  2.7311e-06     	|  9.9406e-07     	|
|  [LightningModule]LitModular.on_validation_batch_end                                                                                                                                                       	|  1.3381e-06     	|  2              	|  2.6762e-06     	|  9.7406e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                             	|  2.6687e-06     	|  1              	|  2.6687e-06     	|  9.7135e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  2.6319e-06     	|  1              	|  2.6319e-06     	|  9.5796e-07     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                        	|  1.2964e-06     	|  2              	|  2.5928e-06     	|  9.4372e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                      	|  2.536e-06      	|  1              	|  2.536e-06      	|  9.2304e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.4871e-06     	|  1              	|  2.4871e-06     	|  9.0525e-07     	|
|  [Callback]ModelSummary.setup                                                                                                                                                                              	|  2.4186e-06     	|  1              	|  2.4186e-06     	|  8.8033e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                        	|  2.407e-06      	|  1              	|  2.407e-06      	|  8.761e-07      	|
|  [Callback]ModelSummary.on_sanity_check_end                                                                                                                                                                	|  2.3209e-06     	|  1              	|  2.3209e-06     	|  8.4474e-07     	|
|  [Callback]ModelSummary.on_validation_epoch_end                                                                                                                                                            	|  2.2468e-06     	|  1              	|  2.2468e-06     	|  8.1779e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                 	|  2.17e-06       	|  1              	|  2.17e-06       	|  7.8982e-07     	|
|  [Callback]ModelSummary.on_save_checkpoint                                                                                                                                                                 	|  2.143e-06      	|  1              	|  2.143e-06      	|  7.7999e-07     	|
|  [Callback]ModelSummary.on_pretrain_routine_start                                                                                                                                                          	|  2.0447e-06     	|  1              	|  2.0447e-06     	|  7.4423e-07     	|
|  [LightningModule]LitModular.on_validation_batch_start                                                                                                                                                     	|  1.0198e-06     	|  2              	|  2.0396e-06     	|  7.4237e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                        	|  2.028e-06      	|  1              	|  2.028e-06      	|  7.3813e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                           	|  2.0256e-06     	|  1              	|  2.0256e-06     	|  7.3728e-07     	|
|  [Callback]ModelSummary.on_before_accelerator_backend_setup                                                                                                                                                	|  2.0089e-06     	|  1              	|  2.0089e-06     	|  7.3118e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                    	|  1.9893e-06     	|  1              	|  1.9893e-06     	|  7.2406e-07     	|
|  [LightningModule]LitModular.on_train_epoch_end                                                                                                                                                            	|  1.9851e-06     	|  1              	|  1.9851e-06     	|  7.2254e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                         	|  1.9749e-06     	|  1              	|  1.9749e-06     	|  7.1881e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                	|  1.9739e-06     	|  1              	|  1.9739e-06     	|  7.1847e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                      	|  1.9562e-06     	|  1              	|  1.9562e-06     	|  7.1203e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  1.9162e-06     	|  1              	|  1.9162e-06     	|  6.9745e-07     	|
|  [Callback]GradientAccumulationScheduler.on_sanity_check_end                                                                                                                                               	|  1.9022e-06     	|  1              	|  1.9022e-06     	|  6.9237e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                         	|  1.9018e-06     	|  1              	|  1.9018e-06     	|  6.922e-07      	|
|  [LightningModule]LitModular.configure_callbacks                                                                                                                                                           	|  1.8841e-06     	|  1              	|  1.8841e-06     	|  6.8576e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                             	|  1.8799e-06     	|  1              	|  1.8799e-06     	|  6.8423e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  1.878e-06      	|  1              	|  1.878e-06      	|  6.8355e-07     	|
|  [Callback]ModelSummary.on_configure_sharded_model                                                                                                                                                         	|  1.8761e-06     	|  1              	|  1.8761e-06     	|  6.8288e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.on_before_accelerator_backend_setup	|  1.8692e-06     	|  1              	|  1.8692e-06     	|  6.8033e-07     	|
|  [Callback]ModelSummary.on_fit_end                                                                                                                                                                         	|  1.8571e-06     	|  1              	|  1.8571e-06     	|  6.7593e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                        	|  1.8389e-06     	|  1              	|  1.8389e-06     	|  6.6932e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  1.8249e-06     	|  1              	|  1.8249e-06     	|  6.6423e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.8128e-06     	|  1              	|  1.8128e-06     	|  6.5983e-07     	|
|  [LightningModule]LitModular.on_validation_epoch_start                                                                                                                                                     	|  1.8128e-06     	|  1              	|  1.8128e-06     	|  6.5983e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  1.8012e-06     	|  1              	|  1.8012e-06     	|  6.5559e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_sanity_check_end                	|  1.7881e-06     	|  1              	|  1.7881e-06     	|  6.5084e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                               	|  1.763e-06      	|  1              	|  1.763e-06      	|  6.4169e-07     	|
|  [LightningModule]LitModular.setup                                                                                                                                                                         	|  1.743e-06      	|  1              	|  1.743e-06      	|  6.344e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  1.7309e-06     	|  1              	|  1.7309e-06     	|  6.3e-07        	|
|  [Callback]GradientAccumulationScheduler.on_save_checkpoint                                                                                                                                                	|  1.7066e-06     	|  1              	|  1.7066e-06     	|  6.2118e-07     	|
|  [Callback]ModelSummary.on_pretrain_routine_end                                                                                                                                                            	|  1.6722e-06     	|  1              	|  1.6722e-06     	|  6.0864e-07     	|
|  [LightningModule]LitModular.on_epoch_end                                                                                                                                                                  	|  8.3237e-07     	|  2              	|  1.6647e-06     	|  6.0593e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_save_checkpoint                 	|  1.6647e-06     	|  1              	|  1.6647e-06     	|  6.0593e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  1.6568e-06     	|  1              	|  1.6568e-06     	|  6.0305e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                           	|  1.648e-06      	|  1              	|  1.648e-06      	|  5.9983e-07     	|
|  [Callback]ModelSummary.teardown                                                                                                                                                                           	|  1.634e-06      	|  1              	|  1.634e-06      	|  5.9474e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  1.6172e-06     	|  1              	|  1.6172e-06     	|  5.8864e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                        	|  1.5721e-06     	|  1              	|  1.5721e-06     	|  5.722e-07      	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                          	|  1.5679e-06     	|  1              	|  1.5679e-06     	|  5.7067e-07     	|
|  [LightningModule]LitModular.on_train_dataloader                                                                                                                                                           	|  1.4598e-06     	|  1              	|  1.4598e-06     	|  5.3135e-07     	|
|  [LightningModule]LitModular.configure_sharded_model                                                                                                                                                       	|  1.2643e-06     	|  1              	|  1.2643e-06     	|  4.6017e-07     	|
|  [LightningModule]LitModular.on_save_checkpoint                                                                                                                                                            	|  1.2368e-06     	|  1              	|  1.2368e-06     	|  4.5017e-07     	|
|  [LightningModule]LitModular.on_fit_start                                                                                                                                                                  	|  1.0701e-06     	|  1              	|  1.0701e-06     	|  3.8949e-07     	|
|  [LightningModule]LitModular.on_train_epoch_start                                                                                                                                                          	|  1.03e-06       	|  1              	|  1.03e-06       	|  3.7491e-07     	|
|  [LightningModule]LitModular.on_validation_epoch_end                                                                                                                                                       	|  1.011e-06      	|  1              	|  1.011e-06      	|  3.6796e-07     	|
|  [LightningModule]LitModular.prepare_data                                                                                                                                                                  	|  9.9884e-07     	|  1              	|  9.9884e-07     	|  3.6356e-07     	|
|  [LightningModule]LitModular.on_fit_end                                                                                                                                                                    	|  9.9419e-07     	|  1              	|  9.9419e-07     	|  3.6186e-07     	|
|  [LightningModule]LitModular.on_train_start                                                                                                                                                                	|  9.6112e-07     	|  1              	|  9.6112e-07     	|  3.4983e-07     	|
|  [LightningModule]LitModular.on_train_end                                                                                                                                                                  	|  9.4669e-07     	|  1              	|  9.4669e-07     	|  3.4457e-07     	|
|  [LightningModule]LitModular.on_validation_end                                                                                                                                                             	|  9.3924e-07     	|  1              	|  9.3924e-07     	|  3.4186e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                             	|  9.1689e-07     	|  1              	|  9.1689e-07     	|  3.3373e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                               	|  9.1502e-07     	|  1              	|  9.1502e-07     	|  3.3305e-07     	|
|  [LightningModule]LitModular.on_pretrain_routine_start                                                                                                                                                     	|  9.099e-07      	|  1              	|  9.099e-07      	|  3.3118e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                          	|  7.5484e-07     	|  1              	|  7.5484e-07     	|  2.7474e-07     	|
|  [LightningModule]LitModular.on_pretrain_routine_end                                                                                                                                                       	|  7.3202e-07     	|  1              	|  7.3202e-07     	|  2.6644e-07     	|
|  [LightningModule]LitModular.teardown                                                                                                                                                                      	|  7.1805e-07     	|  1              	|  7.1805e-07     	|  2.6135e-07     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

With torchmetrics downgrade:

FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                    	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                     	|  -              	|  2254           	|  261.39         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                        	|  255.2          	|  1              	|  255.2          	|  97.634         	|
|  run_training_batch                                                                                                                                                                                        	|  4.8919         	|  50             	|  244.59         	|  93.576         	|
|  [LightningModule]LitModular.optimizer_step                                                                                                                                                                	|  4.8902         	|  50             	|  244.51         	|  93.543         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                   	|  2.8397         	|  50             	|  141.98         	|  54.319         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                              	|  2.0212         	|  50             	|  101.06         	|  38.662         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                            	|  2.0902         	|  2              	|  4.1804         	|  1.5993         	|
|  [LightningModule]LitModular.training_epoch_end                                                                                                                                                            	|  0.98194        	|  1              	|  0.98194        	|  0.37567        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.49866        	|  1              	|  0.49866        	|  0.19077        	|
|  [LightningModule]LitModular.optimizer_zero_grad                                                                                                                                                           	|  0.0051409      	|  50             	|  0.25705        	|  0.098339       	|
|  [LightningModule]LitModular.validation_epoch_end                                                                                                                                                          	|  0.22674        	|  1              	|  0.22674        	|  0.086747       	|
|  on_train_batch_end                                                                                                                                                                                        	|  0.00087542     	|  50             	|  0.043771       	|  0.016746       	|
|  [LightningModule]LitModular.configure_optimizers                                                                                                                                                          	|  0.007784       	|  1              	|  0.007784       	|  0.002978       	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                            	|  0.00011752     	|  52             	|  0.0061112      	|  0.002338       	|
|  on_train_batch_start                                                                                                                                                                                      	|  9.6073e-05     	|  50             	|  0.0048037      	|  0.0018378      	|
|  [LightningModule]LitModular.training_step_end                                                                                                                                                             	|  5.6877e-05     	|  50             	|  0.0028439      	|  0.001088       	|
|  [LightningModule]LitModular.lr_scheduler_step                                                                                                                                                             	|  4.7692e-05     	|  50             	|  0.0023846      	|  0.00091229     	|
|  [Callback]ModelSummary.on_fit_start                                                                                                                                                                       	|  0.0023354      	|  1              	|  0.0023354      	|  0.00089345     	|
|  [Callback]TQDMProgressBar.on_sanity_check_start                                                                                                                                                           	|  0.0012566      	|  1              	|  0.0012566      	|  0.00048075     	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                         	|  0.00034418     	|  2              	|  0.00068836     	|  0.00026335     	|
|  [LightningModule]LitModular.on_validation_model_train                                                                                                                                                     	|  0.00058326     	|  1              	|  0.00058326     	|  0.00022314     	|
|  [LightningModule]LitModular.on_validation_model_eval                                                                                                                                                      	|  0.00048213     	|  1              	|  0.00048213     	|  0.00018445     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                              	|  0.00047033     	|  1              	|  0.00047033     	|  0.00017994     	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                               	|  6.6997e-06     	|  50             	|  0.00033498     	|  0.00012816     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                       	|  0.00015666     	|  2              	|  0.00031333     	|  0.00011987     	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                    	|  0.00024196     	|  1              	|  0.00024196     	|  9.2568e-05     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                               	|  0.00022126     	|  1              	|  0.00022126     	|  8.465e-05      	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                            	|  0.00021002     	|  1              	|  0.00021002     	|  8.035e-05      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                             	|  3.6187e-06     	|  50             	|  0.00018094     	|  6.9222e-05     	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                    	|  3.2456e-06     	|  50             	|  0.00016228     	|  6.2084e-05     	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                  	|  3.0393e-06     	|  50             	|  0.00015196     	|  5.8138e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                  	|  0.00014885     	|  1              	|  0.00014885     	|  5.6946e-05     	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                              	|  2.8566e-06     	|  50             	|  0.00014283     	|  5.4644e-05     	|
|  [Callback]ModelSummary.on_after_backward                                                                                                                                                                  	|  2.7071e-06     	|  50             	|  0.00013535     	|  5.1783e-05     	|
|  [LightningModule]LitModular.validation_step_end                                                                                                                                                           	|  6.1162e-05     	|  2              	|  0.00012232     	|  4.6798e-05     	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                        	|  2.4232e-06     	|  50             	|  0.00012116     	|  4.6354e-05     	|
|  [Callback]ModelSummary.on_batch_start                                                                                                                                                                     	|  2.0801e-06     	|  50             	|  0.000104       	|  3.979e-05      	|
|  [Callback]ModelSummary.on_before_zero_grad                                                                                                                                                                	|  2.0106e-06     	|  50             	|  0.00010053     	|  3.846e-05      	|
|  [Callback]ModelSummary.on_batch_end                                                                                                                                                                       	|  1.9792e-06     	|  50             	|  9.896e-05      	|  3.786e-05      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                    	|  1.8475e-06     	|  50             	|  9.2375e-05     	|  3.534e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  1.8275e-06     	|  50             	|  9.1374e-05     	|  3.4958e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  1.8205e-06     	|  50             	|  9.1027e-05     	|  3.4825e-05     	|
|  [Callback]ModelSummary.on_before_backward                                                                                                                                                                 	|  1.7928e-06     	|  50             	|  8.9641e-05     	|  3.4295e-05     	|
|  [LightningModule]LitModular.on_train_batch_end                                                                                                                                                            	|  1.7903e-06     	|  50             	|  8.9514e-05     	|  3.4246e-05     	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                 	|  1.78e-06       	|  50             	|  8.9002e-05     	|  3.405e-05      	|
|  [Callback]ModelSummary.on_before_optimizer_step                                                                                                                                                           	|  1.7749e-06     	|  50             	|  8.8747e-05     	|  3.3952e-05     	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                          	|  1.769e-06      	|  50             	|  8.845e-05      	|  3.3839e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  1.7118e-06     	|  50             	|  8.5589e-05     	|  3.2744e-05     	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                               	|  1.7034e-06     	|  50             	|  8.5168e-05     	|  3.2583e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  1.6997e-06     	|  50             	|  8.4986e-05     	|  3.2514e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  1.6982e-06     	|  50             	|  8.4911e-05     	|  3.2485e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  1.6947e-06     	|  50             	|  8.4733e-05     	|  3.2417e-05     	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                      	|  1.675e-06      	|  50             	|  8.3749e-05     	|  3.204e-05      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                	|  1.6588e-06     	|  50             	|  8.2941e-05     	|  3.1731e-05     	|
|  [LightningModule]LitModular.on_after_backward                                                                                                                                                             	|  1.5323e-06     	|  50             	|  7.6613e-05     	|  2.931e-05      	|
|  [LightningModule]LitModular.on_before_backward                                                                                                                                                            	|  1.4007e-06     	|  50             	|  7.0037e-05     	|  2.6794e-05     	|
|  [LightningModule]LitModular.on_before_zero_grad                                                                                                                                                           	|  1.3762e-06     	|  50             	|  6.881e-05      	|  2.6325e-05     	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                          	|  1.376e-06      	|  50             	|  6.88e-05       	|  2.6321e-05     	|
|  [LightningModule]LitModular.on_train_batch_start                                                                                                                                                          	|  1.2468e-06     	|  50             	|  6.2338e-05     	|  2.3849e-05     	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                       	|  1.12e-06       	|  50             	|  5.6002e-05     	|  2.1425e-05     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.setup                              	|  4.8796e-05     	|  1              	|  4.8796e-05     	|  1.8668e-05     	|
|  [LightningModule]LitModular.on_before_optimizer_step                                                                                                                                                      	|  8.5433e-07     	|  50             	|  4.2717e-05     	|  1.6342e-05     	|
|  [Callback]ModelSummary.on_sanity_check_start                                                                                                                                                              	|  2.1686e-05     	|  1              	|  2.1686e-05     	|  8.2967e-06     	|
|  [Callback]ModelSummary.on_validation_batch_end                                                                                                                                                            	|  6.5544e-06     	|  2              	|  1.3109e-05     	|  5.0151e-06     	|
|  [Callback]ModelSummary.on_validation_batch_start                                                                                                                                                          	|  5.0073e-06     	|  2              	|  1.0015e-05     	|  3.8313e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  9.9023e-06     	|  1              	|  9.9023e-06     	|  3.7884e-06     	|
|  [Callback]GradientAccumulationScheduler.on_sanity_check_start                                                                                                                                             	|  7.771e-06      	|  1              	|  7.771e-06      	|  2.973e-06      	|
|  [Callback]TQDMProgressBar.on_sanity_check_end                                                                                                                                                             	|  6.7828e-06     	|  1              	|  6.7828e-06     	|  2.5949e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_sanity_check_start              	|  6.597e-06      	|  1              	|  6.597e-06      	|  2.5239e-06     	|
|  [Callback]ModelSummary.on_train_epoch_end                                                                                                                                                                 	|  6.5509e-06     	|  1              	|  6.5509e-06     	|  2.5062e-06     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                      	|  5.4589e-06     	|  1              	|  5.4589e-06     	|  2.0885e-06     	|
|  [Callback]ModelSummary.on_train_end                                                                                                                                                                       	|  5.3649e-06     	|  1              	|  5.3649e-06     	|  2.0525e-06     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                             	|  5.2089e-06     	|  1              	|  5.2089e-06     	|  1.9928e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                           	|  5.1842e-06     	|  1              	|  5.1842e-06     	|  1.9834e-06     	|
|  [Callback]ModelSummary.on_validation_end                                                                                                                                                                  	|  4.8801e-06     	|  1              	|  4.8801e-06     	|  1.867e-06      	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                    	|  2.404e-06      	|  2              	|  4.808e-06      	|  1.8394e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                           	|  2.2328e-06     	|  2              	|  4.4657e-06     	|  1.7085e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                         	|  2.1548e-06     	|  2              	|  4.3097e-06     	|  1.6488e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                  	|  2.0294e-06     	|  2              	|  4.0587e-06     	|  1.5528e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  1.9802e-06     	|  2              	|  3.9604e-06     	|  1.5152e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  1.9793e-06     	|  2              	|  3.9586e-06     	|  1.5145e-06     	|
|  [Callback]TQDMProgressBar.on_save_checkpoint                                                                                                                                                              	|  3.9348e-06     	|  1              	|  3.9348e-06     	|  1.5054e-06     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                      	|  3.506e-06      	|  1              	|  3.506e-06      	|  1.3413e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                              	|  3.485e-06      	|  1              	|  3.485e-06      	|  1.3333e-06     	|
|  [Callback]ModelSummary.on_train_start                                                                                                                                                                     	|  3.4212e-06     	|  1              	|  3.4212e-06     	|  1.3089e-06     	|
|  [Callback]ModelSummary.on_epoch_start                                                                                                                                                                     	|  1.6654e-06     	|  2              	|  3.3309e-06     	|  1.2743e-06     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                    	|  3.316e-06      	|  1              	|  3.316e-06      	|  1.2686e-06     	|
|  [Callback]ModelSummary.on_train_epoch_start                                                                                                                                                               	|  3.234e-06      	|  1              	|  3.234e-06      	|  1.2373e-06     	|
|  [Callback]ModelSummary.on_epoch_end                                                                                                                                                                       	|  1.6133e-06     	|  2              	|  3.2266e-06     	|  1.2344e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  1.6112e-06     	|  2              	|  3.2224e-06     	|  1.2328e-06     	|
|  [LightningModule]LitModular.on_val_dataloader                                                                                                                                                             	|  3.1828e-06     	|  1              	|  3.1828e-06     	|  1.2177e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                    	|  1.5849e-06     	|  2              	|  3.1698e-06     	|  1.2127e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                      	|  1.5735e-06     	|  2              	|  3.1469e-06     	|  1.2039e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  1.5707e-06     	|  2              	|  3.1414e-06     	|  1.2018e-06     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                         	|  3.0426e-06     	|  1              	|  3.0426e-06     	|  1.164e-06      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  3.011e-06      	|  1              	|  3.011e-06      	|  1.1519e-06     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                      	|  2.7022e-06     	|  1              	|  2.7022e-06     	|  1.0338e-06     	|
|  [LightningModule]LitModular.on_validation_batch_end                                                                                                                                                       	|  1.3215e-06     	|  2              	|  2.6431e-06     	|  1.0112e-06     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                       	|  2.6268e-06     	|  1              	|  2.6268e-06     	|  1.0049e-06     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                             	|  2.5672e-06     	|  1              	|  2.5672e-06     	|  9.8215e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.4638e-06     	|  1              	|  2.4638e-06     	|  9.426e-07      	|
|  [Callback]ModelSummary.setup                                                                                                                                                                              	|  2.4578e-06     	|  1              	|  2.4578e-06     	|  9.4028e-07     	|
|  [Callback]ModelSummary.on_sanity_check_end                                                                                                                                                                	|  2.4261e-06     	|  1              	|  2.4261e-06     	|  9.2817e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                        	|  2.3651e-06     	|  1              	|  2.3651e-06     	|  9.0483e-07     	|
|  [Callback]ModelSummary.on_validation_start                                                                                                                                                                	|  2.339e-06      	|  1              	|  2.339e-06      	|  8.9485e-07     	|
|  [LightningModule]LitModular.on_validation_batch_start                                                                                                                                                     	|  1.1616e-06     	|  2              	|  2.3232e-06     	|  8.8879e-07     	|
|  [LightningModule]LitModular.on_train_epoch_end                                                                                                                                                            	|  2.299e-06      	|  1              	|  2.299e-06      	|  8.7953e-07     	|
|  [Callback]ModelSummary.on_save_checkpoint                                                                                                                                                                 	|  2.2682e-06     	|  1              	|  2.2682e-06     	|  8.6777e-07     	|
|  [LightningModule]LitModular.configure_callbacks                                                                                                                                                           	|  2.2417e-06     	|  1              	|  2.2417e-06     	|  8.5762e-07     	|
|  [LightningModule]LitModular.on_epoch_end                                                                                                                                                                  	|  1.0536e-06     	|  2              	|  2.1071e-06     	|  8.0613e-07     	|
|  [Callback]ModelSummary.on_before_accelerator_backend_setup                                                                                                                                                	|  2.0801e-06     	|  1              	|  2.0801e-06     	|  7.958e-07      	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                 	|  2.0331e-06     	|  1              	|  2.0331e-06     	|  7.7781e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                	|  2.0103e-06     	|  1              	|  2.0103e-06     	|  7.6908e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                         	|  2.0098e-06     	|  1              	|  2.0098e-06     	|  7.689e-07      	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                        	|  9.9605e-07     	|  2              	|  1.9921e-06     	|  7.6213e-07     	|
|  [Callback]ModelSummary.on_validation_epoch_end                                                                                                                                                            	|  1.9921e-06     	|  1              	|  1.9921e-06     	|  7.6213e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                    	|  1.9777e-06     	|  1              	|  1.9777e-06     	|  7.5661e-07     	|
|  [Callback]ModelSummary.on_configure_sharded_model                                                                                                                                                         	|  1.946e-06      	|  1              	|  1.946e-06      	|  7.4449e-07     	|
|  [LightningModule]LitModular.on_epoch_start                                                                                                                                                                	|  9.6462e-07     	|  2              	|  1.9292e-06     	|  7.3808e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  1.9283e-06     	|  1              	|  1.9283e-06     	|  7.3772e-07     	|
|  [Callback]ModelSummary.on_fit_end                                                                                                                                                                         	|  1.9274e-06     	|  1              	|  1.9274e-06     	|  7.3737e-07     	|
|  [Callback]ModelSummary.on_pretrain_routine_start                                                                                                                                                          	|  1.9092e-06     	|  1              	|  1.9092e-06     	|  7.3042e-07     	|
|  [Callback]GradientAccumulationScheduler.on_sanity_check_end                                                                                                                                               	|  1.899e-06      	|  1              	|  1.899e-06      	|  7.265e-07      	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                       	|  1.8971e-06     	|  1              	|  1.8971e-06     	|  7.2579e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.on_before_accelerator_backend_setup	|  1.8808e-06     	|  1              	|  1.8808e-06     	|  7.1955e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  1.8533e-06     	|  1              	|  1.8533e-06     	|  7.0904e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                               	|  1.8431e-06     	|  1              	|  1.8431e-06     	|  7.0512e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                             	|  1.8128e-06     	|  1              	|  1.8128e-06     	|  6.9354e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  1.8128e-06     	|  1              	|  1.8128e-06     	|  6.9354e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                         	|  1.811e-06      	|  1              	|  1.811e-06      	|  6.9283e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  1.7993e-06     	|  1              	|  1.7993e-06     	|  6.8837e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                               	|  1.7872e-06     	|  1              	|  1.7872e-06     	|  6.8374e-07     	|
|  [Callback]GradientAccumulationScheduler.on_save_checkpoint                                                                                                                                                	|  1.7867e-06     	|  1              	|  1.7867e-06     	|  6.8356e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                           	|  1.7839e-06     	|  1              	|  1.7839e-06     	|  6.825e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.776e-06      	|  1              	|  1.776e-06      	|  6.7947e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_sanity_check_end                	|  1.75e-06       	|  1              	|  1.75e-06       	|  6.6949e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                      	|  1.7318e-06     	|  1              	|  1.7318e-06     	|  6.6254e-07     	|
|  [Callback]ModelSummary.on_pretrain_routine_end                                                                                                                                                            	|  1.7211e-06     	|  1              	|  1.7211e-06     	|  6.5845e-07     	|
|  [Callback]ModelSummary.teardown                                                                                                                                                                           	|  1.715e-06      	|  1              	|  1.715e-06      	|  6.5613e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                           	|  1.7127e-06     	|  1              	|  1.7127e-06     	|  6.5524e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  1.6917e-06     	|  1              	|  1.6917e-06     	|  6.4722e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                        	|  1.6792e-06     	|  1              	|  1.6792e-06     	|  6.4241e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                          	|  1.675e-06      	|  1              	|  1.675e-06      	|  6.4081e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  1.668e-06      	|  1              	|  1.668e-06      	|  6.3814e-07     	|
|  [Callback]ModelSummary.on_validation_epoch_start                                                                                                                                                          	|  1.655e-06      	|  1              	|  1.655e-06      	|  6.3315e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  1.655e-06      	|  1              	|  1.655e-06      	|  6.3315e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_save_checkpoint                 	|  1.6508e-06     	|  1              	|  1.6508e-06     	|  6.3154e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  1.6452e-06     	|  1              	|  1.6452e-06     	|  6.2941e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  1.6429e-06     	|  1              	|  1.6429e-06     	|  6.2852e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                        	|  1.6131e-06     	|  1              	|  1.6131e-06     	|  6.1711e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                         	|  1.573e-06      	|  1              	|  1.573e-06      	|  6.0179e-07     	|
|  [LightningModule]LitModular.on_train_dataloader                                                                                                                                                           	|  1.5548e-06     	|  1              	|  1.5548e-06     	|  5.9485e-07     	|
|  [LightningModule]LitModular.configure_sharded_model                                                                                                                                                       	|  1.553e-06      	|  1              	|  1.553e-06      	|  5.9413e-07     	|
|  [LightningModule]LitModular.on_save_checkpoint                                                                                                                                                            	|  1.2894e-06     	|  1              	|  1.2894e-06     	|  4.933e-07      	|
|  [LightningModule]LitModular.on_pretrain_routine_start                                                                                                                                                     	|  1.282e-06      	|  1              	|  1.282e-06      	|  4.9045e-07     	|
|  [LightningModule]LitModular.on_train_end                                                                                                                                                                  	|  1.2391e-06     	|  1              	|  1.2391e-06     	|  4.7406e-07     	|
|  [LightningModule]LitModular.setup                                                                                                                                                                         	|  1.2121e-06     	|  1              	|  1.2121e-06     	|  4.6373e-07     	|
|  [LightningModule]LitModular.on_validation_end                                                                                                                                                             	|  1.2e-06        	|  1              	|  1.2e-06        	|  4.5909e-07     	|
|  [LightningModule]LitModular.prepare_data                                                                                                                                                                  	|  1.078e-06      	|  1              	|  1.078e-06      	|  4.1242e-07     	|
|  [LightningModule]LitModular.on_fit_start                                                                                                                                                                  	|  1.072e-06      	|  1              	|  1.072e-06      	|  4.101e-07      	|
|  [LightningModule]LitModular.on_validation_start                                                                                                                                                           	|  1.0622e-06     	|  1              	|  1.0622e-06     	|  4.0636e-07     	|
|  [LightningModule]LitModular.on_train_start                                                                                                                                                                	|  9.9465e-07     	|  1              	|  9.9465e-07     	|  3.8053e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                             	|  9.8394e-07     	|  1              	|  9.8394e-07     	|  3.7643e-07     	|
|  [LightningModule]LitModular.on_train_epoch_start                                                                                                                                                          	|  9.7184e-07     	|  1              	|  9.7184e-07     	|  3.718e-07      	|
|  [LightningModule]LitModular.on_validation_epoch_end                                                                                                                                                       	|  9.4716e-07     	|  1              	|  9.4716e-07     	|  3.6236e-07     	|
|  [LightningModule]LitModular.on_fit_end                                                                                                                                                                    	|  8.8895e-07     	|  1              	|  8.8895e-07     	|  3.4009e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                               	|  8.028e-07      	|  1              	|  8.028e-07      	|  3.0713e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                          	|  7.9768e-07     	|  1              	|  7.9768e-07     	|  3.0517e-07     	|
|  [LightningModule]LitModular.on_validation_epoch_start                                                                                                                                                     	|  7.893e-07      	|  1              	|  7.893e-07      	|  3.0197e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                        	|  7.7905e-07     	|  1              	|  7.7905e-07     	|  2.9805e-07     	|
|  [LightningModule]LitModular.teardown                                                                                                                                                                      	|  7.5903e-07     	|  1              	|  7.5903e-07     	|  2.9039e-07     	|
|  [LightningModule]LitModular.on_pretrain_routine_end                                                                                                                                                       	|  7.3109e-07     	|  1              	|  7.3109e-07     	|  2.797e-07      	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

dxoigmn added 6 commits December 9, 2022 10:59

Downgrade torchmetrics to ~=0.6.0

af955f8

Remove use if private function

a6555d8

Fix configuration to use old torchmetrics

3d34c8f

Downgrade torchmetrics to ==0.6.0

6542ab0

Flatten metrics

690cd17

Be more forward compatible

30ad3ac

dxoigmn mentioned this pull request Dec 9, 2022

Downgrade torchmetrics to 0.6.0 #18

Closed

12 tasks

dxoigmn requested a review from mzweilin December 9, 2022 20:25

Add a comment about forward compatibility.

d77fa56

mzweilin approved these changes Dec 9, 2022

View reviewed changes

mzweilin merged commit 5eed680 into main Dec 9, 2022

mzweilin deleted the downgrade_torchmetrics branch December 9, 2022 21:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Downgrade torchmetrics to 0.6.0 #19

Downgrade torchmetrics to 0.6.0 #19

dxoigmn commented Dec 9, 2022

mzweilin left a comment

lumurillo commented Dec 9, 2022

Downgrade torchmetrics to 0.6.0 #19

Downgrade torchmetrics to 0.6.0 #19

Conversation

dxoigmn commented Dec 9, 2022

What does this PR do?

Type of change

Testing

Before submitting

Did you have fun?

mzweilin left a comment

Choose a reason for hiding this comment

lumurillo commented Dec 9, 2022