Total time in Simpleprofiler does not add up to the function run-times

[Abhishaike]

Bug description

Used the SimpleProfiler while training a model with DDP, and found that the constituent times of the functions did not add up to the total. Is this a bug?

Alternatively, do these calls each represent one GPU's worth of profiling? Like, should I multiply each of the function calls by my number of GPU's (4 in this case) to get a better estimate?

Alternatively alternatively, should I view the bulk of the 'total time' as the 'cost' of the DDP spin-up? If so, should this be documented somewhere?

cc @carmocca @kaushikb11 @ninginthecloud @rohitgr7 @nbcsm @guotuofeng

[rohitgr7]

they won't add up because some of them includes the time of other method calls. For eg: run_training_batch is part of run_training_epoch.

[Abhishaike]

This makes sense, but there nevertheless is a massive discrepancy between any given time and the total time

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!