torch.profiler support for tf32 #153901
Description
🚀 The feature, motivation and pitch
torch.profiler saves the equivalent of tensor.scalar_type when you save a Tensor and record_shapes is on. However, it's not clear whether these tensors are being run in the context of tf32 being enabled or not; this will always show up as an fp32. This matters when we try to estimate the achieved flops of an operation because the number will be much higher if tf32 is enabled.
Alternatives
After talking with @sraikund16, ideally we'd store this information as the dtype of the tensor so the profiler wouldn't be responsible for this and it could just flow through, but since tf32 isn't a data type, we'll probably have to just add a boolean value to the trace somewhere.
cc: @eellison
Additional context
No response
cc @robieta @chaekit @guotuofeng @guyang3532 @dzhulgakov @davidberard98 @briancoutinho @sraikund16 @sanrise