How can we profile each layer?

[Adnios]

The profiling results are not separated by layer, but rather by operation. But I really appreciate it if we can profile each layer.

[TristonC]

I don't think the current profiler can. @szha @ptrendx Do you have any suggestions on this request?

[szha]

The layer structure is only in the frontend and thus is not preserved in the backend at the time of scheduling/profiling. One workaround could be to only hybridize some children blocks in which case they become a whole cached op in the profiler result.

[ptrendx]

Generally speaking at least for GPU operation it is preferred to use a dedicated profiler like NSight Systems. I did start a PR quite some time ago (#18683) which I never had time to finish (@DickJC123, could you pick it up?), which expands the NVTX markers (visible in the output of NSight Systems) to include the names of the ops - then you could see the ops together with the GPU kernels in the timeline. That support actually exists in the NGC container (https://ngc.nvidia.com/catalog/containers/nvidia:mxnet) but it is based on the MXNet 1.x currently.