Question about layout for groupmatmul

[SunRuiXin]

I have a question: In the cublasLt library, why are the pointers for GroupMM support (such as CUBLASLT_GROUPED_MATRIX_LAYOUT_COLS_ARRAY, CUBLASLT_GROUPED_MATRIX_LAYOUT_LD_ARRAY, etc.) all pointing to device memory?

[rsdubtso]

Hi @SunRuiXin. It is because it is meant to be used in MoE models where router produces shape information on the device to avoid having to transfer this information back to host which requires synchronization.

[SunRuiXin]

Hi @SunRuiXin. It is because it is meant to be used in MoE models where router produces shape information on the device to avoid having to transfer this information back to host which requires synchronization.

Thank you for the clarification.