Request for padded generic kernel

[cloudhan]

Currently, composable kernel supplies kernels with very sparse shape (runtime parameters) support for an individual kernel. To use this library, we need to heavily rely on tuning mechanism to find a supported (as well as faster) one. Here we want to have the default kernels (that is, the kerenls come out of InstanceFactory with index 0) to support runtime parameters densely. We want thoes ( padded generic) kernels to support all runtime parameter combination, and impl->IsSupportedArgument(arg), the padded generic ones must always return true.

• FlashAttention
• Gemm + elementwise
• pure elementwise kernels

[PeixuanZuo]

Add more kernel here:

• GroupNorm
• Softmax

[zjing14]

@cloudhan @PeixuanZuo Could you provide the exact DeviceOp (e.g., DeviceBatchedGemmSoftmaxGemmPermute_Xdl_CShuffle) you are using, since for an Op we may have multiple DeviceOps? Thanks

[cloudhan]

For the following two, we are directly instantiating device op in our code, instead of relying on client api

• DeviceBatchedGemmSoftmaxGemmPermute_Xdl_CShuffle for flash attention
• DeviceNormalizationImpl for GroupNorm

following are using client api

• ck::tensor_operation::device::DeviceSoftmax
• ck::tensor_operation::device::DeviceGemmMultipleD for gemm + elementwise
• ck::tensor_operation::device::DeviceElementwiseImpl for pure elementwise

[qianfengz]

For each DeviceOperationInstanceFactory class, should we provide a GetGenericInstance() method to return the unique generic instance ?

[cloudhan]

Yeah, that is the plan. It does not need to be that performant, but it should be generic enough to cover a large portion of common use cases.

[zjing14]

@cloudhan all PRs are merged. Please check.