Some common CUDA kernel implementations (Not the fastest).

Fast Convoluion Implementation via CUDA

Fast SGEMM emulation on Tensor Cores

My attempt of making a GEMM kernel...

HPC course for Grad 3/4th 2021

CUDA Kernels on V100

CUDA optimization samples including sgemm, reduce... To be continued.

FP64 equivalent GEMM via Int8 Tensor Cores using the Ozaki scheme

CUDA kernel functions

Use tensor core to calculate back-to-back HGEMM (half-precision general matrix multiplication) with MMA PTX instruction.

PyTorch half precision gemm lib w/ fused optional bias + optional relu/gelu

The simplest but fast implementation of matrix multiplication in CUDA.

Several optimization methods of half-precision general matrix vector multiplication (HGEMV) using CUDA core.

Code for testing the native float16 matrix multiplication performance on Tesla P100 and V100 GPU based on cublasHgemm

code for benchmarking GPU performance based on cublasSgemm and cublasHgemm

Optimizing SGEMM kernel functions on NVIDIA GPUs to a close-to-cuBLAS performance.

Several optimization methods of half-precision general matrix multiplication (HGEMM) using tensor core with WMMA API and MMA PTX instruction.

🎉 Modern CUDA Learn Notes with PyTorch: CUDA Cores, Tensor Cores, fp32/tf32, fp16/bf16, fp8/int8, flash_attn, rope, sgemm, hgemm, sgemv, warp/block reduce, elementwise, softmax, layernorm, rmsnorm.