diff --git a/_posts/2025-08-11-cuda-debugging.md b/_posts/2025-08-11-cuda-debugging.md index a45e9a3..02ef393 100644 --- a/_posts/2025-08-11-cuda-debugging.md +++ b/_posts/2025-08-11-cuda-debugging.md @@ -5,6 +5,15 @@ author: "Kaichao You" image: /assets/logos/vllm-logo-text-light.png --- +TL;DR: If you hit `an illegal memory access was encountered` error, you can enable CUDA core dump to debug the issue. Simply set the following environment variables and run your program again to collect the coredump file, then you can use `cuda-gdb` to debug the issue. + +```bash +CUDA_ENABLE_COREDUMP_ON_EXCEPTION=1 \ +CUDA_COREDUMP_SHOW_PROGRESS=1 \ +CUDA_COREDUMP_GENERATION_FLAGS='skip_nonrelocated_elf_images,skip_global_memory,skip_shared_memory,skip_local_memory,skip_constbank_memory' \ +CUDA_COREDUMP_FILE="/tmp/cuda_coredump_%h.%p.%t" +``` + # Introduction Have you ever felt you are developing cuda kernels and your tests often run into illegal memory access (IMA for short) and you have no idea how to debug? We definitely felt this pain again and again while working on vLLM, a high-performance inference engine for LLM models.