-
Notifications
You must be signed in to change notification settings - Fork 240
Description
Hi, I had some errors when running the repo on GH200 (Hopper architecture). It seems the logs indicate hardcoding certain GPU configs which does fit into the newer GPU type. But I cannot find where such hardcode comes from. Could you please give any suggestions to avoid such errors? Thanks for the help!
The running logs are as below. And after a while, the code will be ended with "CUDA error: an illegal memory access was encountered":
"Starting work on task 0
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 117 SMs (39 CGAs with 3 CTAs each)
Running with some CTAs in CGA Out of bounds
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 117 SMs (39 CGAs with 3 CTAs each)
Running with some CTAs in CGA Out of bounds
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 117 SMs (39 CGAs with 3 CTAs each)
Running with some CTAs in CGA Out of bounds
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 117 SMs (39 CGAs with 3 CTAs each)
Running with some CTAs in CGA Out of bounds
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 132 SMs (132 CGAs with 1 CTAs each)
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 132 SMs (132 CGAs with 1 CTAs each)
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 132 SMs (132 CGAs with 1 CTAs each)
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 132 SMs (132 CGAs with 1 CTAs each)
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 132 SMs (66 CGAs with 2 CTAs each)
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 132 SMs (66 CGAs with 2 CTAs each)
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 132 SMs (66 CGAs with 2 CTAs each)
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 132 SMs (66 CGAs with 2 CTAs each)
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 132 SMs (66 CGAs with 2 CTAs each)
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 132 SMs (66 CGAs with 2 CTAs each)
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 105 SMs (15 CGAs with 7 CTAs each)
Skyline(sm_counts_gpc) specified by user = 816161616161818000200200202000000000000
=> Using 105 SMs (15 CGAs with 7 CTAs each)"