[BUG] Crash in VDB with self contact resolution and graph capturing

[CliffieVanR]

Bug Description

We're experiencing a consistent CUDA access violation when running VDB with self0collisions enabled and using the graph capturing functionality. You can repro by switching to the VDB integrator in example_cloth.py, enabling self contact resolution, and enabling the graph capture. The crash appears to be during the first attempt to execute the previously captured graph. There's a call to array_fill_device which in turns calls alloc_device, alloc_device_async and finally a crash inside cuda_stream_is_capturing.

System Information

No response

[shi-eric]

Thanks for reporting! @AnkaChan can you look into this?

[AnkaChan]

Hi Clifflie can you tell me what is the version of Warp you are using? We had this bug in 1.5.0 but it's fixe din 1.6.0. Also can you tell me what is the CUDA and Python version you are using?

[CliffieVanR]

Hi Anka, We're using 1.6.0. Cuda 12.8, driver 572.16, and Python 3.12.9. I've just double-checked and am still seeing the illegal memory accesses when using the VDB integrator with handle_self_contact = True in example_cloth.py. I'm curious why the graph capture provides such a significant speed improvement. Are the kernel launches in Python synchronous? Best, Cliff

…

On Thu, Mar 20, 2025 at 6:20 PM Anka Chen ***@***.***> wrote: Hi Clifflie can you tell me what is the version of Warp you are using? We had this bug in 1.5.0 but it's fixe din 1.6.0. Also can you tell me what is the CUDA and Python version you are using? — Reply to this email directly, view it on GitHub <#587 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPMNJMCPMBIOGU3Z3PINUNL2VMBFDAVCNFSM6AAAAABZNTB4G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBRGMYDSMJZGE> . You are receiving this because you authored the thread.Message ID: ***@***.***> [image: AnkaChan]*AnkaChan* left a comment (NVIDIA/warp#587) <#587 (comment)> Hi Clifflie can you tell me what is the version of Warp you are using? We had this bug in 1.5.0 but it's fixe din 1.6.0. Also can you tell me what is the CUDA and Python version you are using? — Reply to this email directly, view it on GitHub <#587 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPMNJMCPMBIOGU3Z3PINUNL2VMBFDAVCNFSM6AAAAABZNTB4G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBRGMYDSMJZGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[shi-eric]

Wait, 1.6.1 is the version that fixed the graph capture bug. See the release notes: https://github.com/NVIDIA/warp/releases/tag/v1.6.1

[shi-eric]

Hi Anka,

We're using 1.6.0. Cuda 12.8, driver 572.16, and Python 3.12.9. I've just
double-checked and am still seeing the illegal memory accesses when using
the VDB integrator with handle_self_contact = True in example_cloth.py.

I'm curious why the graph capture provides such a significant speed
improvement. Are the kernel launches in Python synchronous?

Best,
Cliff
…

To answer the second question, the kernel launches are not asynchronous, but they have two big sources of overhead (assume the kernel is already compiled). One is the Python overhead, which is basically all the operations that happen from an invocation of wp.launch() in the Python interpreter to the kernel launch actually being triggered using the CUDA driver API in the C code:

warp/warp/native/warp.cu

Lines 3483 to 3535 in f3814e7

	size_t cuda_launch_kernel(void* context, void* kernel, size_t dim, int max_blocks, int block_dim, int shared_memory_bytes, void** args, void* stream)
	{
	ContextGuard guard(context);
	if (block_dim <= 0)
	{
	#if defined(_DEBUG)
	fprintf(stderr, "Warp warning: Launch got block_dim %d. Setting to 256.\n", block_dim);
	#endif
	block_dim = 256;
	}
	// CUDA specs up to compute capability 9.0 says the max x-dim grid is 2**31-1, so
	// grid_dim is fine as an int for the near future
	int grid_dim = (dim + block_dim - 1)/block_dim;
	if (max_blocks <= 0) {
	max_blocks = 2147483647;
	}
	if (grid_dim < 0)
	{
	#if defined(_DEBUG)
	fprintf(stderr, "Warp warning: Overflow in grid dimensions detected for %zu total elements and 256 threads "
	"per block.\n Setting block count to %d.\n", dim, max_blocks);
	#endif
	grid_dim = max_blocks;
	}
	else
	{
	if (grid_dim > max_blocks)
	{
	grid_dim = max_blocks;
	}
	}
	begin_cuda_range(WP_TIMING_KERNEL, stream, context, get_cuda_kernel_name(kernel));
	CUresult res = cuLaunchKernel_f(
	(CUfunction)kernel,
	grid_dim, 1, 1,
	block_dim, 1, 1,
	shared_memory_bytes,
	static_cast<CUstream>(stream),
	args,
	0);
	check_cu(res);
	end_cuda_range(WP_TIMING_KERNEL, stream);
	return res;
	}

The other is the overhead from CUDA in submitting operations to the device, which is explained in https://developer.nvidia.com/blog/cuda-graphs/

In Warp, using graphs eliminate most of these overheads at the expense of some new overhead in constructing, initiating, and launching graphs, but the impact is very dependent on the application. It works the best when the two overhead sources are large compared to kernel computation time, e.g. lots of small kernels being launched on the GPU.

You can profile your Warp code in Nsight Systems to see on a timeline how graphs are improving your GPU utilization, let us know if you have questions on how to do this.

[CliffieVanR]

Hi Eric, Many thanks for the details. I presume we can just set our code running and attach Nsight Systems to it? In the cloth example we've been trying, the graph reduced our sim step time from 70ms to 25ms, so definitely an optimisation we're like to have! Best, Cliff

…

On Fri, Mar 21, 2025 at 4:53 PM Eric Shi ***@***.***> wrote: Hi Anka, We're using 1.6.0. Cuda 12.8, driver 572.16, and Python 3.12.9. I've just double-checked and am still seeing the illegal memory accesses when using the VDB integrator with handle_self_contact = True in example_cloth.py. I'm curious why the graph capture provides such a significant speed improvement. Are the kernel launches in Python synchronous? Best, Cliff … <#m_1595620600514617556_> To answer the second question, the kernel launches are not asynchronous, but they have two big sources of overhead (assume the kernel is already compiled). One is the Python overhead, which is basically all the operations that happen from an invocation of wp.launch() in the Python interpreter to the kernel launch actually being triggered using the CUDA driver API in the C code: https://github.com/NVIDIA/warp/blob/f3814e7e5459e5fd13032cf0fddb3daddd510f30/warp/native/warp.cu#L3483-L3535 The other is the overhead from CUDA in submitting operations to the device, which is explained in https://developer.nvidia.com/blog/cuda-graphs/ In Warp, using graphs eliminate most of these overheads at the expense of some new overhead in constructing, initiating, and launching graphs, but the impact is very dependent on the application. It works the best when the two overhead sources are large compared to kernel computation time, e.g. lots of small kernels being launched on the GPU. You can profile your Warp code in Nsight Systems to see on a timeline how graphs are improving your GPU utilization, let us know if you have questions on how to do this. — Reply to this email directly, view it on GitHub <#587 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPMNJMEZDHLORIKXDRBKQOD2VQ7XJAVCNFSM6AAAAABZNTB4G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBTHEZTCMJVG4> . You are receiving this because you authored the thread.Message ID: ***@***.***> [image: shi-eric]*shi-eric* left a comment (NVIDIA/warp#587) <#587 (comment)> Hi Anka, We're using 1.6.0. Cuda 12.8, driver 572.16, and Python 3.12.9. I've just double-checked and am still seeing the illegal memory accesses when using the VDB integrator with handle_self_contact = True in example_cloth.py. I'm curious why the graph capture provides such a significant speed improvement. Are the kernel launches in Python synchronous? Best, Cliff … <#m_1595620600514617556_> To answer the second question, the kernel launches are not asynchronous, but they have two big sources of overhead (assume the kernel is already compiled). One is the Python overhead, which is basically all the operations that happen from an invocation of wp.launch() in the Python interpreter to the kernel launch actually being triggered using the CUDA driver API in the C code: https://github.com/NVIDIA/warp/blob/f3814e7e5459e5fd13032cf0fddb3daddd510f30/warp/native/warp.cu#L3483-L3535 The other is the overhead from CUDA in submitting operations to the device, which is explained in https://developer.nvidia.com/blog/cuda-graphs/ In Warp, using graphs eliminate most of these overheads at the expense of some new overhead in constructing, initiating, and launching graphs, but the impact is very dependent on the application. It works the best when the two overhead sources are large compared to kernel computation time, e.g. lots of small kernels being launched on the GPU. You can profile your Warp code in Nsight Systems to see on a timeline how graphs are improving your GPU utilization, let us know if you have questions on how to do this. — Reply to this email directly, view it on GitHub <#587 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPMNJMEZDHLORIKXDRBKQOD2VQ7XJAVCNFSM6AAAAABZNTB4G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBTHEZTCMJVG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[shi-eric]

Hi Eric,

Many thanks for the details. I presume we can just set our code running and
attach Nsight Systems to it? In the cloth example we've been trying, the
graph reduced our sim step time from 70ms to 25ms, so definitely an
optimisation we're like to have!

Best,
Cliff
…

Yes, one thing to note is that by default, Nsight will profile the entire graph launch as a single workload, which can obfuscate things. You can enable the profiling of individual graph nodes by following the instructions here https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#graph-profiling (I need to add this info to our docs...)

[CliffieVanR]

Good to know - thanks!

…

On Mon, Mar 24, 2025 at 3:43 PM Eric Shi ***@***.***> wrote: Hi Eric, Many thanks for the details. I presume we can just set our code running and attach Nsight Systems to it? In the cloth example we've been trying, the graph reduced our sim step time from 70ms to 25ms, so definitely an optimisation we're like to have! Best, Cliff … <#m_4611923549144195148_> Yes, one thing to note is that by default, Nsight will profile the entire graph launch as a single workload, which can obfuscate things. You can enable the profiling of individual graph nodes by following the instructions here https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#graph-profiling (I need to add this info to our docs...) — Reply to this email directly, view it on GitHub <#587 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPMNJMACCSVCQ57KRCYUPNT2WARZPAVCNFSM6AAAAABZNTB4G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBYGU3TKNRTG4> . You are receiving this because you authored the thread.Message ID: ***@***.***> [image: shi-eric]*shi-eric* left a comment (NVIDIA/warp#587) <#587 (comment)> Hi Eric, Many thanks for the details. I presume we can just set our code running and attach Nsight Systems to it? In the cloth example we've been trying, the graph reduced our sim step time from 70ms to 25ms, so definitely an optimisation we're like to have! Best, Cliff … <#m_4611923549144195148_> Yes, one thing to note is that by default, Nsight will profile the entire graph launch as a single workload, which can obfuscate things. You can enable the profiling of individual graph nodes by following the instructions here https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html#graph-profiling (I need to add this info to our docs...) — Reply to this email directly, view it on GitHub <#587 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPMNJMACCSVCQ57KRCYUPNT2WARZPAVCNFSM6AAAAABZNTB4G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBYGU3TKNRTG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[shi-eric]

I'll assume this bug was fixed, but please re-open this issue if you see graph capture issues with the VBD integrator in Warp 1.7.

[CliffieVanR]

Hi Eric, Yes - all fixed, thanks. Apologies for not letting you know sooner. Cliff

…

On Thu, Apr 3, 2025 at 5:59 PM Eric Shi ***@***.***> wrote: I'll assume this bug was fixed, but please re-open this issue if you see graph capture issues with the VBD integrator in Warp 1.7. — Reply to this email directly, view it on GitHub <#587 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPMNJMGVL4UXWID6AXB4FGT2XVSHDAVCNFSM6AAAAABZNTB4G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONZWGQYTSMRUGE> . You are receiving this because you authored the thread.Message ID: ***@***.***> [image: shi-eric]*shi-eric* left a comment (NVIDIA/warp#587) <#587 (comment)> I'll assume this bug was fixed, but please re-open this issue if you see graph capture issues with the VBD integrator in Warp 1.7. — Reply to this email directly, view it on GitHub <#587 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPMNJMGVL4UXWID6AXB4FGT2XVSHDAVCNFSM6AAAAABZNTB4G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONZWGQYTSMRUGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>