[cuda] Use cuMemsetD32 to fill scalar ndarray

[qiao-bo]

Replace the Ndarray fill method from the previous kernel-based approach with a memset-based implementation.

Performance improvement:
test_script.py

import taichi as ti
import time

ti.init(arch=ti.cuda)

print("test fill")
N = 2048*2048

a = ti.ndarray(ti.f32, N)
a.fill(2.5)

iterations = 1000
t_start = time.perf_counter()
for i in range(iterations):
    a.fill(float(i))
    #assert a[i] == float(i)
t_used = time.perf_counter() - t_start 
print('total time:', "{:.3f}".format(t_used*1000), "ms")

tested on RTX 3080.

iterations	Master	Memset
10	0.280 s	0.060 ms
100	4.778 s	0.530 ms
1000	46.425 s	0.005 s

Status:
This PR addresses only scalar ndarray on CUDA. Support of Matrix/Vector.ndarray and CPU will be followed in later PRs.

[netlify]

✔️ Deploy Preview for jovial-fermat-aa59dc canceled.

🔨 Explore the source changes: 9e90eb1

🔍 Inspect the deploy log: https://app.netlify.com/sites/jovial-fermat-aa59dc/deploys/61ce816fa359c00008619276

[qiao-bo]

/format

[bobcao3]

Maybe we can utilize cuStreamBeginCapture to make the command list actually deferred

[qiao-bo]

Maybe we can utilize cuStreamBeginCapture to make the command list actually deferred

@bobcao3 thanks for the information. At the moment, we do not want to defer the fill execution. In this case, we don't need to over utilize the CommandList utility. Maybe a simpler solution is to have the fill method in CudaDevice itself. WDYT?

[bobcao3]

Maybe we can utilize cuStreamBeginCapture to make the command list actually deferred

@bobcao3 thanks for the information. At the moment, we do not want to defer the fill execution. In this case, we don't need to over utilize the CommandList utility. Maybe a simpler solution is to have the fill method in CudaDevice itself. WDYT?

@qiao-bo I don't think that's ideal. Not all APIs support immediate mode resource content filling. In addition, in hardware these commands are executed by the DMA / Transfer engine which uses a queue. Since all of this is on the C++ side, it is fine to have it under CommandList, it's reasonably fast. In addition, after we move everything into CommandList, we should expect a drop in overall host-side overhead as CUDA has a fully completed command graph to work with & uses less implicit sync.

[qiao-bo]

Maybe we can utilize cuStreamBeginCapture to make the command list actually deferred

@bobcao3 thanks for the information. At the moment, we do not want to defer the fill execution. In this case, we don't need to over utilize the CommandList utility. Maybe a simpler solution is to have the fill method in CudaDevice itself. WDYT?

@qiao-bo I don't think that's ideal. Not all APIs support immediate mode resource content filling. In addition, in hardware these commands are executed by the DMA / Transfer engine which uses a queue. Since all of this is on the C++ side, it is fine to have it under CommandList, it's reasonably fast. In addition, after we move everything into CommandList, we should expect a drop in overall host-side overhead as CUDA has a fully completed command graph to work with & uses less implicit sync.

I agree with the logic. But seems like we have not implemented any device utilities to support immediate content filling for llvm backend? i see a potential major change here, does it still fit into this PR?

[qiao-bo]

/format

[qiao-bo]

/format

[bobcao3]

LGTM

[k-ye]

@qiao-bo nit: Do we need mutable reference DeviceAllocation&?

If not, let's make them const. Otherwise let's follow Google's style and use pointer

[qiao-bo]

@k-ye , made const in #3921. Thanks