Figure out a better number of threads to launch kernels with

[coreylowman]

Apparently always using 1024 (the max) is not the best, because it could overload threads. This is what cudarc currently does.

For example doing a small change to using 128, I see a slight boost in speedup for a small resnet model.

What are ways we could improve this?

Additionally, could some kernels be improved by using 2d or 3d block/grid dims?