Implement cuda kernels for unary ops

[coreylowman]

These can mirror abs & exp which are already implemented:

• abs
• clamp
• cos
• exp
• ln
• nans_to
• negate
• powi
• powf
• relu
• sigmoid
• sin
• square
• tanh

[ViliamVadocz]

Currently the test for abs fails on my machine. There is a small issue in the abs_backward implementation.

running 1 test
test tensor_ops::abs::tests::test_abs ... FAILED

failures:

---- tensor_ops::abs::tests::test_abs stdout ----
thread 'tensor_ops::abs::tests::test_abs' panicked at 'assertion failed: `(left == right)`
  left: `[0.2, 0.2, 0.0, -0.2, -0.2]`,
 right: `[-0.2, -0.2, 0.0, 0.2, 0.2]`', src\tensor_ops\abs\mod.rs:53:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I fixed it by changing line 27 in abs.cu. signbit(x) is true when the x is negative.

-    float dx = inp[i] == 0.0 ? 0.0 : (signbit(inp[i]) ? 1.0 : -1.0);
+    float dx = inp[i] == 0.0 ? 0.0 : (signbit(inp[i]) ? -1.0 : 1.0);

You could also write it a bit less confusingly like this:

float dx = inp[i] == 0.0 ? 0.0 : copysignf(1.0, inp[i]);

[ViliamVadocz]

If I understand correctly, the current kernels should be written for single-precision floating point numbers (float or f32). If so, it might be appropriate to use function specifically for floats such as fabsf or expf to avoid any accidental conversions to doubles and back.

Here's a list of the available functions: Single Precision Mathematical Functions

---

What are your thoughts on using intrinsics?

[coreylowman]

Oooh good catch about the single precision stuff, I had no idea, so glad ya'll know 😀 What is the difference between intrinsics vs the non intrinsics?

[coreylowman]

Thanks for the PR @nkoppel

[nkoppel]

Intrinsics are faster than their non-intrinsic counterparts, at the cost of lower precision. Here is documentation for the standard mathematical functions, and here are their intrinsic counterparts. Note that ulp error stands for units in last place.

It's probably best to avoid using intrinsics for now, because we can't guarantee that intrinsics will be worth it for every use case, and because these functions will usually only take a small fraction of the runtime anyways.

[coreylowman]

Makes sense, it also seems like the -use_fast_math flag forces them to compile to intrinsics? So we should use that flag instead of hardcoding to intrinsics in the future if we want to move to them

[nkoppel]

It may be best to expose -use_fast_math to users through a cargo feature so that they can keep accuracy and avoid potential weird behavior from using intrinsics when needed.

[ViliamVadocz]

One big downside of intrinsics other than reduced accuracy is they might handle subnormals (NaNs, infs, etc.) differently. I think it definitely should not be enabled by default but instead behind a feature flag.