Add f16 dtype support for tensors

[coreylowman]

This can use the half crate.

Cuda has many arithmetic functions available for half as seen here

hgemm support depends on coreylowman/cudarc#65

[coreylowman]

See #421 for draft of what adding another dtype looks like

[nkoppel]

I've started work on this, but I've encountered a major issue. half doesn't have any integration with rand_distr or vice-versa, so f16 doesn't implement SampleUniform, which is currently required to implement Dtype. To resolve this, either half needs to add rand and rand_distr integration, or we need to add an alias trait for SampleUniform to use internally. The former is definitely preferable, but this may delay the addition of f16 support.

[ViliamVadocz]

@nkoppel How far along are you in the implementation? I am interested in working on the cuda side of things.

[ViliamVadocz]

I've started work on this, but I've encountered a major issue. half doesn't have any integration with rand_distr or vice-versa, so f16 doesn't implement SampleUniform, which is currently required to implement Dtype. To resolve this, either half needs to add rand and rand_distr integration, or we need to add an alias trait for SampleUniform to use internally. The former is definitely preferable, but this may delay the addition of f16 support.

What about making a wrapper for f16 and implementing SampleUniform (and other needed traits)?

[coreylowman]

I was just looking into this a bit for the half crate, but even when implementing SampleUniform you run into foreign traits. So I think even we did a wrapper type we would need some changes from rand/rand_distr I think

[coreylowman]

Okay just kidding I was missing something! I have a PR opened in half, so hopefully we can depend on that

[opfromthestart]

Is anyone still working on this? I'd work on it but I wouldn't want to redo things.

[coreylowman]

I'm not - still waiting on half crate to merge my PR. You can probably start working on this if you use my fork of half with the update