is Float16 a real computational type or not?

[StefanKarpinski]

At this point, Float16 is in a kind of weird limbo where it is sort of a storage type but we keep haphazardly adding more and more functionality to it. I think we really need to figure out where to draw the line here. Either we severely limit the scope of Float16's functionality and stop using it in tests for anything but converting to and from other numeric types, or we just go full throttle and make it a full computation type with as much functionality as Float32 or Float64 – albeit slower, since many operations will not be implemented as native machine ops.

[jiahao]

I've wondered about this as well. Until extremely recently, we've had some tests for linear algebra over Float16s, which have turned out to be very stressful on our test criteria.

[andreasnoack]

I saw your last comment about this and I am preparing to remove the Float16 linalg tests. They are also overfloating all the time.

[andreasnoack]

@jiahao Almost the same comment at the same time. In #5902 I have removed Float16 from the tests.

[JeffBezanson]

I think it should have the same status as other numeric types that lack
extensive hardware and library support, like rational. Probably all we need
to do is generalize the definitions that make libm functions applicable to
ints and rationals.
On Feb 24, 2014 5:48 PM, "Andreas Noack Jensen" notifications@github.com
wrote:

I saw your last comment about this and I am preparing to remove the
Float16 linalg tests. They are also overfloating all the time.

Reply to this email directly or view it on GitHubhttps://github.com//issues/5942#issuecomment-35949938
.

[JeffBezanson]

Oh and fixed point would be another type in this category, though it is
more plausible to compute with.
On Feb 24, 2014 5:55 PM, "Jeff Bezanson" jeff.bezanson@gmail.com wrote:

I think it should have the same status as other numeric types that lack
extensive hardware and library support, like rational. Probably all we need
to do is generalize the definitions that make libm functions applicable to
ints and rationals.
On Feb 24, 2014 5:48 PM, "Andreas Noack Jensen" notifications@github.com
wrote:

I saw your last comment about this and I am preparing to remove the
Float16 linalg tests. They are also overfloating all the time.

Reply to this email directly or view it on GitHubhttps://github.com//issues/5942#issuecomment-35949938
.

[pao]

fixed point would be another type in this category, though it is more plausible to compute with

It's very plausible in some domains, but these are also places you are unlikely to get the Julia runtime to work.

[JeffBezanson]

Is that a challenge? :-)
On Feb 24, 2014 6:37 PM, "pao" notifications@github.com wrote:

fixed point would be another type in this category, though it is more
plausible to compute with

It's very plausible in some domains, but these are also places you are
unlikely to get the Julia runtime to work.

Reply to this email directly or view it on GitHubhttps://github.com//issues/5942#issuecomment-35954405
.

[JeffBezanson]

Currently the fallbacks for transcendental functions look like

        ($f)(x::Real) = ($f)(float(x))

How about we use instead

        ($f)(x::Integer) = ($f)(float(x))
        ($f)(x::Real) = oftype(x, ($f)(float64(x)))

Then we could remove most of the definitions in float16.jl, and it wouldn't seem haphazard anymore.

[jiahao]

This seems to work.

• It looks like you can also remove the ($f)(x::FloatingPoint) fallback methods in math.jl, which just complain that the method is not defined.
• The complex definitions seem to be redundant with the generic implementations in complex.jl, modulo the fact that some type instability results from the current behavior that float16(1) * im is not a Complex32, but rather a Complex64.

[jiahao]

julia> typeof(float16(1.0)*true)
Float32

julia> promote_type(Bool, Float16)
Float32

[jiahao]

MathConsts also need special handling.

With the proposed fallbacks:

julia> sqrt(pi)
ERROR: no method convert(Type{MathConst{:π}}, Float64)
 in sqrt at math.jl:274

[JeffBezanson]

I guess there is a case to be made that any operation on Float16 should return a Float32, since that way there's no roundoff and subsequent operations will actually be faster. That's probably what it means to be a "storage type". There is a strong similarity to how we do e.g. Int8 + Int8.

[jiahao]

Fair enough, but having to explicitly recast the complex elementary functions strikes me as really quite inelegant, given the generic implementations in complex.jl:

julia> for f in (sin, cos, tan, asin, acos, atan, sinh, cosh, tanh, asinh, acosh, atanh, exp, log, sqrt)
          x=float16(rand()+rand()*im); @show f, typeof(f(x))
       end
(f,typeof(f(x))) => (sin,Complex{Float16})
(f,typeof(f(x))) => (cos,Complex{Float16})
(f,typeof(f(x))) => (tan,Complex{Float32})
(f,typeof(f(x))) => (asin,Complex{Float32})
(f,typeof(f(x))) => (acos,Complex{Float32})
(f,typeof(f(x))) => (atan,Complex{Float32})
(f,typeof(f(x))) => (sinh,Complex{Float16})
(f,typeof(f(x))) => (cosh,Complex{Float16})
(f,typeof(f(x))) => (tanh,Complex{Float32})
(f,typeof(f(x))) => (asinh,Complex{Float32})
(f,typeof(f(x))) => (acosh,Complex{Float32})
(f,typeof(f(x))) => (atanh,Complex{Float32})
(f,typeof(f(x))) => (exp,Complex{Float16})
(f,typeof(f(x))) => (log,Complex{Float32})
(f,typeof(f(x))) => (sqrt,Complex{Float32})

[StefanKarpinski]

It's resolved at this point that Float16 is a real computational type: many people want to use if for a variety of purposes and find it frustrating that it doesn't behave the same way as Float32 and Float64; moreover, GPUs have actual support for using Float16 as a computational type.

[StefanKarpinski]

What if anything is left to do here?

[tkelman]

ping @vchuravy for your thoughts

[vchuravy]

There are two main things currently missing.

1. Lowering of Float16 to half instead of i16 (I think we could do this now)
2. RTLIB: A runtime-library for Julia Juleps#11 or at least Basic support for builtins from compiler-rt #18734

[yuyichao]

Both of those affect performance but not functions right?

[vchuravy]

Yes, but if we actually want to make use of Float16 we need both of them. Right now we are eagerly using Float32 as a fallback, preventing any benefit from actually using Float16. (except maybe reduce memory usage and bandwidth)

[StefanKarpinski]

Strict performance improvements can happen any time and be backported, so that makes this not a release-blocking issue – it's an issue that anyone who wants to can tackle whenever.

[tkelman]

The build system changes and library restructuring to integrate compiler-rt and the rtlib julep are probably too big to backport, but they're making progress.

[JeffBezanson]

Closing since there seem to be no more semantic changes needed.