float16 precision

[trungnt13]

My main purpose is only store data in float16, then convert to float32 for doing computation
However, the convertion largely change the precision of the number when convert float16 back to float32.
Since the byte conversion process seems to ignore the precision, is there any better way to store float in float16 dtype ?

from __future__ import print_function, division, absolute_import
import numpy as np

np.random.seed(1208)
x = np.random.rand(1)
print(x, x.dtype)
x = x.astype('float32')
print(x, x.dtype)
print('Convert:')
x1 = np.cast['float16'](x)
print(x1, x1.dtype)
x1 = x1.astype('float32')
print(x1, x1.dtype)
print('Round and Convert:')
print(np.around(x, 6), np.around(x, 6).dtype)
x2 = np.cast['float16'](np.around(x, 6))
print(x2, x2.dtype)
x2 = x2.astype('float32')
print(x2, x2.dtype)

Output:

[ 0.0429911] float64
[ 0.0429911] float32
Convert:
[ 0.04299927] float16
[ 0.04299927] float32
Round and Convert:
[ 0.042991] float32
[ 0.04299927] float16
[ 0.04299927] float32

float16 always drop more precision than rounding the number, given the fact that it can preserve precision upto 4 number in the fraction

from __future__ import print_function, division, absolute_import
import numpy as np

np.random.seed(1208)
X = np.random.rand(12, 12)
X16 = X.astype('float16').astype('float32')
XR = np.around(X, decimals=4)

print('Float16:', np.sum(np.abs(X - X16)))
print('Round:', np.sum(np.abs(X - XR)))

[ahaldane]

It looks like numpy is doing the right thing here.

When you convert from 32bit float to 16bit float and back, some of the bits are lost, but note that this does not mean the decimal representation will be shorter. When you print out a floating point number, the computer only prints the closest decimal representation of the binary value.

It is easier to see what is going on in binary:

# copied from stackoverflow, prints float in binary
import struct
def binary(num):
    return ''.join(bin(ord(c)).replace('0b', '').rjust(8, '0') 
                   for c in struct.pack('!f', num))

x = np.array([0.0429911], dtype='f4')
x1 = x.astype('f2')
x2 = x1.astype('f4')

print binary(x), x
print binary(x1), x1
print binary(x2), x2

output:

00111101001100000001011101110000 [ 0.0429911]
00111101001100000010000000000000 [ 0.04299927]
00111101001100000010000000000000 [ 0.04299927]

as you can see, in binary the final result is truncated (set to 0) yet the decimal representation is longer.

I'm going to go ahead and close.