why need to initialize read vector and memory?

[dragen1860]

Dear author:
I found you initialize read vector and memory as :

		self.register_buffer('mem_bias', torch.Tensor(N, M))

		# Initialize memory bias
		stdev = 1 / (np.sqrt(N + M))
		nn.init.uniform_(self.mem_bias, -stdev, stdev)

and

init_r_bias = torch.randn(1, M).to('cuda') * 0.01
# the initial value of read vector is not optimized.
self.register_buffer("read{}_bias".format(self.num_read_heads), init_r_bias)

I wonder whether the initialization scheme will make a big difference,
or I can just all initialized to torch.zeros()??

[loudinthecloud]

Since the memory is content addressable, it must be initialized to some bias value. This ensures that all/some cells can be addressed specifically, if needed...

The NTM paper mentions it in section 4:

For NTM the previous state of the controller, the value of the previous read vectors, and the contents of the memory were all reset to bias values.