batch normalization with buckets #2663
Comments
|
and that if I do not use bucketing, instead I use fixed length, mxnet will throw
If I compile with DEBUG and backtrace, the error information was given as #0 0x00007fffbf725d00 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1 |
|
FeedForward model will be deprecated soon. |
|
@piiswrong Thank you for helping me out. I have tried the
I think the problems happens in if shared_exec is None:
aux_arrays = [nd.zeros(s, context, dtype=t) for s, t in zip(aux_shapes, aux_types)]
else:
for j, arr in enumerate(shared_exec.aux_arrays):
assert aux_shapes[j] == arr.shape
assert aux_types[j] == arr.dtype
aux_arrays = shared_exec.aux_arrays[:]which should be changed to if shared_exec is None:
aux_arrays = [nd.zeros(s, context, dtype=t) for s, t in zip(aux_shapes, aux_types)]
else:
aux_arrays = [shared_exec.aux_dict[name] for name in aux_names]
for j, arr in enumerate(aux_arrays):
assert aux_shapes[j] == arr.shape
assert aux_types[j] == arr.dtype |
|
your symbols for different buckets need to have the same number of arguments and auxiliary variables. You can verify it with sym.list_auxiliary_states() |
|
Thanks for letting me know that. But what I want to do is to add batch normalization to lstm, then the auxiliary states of |
|
You can set them to the same variable |
|
Thanks, but how can I do that? It seem the |
|
moving_var=varialbe('moving_var') |
|
Thanks. I will try that |
|
I have tried using setting gamma and beta, but it seems that the auxiliary variable of moving_mean and moving_var cannot be set in this way, is there a way to set those? Thanks |
|
And I also notice that the training results of |
|
looks like you have to bind them to the same ndarray with sym.bind |
|
This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks! |
I want to implement lstm with batch normalization. When I run a sequence to sequence sample with bucketing, mxnet shows an error of
`
Traceback (most recent call last):
File "/home/odin/Documents/sent2mat_mx/model.py", line 206, in
model(config)
File "/home/odin/Documents/sent2mat_mx/model.py", line 201, in model
lr_decay],
File "/home/odin/local/mxnet/python/mxnet/model.py", line 788, in fit
sym_gen=self.sym_gen)
File "/home/odin/local/mxnet/python/mxnet/model.py", line 222, in _train_multi_device
executor_manager.load_data_batch(data_batch)
File "/home/odin/local/mxnet/python/mxnet/executor_manager.py", line 387, in load_data_batch
shared_group=self.execgrp)
File "/home/odin/local/mxnet/python/mxnet/executor_manager.py", line 224, in init
shared_data_arrays=self.shared_data_arrays[i])
File "/home/odin/local/mxnet/python/mxnet/executor_manager.py", line 170, in _bind_exec
assert aux_shape[i] == a.shape
IndexError: list index out of range
`
It seems to me that when generating executor_manager (model.py line 184), it tends to use the default bucket length, which is the longest bucket. However, when feeding data, (model.py line 222), the bucket length is not essentially the same as the longest bucket. The auxiliary states are then not the same as previous ones.
Is it possible to solve this problem? Any help would be appreciated.
The text was updated successfully, but these errors were encountered: