canonical estimator results in shape error #161

cyrusmvahid · 2019-07-01T11:34:32Z

Description

I am trying to run the twit example on other estimators. In this case canonical rnn.
Here is the code:

canonical_estimator = CanonicalRNNEstimator(freq="5min", context_length=10, prediction_length=12, trainer=Trainer(epochs=EPOCHS))
canonical_predictor = canonical_estimator.train(training_data=training_data)

I receive shape error.

MXNetError: Error in operator canonicaltrainingnetwork5__minus0: [11:35:55] /work/mxnet/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node canonicaltrainingnetwork5__minus0 at 1-th input: expected [32,10,1], got [32,10]

my context length is 10. It seems as the ndarray has been squeezed somewhere.

To Reproduce

canonical_estimator = CanonicalRNNEstimator(freq="5min", context_length=10, prediction_length=12, trainer=Trainer(epochs=EPOCHS))
canonical_predictor = canonical_estimator.train(training_data=training_data)

Error Message

I am trying to run the twit example on other estimators. In this case canonical rnn.
Here is the code:

canonical_estimator = CanonicalRNNEstimator(freq="5min", context_length=10, prediction_length=12, trainer=Trainer(epochs=EPOCHS))
canonical_predictor = canonical_estimator.train(training_data=training_data)

I receive shape error.

MXNetError: Error in operator canonicaltrainingnetwork5__minus0: [11:35:55] /work/mxnet/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node canonicaltrainingnetwork5__minus0 at 1-th input: expected [32,10,1], got [32,10]

my context length is 10. It seems as the ndarray has been squeezed somewhere.

Environment

Operating system:
Python version:
GluonTS version:

(Add as much information about your environment as possible, e.g. dependencies versions.)

The text was updated successfully, but these errors were encountered:

cyrusmvahid · 2019-07-01T11:38:06Z

I am trying to run the twit example on other estimators. In this case canonical rnn.
Here is the code:

canonical_estimator = CanonicalRNNEstimator(freq="5min", context_length=10, prediction_length=12, trainer=Trainer(epochs=EPOCHS))
canonical_predictor = canonical_estimator.train(training_data=training_data)

I receive shape error.

MXNetError: Error in operator canonicaltrainingnetwork5__minus0: [11:35:55] /work/mxnet/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node canonicaltrainingnetwork5__minus0 at 1-th input: expected [32,10,1], got [32,10]

my context length is 10. It seems as the ndarray has been squeezed somewhere.

Full error:

infer_shape error. Arguments:
  data0: (32, 1)
  data1: (32, 10, 5)
  data2: (32, 10)
---------------------------------------------------------------------------
DeferredInitializationError               Traceback (most recent call last)
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _call_cached_op(self, *args)
    802             cargs = [args[i] if is_arg else i.data()
--> 803                      for is_arg, i in self._cached_op_args]
    804         except DeferredInitializationError:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in <listcomp>(.0)
    802             cargs = [args[i] if is_arg else i.data()
--> 803                      for is_arg, i in self._cached_op_args]
    804         except DeferredInitializationError:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/parameter.py in data(self, ctx)
    493                                "instead." % (self.name, str(ctx), self._stype))
--> 494         return self._check_and_get(self._data, ctx)
    495 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/parameter.py in _check_and_get(self, arr_list, ctx)
    207                 "You can also avoid deferred initialization by specifying in_units, " \
--> 208                 "num_features, etc., for network layers."%(self.name))
    209         raise RuntimeError(

DeferredInitializationError: Parameter 'rnn8_lstm0_l0_i2h_weight' has not been initialized yet because initialization was deferred. Actual initialization happens during the first forward pass. Please pass one batch of data through the network before accessing Parameters. You can also avoid deferred initialization by specifying in_units, num_features, etc., for network layers.

During handling of the above exception, another exception occurred:

MXNetError                                Traceback (most recent call last)
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _deferred_infer_shape(self, *args)
    788         try:
--> 789             self.infer_shape(*args)
    790         except Exception as e:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in infer_shape(self, *args)
    861         """Infers shape of Parameters from inputs."""
--> 862         self._infer_attrs('infer_shape', 'shape', *args)
    863 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _infer_attrs(self, infer_fn, attr, *args)
    850             arg_attrs, _, aux_attrs = getattr(out, infer_fn)(
--> 851                 **{i.name: getattr(j, attr) for i, j in zip(inputs, args)})
    852             if arg_attrs is None:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/symbol/symbol.py in infer_shape(self, *args, **kwargs)
    995         try:
--> 996             res = self._infer_shape_impl(False, *args, **kwargs)
    997             if res[1] is None:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/symbol/symbol.py in _infer_shape_impl(self, partial, *args, **kwargs)
   1125             ctypes.byref(aux_shape_data),
-> 1126             ctypes.byref(complete)))
   1127         if complete.value != 0:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/base.py in check_call(ret)
    251     if ret != 0:
--> 252         raise MXNetError(py_str(_LIB.MXGetLastError()))
    253 

MXNetError: Error in operator canonicaltrainingnetwork5__minus0: [11:35:55] /work/mxnet/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node canonicaltrainingnetwork5__minus0 at 1-th input: expected [32,10,1], got [32,10]

Stack trace returned 10 entries:
[bt] (0) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3f935a) [0x7f928cf2a35a]
[bt] (1) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3f9981) [0x7f928cf2a981]
[bt] (2) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x73e45d) [0x7f928d26f45d]
[bt] (3) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x7a25b6) [0x7f928d2d35b6]
[bt] (4) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x9c33d8) [0x7f928d4f43d8]
[bt] (5) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e0e10a) [0x7f928f93f10a]
[bt] (6) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e10a84) [0x7f928f941a84]
[bt] (7) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShape+0x15ba) [0x7f928f8a79aa]
[bt] (8) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f92fc62dec0]
[bt] (9) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f92fc62d87d]



During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-143-b6d286edd137> in <module>()
      1 canonical_estimator = CanonicalRNNEstimator(freq="5min", context_length=10, prediction_length=12, trainer=Trainer(epochs=EPOCHS))
----> 2 canonical_predictor = canonical_estimator.train(training_data=training_data)
      3 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/gluonts/model/estimator.py in train(self, training_data)
    187     def train(self, training_data: Dataset) -> Predictor:
    188 
--> 189         training_transformation, trained_net = self.train_model(training_data)
    190 
    191         # ensure that the prediction network is created within the same MXNet

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/gluonts/model/estimator.py in train_model(self, training_data)
    180             net=trained_net,
    181             input_names=get_hybrid_forward_input_names(trained_net),
--> 182             train_iter=training_data_loader,
    183         )
    184 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/gluonts/trainer/_base.py in __call__(self, net, input_names, train_iter)
    256 
    257                             with mx.autograd.record():
--> 258                                 output = net(*inputs)
    259 
    260                                 # network can returns several outputs, the first being always the loss

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in __call__(self, *args)
    538             hook(self, args)
    539 
--> 540         out = self.forward(*args)
    541 
    542         for hook in self._forward_hooks.values():

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in forward(self, x, *args)
    905             with x.context as ctx:
    906                 if self._active:
--> 907                     return self._call_cached_op(x, *args)
    908 
    909                 try:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _call_cached_op(self, *args)
    803                      for is_arg, i in self._cached_op_args]
    804         except DeferredInitializationError:
--> 805             self._deferred_infer_shape(*args)
    806             cargs = []
    807             for is_arg, i in self._cached_op_args:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _deferred_infer_shape(self, *args)
    791             error_msg = "Deferred initialization failed because shape"\
    792                         " cannot be inferred. {}".format(e)
--> 793             raise ValueError(error_msg)
    794 
    795     def _call_cached_op(self, *args):

ValueError: Deferred initialization failed because shape cannot be inferred. Error in operator canonicaltrainingnetwork5__minus0: [11:35:55] /work/mxnet/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node canonicaltrainingnetwork5__minus0 at 1-th input: expected [32,10,1], got [32,10]

Stack trace returned 10 entries:
[bt] (0) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3f935a) [0x7f928cf2a35a]
[bt] (1) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3f9981) [0x7f928cf2a981]
[bt] (2) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x73e45d) [0x7f928d26f45d]
[bt] (3) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x7a25b6) [0x7f928d2d35b6]
[bt] (4) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x9c33d8) [0x7f928d4f43d8]
[bt] (5) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e0e10a) [0x7f928f93f10a]
[bt] (6) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e10a84) [0x7f928f941a84]
[bt] (7) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShape+0x15ba) [0x7f928f8a79aa]
[bt] (8) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f92fc62dec0]
[bt] (9) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f92fc62d87d]

lostella · 2019-07-01T11:40:42Z

@cyrusmvahid thanks for submitting this -- could you enclose snippets and error traces in triple ticks (`) to have it formatted nicely?

like this

cyrusmvahid · 2019-07-01T11:44:10Z

done
sorry

lostella · 2019-07-01T11:47:27Z

done
sorry

Thank you, but remember to put triple ticks (```) one line before and one line after snippets and error traces. Sorry for being so pedantic, but nicely formatted issues are more likely to be looked into.

cyrusmvahid · 2019-07-01T11:48:16Z

sure

…

On Mon, Jul 1, 2019 at 1:47 PM Lorenzo Stella ***@***.***> wrote: done sorry Thank you, but remember to put *triple* ticks (```) one line before and one line after snippets and error traces. Sorry for being so pedantic, but nicely formatted issues are more likely to be looked into. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#161>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABLOVGKEB7UEZ5UDHGYVEXDP5HVFDANCNFSM4H4ROYDA> .

lostella · 2019-08-23T18:24:19Z

Fixed in #254, closing

cyrusmvahid added the bug Something isn't working label Jul 1, 2019

lostella mentioned this issue Aug 23, 2019

Fix shapes in CanonicalNetworkBase #254

Merged

lostella closed this as completed Aug 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

canonical estimator results in shape error #161

canonical estimator results in shape error #161

cyrusmvahid commented Jul 1, 2019 •

edited by lostella

cyrusmvahid commented Jul 1, 2019 •

edited by lostella

lostella commented Jul 1, 2019

cyrusmvahid commented Jul 1, 2019

lostella commented Jul 1, 2019

cyrusmvahid commented Jul 1, 2019 via email

lostella commented Aug 23, 2019

canonical estimator results in shape error #161

canonical estimator results in shape error #161

Comments

cyrusmvahid commented Jul 1, 2019 • edited by lostella

Description

To Reproduce

Error Message

Environment

cyrusmvahid commented Jul 1, 2019 • edited by lostella

lostella commented Jul 1, 2019

cyrusmvahid commented Jul 1, 2019

lostella commented Jul 1, 2019

cyrusmvahid commented Jul 1, 2019 via email

lostella commented Aug 23, 2019

cyrusmvahid commented Jul 1, 2019 •

edited by lostella

cyrusmvahid commented Jul 1, 2019 •

edited by lostella