remember initial state #3

vivanov879 · 2016-12-23T09:04:50Z

No description provided.

vivanov879 · 2016-12-23T09:05:50Z

trying out the change --
had an error right in the end oftraining

vivanov879 · 2016-12-23T09:06:01Z

/usr/bin/python3.5 /home/vivanov/PycharmProjects/PLSTM/simplePhasedLSTM.py
Compiling RNN...
DONE!
Compiling cost functions...
DONE!
/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients_impl.py:91: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Calculating gradients...
DONE!
Initializing variables...
DONE!
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1021, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1003, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 469, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: ConcatOp : Dimensions of inputs should match: shape[0] = [320,1] vs. shape[1] = [32,100]
[[Node: 0/RNN/while/PhasedLSTMCell/concat = Concat[N=2, T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](0/RNN/while/PhasedLSTMCell/concat/concat_dim, 0/RNN/while/PhasedLSTMCell/Slice_1, 0/RNN/while/Identity_3)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/vivanov/PycharmProjects/PLSTM/simplePhasedLSTM.py", line 276, in
if name == "main":
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "/home/vivanov/PycharmProjects/PLSTM/simplePhasedLSTM.py", line 265, in main
y: test_ys,
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 766, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 964, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1014, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1034, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: ConcatOp : Dimensions of inputs should match: shape[0] = [320,1] vs. shape[1] = [32,100]
[[Node: 0/RNN/while/PhasedLSTMCell/concat = Concat[N=2, T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](0/RNN/while/PhasedLSTMCell/concat/concat_dim, 0/RNN/while/PhasedLSTMCell/Slice_1, 0/RNN/while/Identity_3)]]

Caused by op '0/RNN/while/PhasedLSTMCell/concat', defined at:
File "/home/vivanov/PycharmProjects/PLSTM/simplePhasedLSTM.py", line 276, in
if name == "main":
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "/home/vivanov/PycharmProjects/PLSTM/simplePhasedLSTM.py", line 190, in main
print ("Compiling RNN...",)
File "/home/vivanov/PycharmProjects/PLSTM/simplePhasedLSTM.py", line 150, in RNN
initial_states = [tf.nn.rnn_cell.LSTMStateTuple(tf.zeros([FLAGS.batch_size, FLAGS.n_hidden], tf.float32), tf.zeros([FLAGS.batch_size, FLAGS.n_hidden], tf.float32)) for _ in range(FLAGS.n_layers)]
File "/home/vivanov/PycharmProjects/PLSTM/PhasedLSTMCell.py", line 360, in multiPLSTM
outputs, initial_states[k] = tf.nn.dynamic_rnn(cell, newX, dtype=tf.float32, sequence_length=lens, initial_state=initial_states[k])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py", line 845, in dynamic_rnn
dtype=dtype)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py", line 1012, in _dynamic_rnn_loop
swap_memory=swap_memory)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2636, in while_loop
result = context.BuildLoop(cond, body, loop_vars, shape_invariants)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2469, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2419, in _BuildLoop
body_result = body(*packed_vars_for_body)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py", line 995, in _time_step
skip_conditionals=True)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py", line 403, in _rnn_step
new_output, new_state = call_cell()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py", line 983, in
call_cell = lambda: cell(input_t, state)
File "/home/vivanov/PycharmProjects/PLSTM/PhasedLSTMCell.py", line 292, in call
cell_inputs = array_ops.concat(1, [filtered_inputs, m_prev])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/array_ops.py", line 1005, in concat
name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 438, in _concat
values=values, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1128, in init
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): ConcatOp : Dimensions of inputs should match: shape[0] = [320,1] vs. shape[1] = [32,100]
[[Node: 0/RNN/while/PhasedLSTMCell/concat = Concat[N=2, T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](0/RNN/while/PhasedLSTMCell/concat/concat_dim, 0/RNN/while/PhasedLSTMCell/Slice_1, 0/RNN/while/Identity_3)]]

vivanov879 · 2016-12-23T09:07:14Z

now it remembers states between runs

indiejoseph · 2016-12-23T09:09:59Z

simplePhasedLSTM.py

-
-    outputs = multiPLSTM(_X, lens, FLAGS.n_layers, FLAGS.n_hidden, n_input)
+    initial_states = [tf.nn.rnn_cell.LSTMStateTuple(tf.zeros([FLAGS.batch_size, FLAGS.n_hidden], tf.float32), tf.zeros([FLAGS.batch_size, FLAGS.n_hidden], tf.float32)) for _ in range(FLAGS.n_layers)]
+    outputs, initial_states = multiPLSTM(_X, lens, FLAGS.n_layers, FLAGS.n_hidden, n_input, initial_states)


anyway to make it work with tf.nn.rnn_cell.MultiRNNCell? it's more intitutive and can use cell.zero_state to generate the initial_states

i think it's possible but wont make initial_state any simpler
same number of words and same sense

vivanov879 · 2016-12-23T09:14:54Z

i tracked the bug in my code -- it's wrong shapes during test -- will fix now

vivanov879 · 2016-12-23T09:16:56Z

different batch size during test -- fixed that -- now testing

vivanov879 · 2016-12-23T09:22:58Z

I want it to remember states like an ordinary multilayer LSTM would. I keep each layer's state in a list. Now PLSTM uses previous end state as initial states of the current run

vivanov879 · 2016-12-23T09:26:37Z

/usr/bin/python3.5 /home/vivanov/PycharmProjects/PLSTM/simplePhasedLSTM.py
Compiling RNN...
DONE!
Compiling cost functions...
DONE!
/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients_impl.py:91: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Calculating gradients...
DONE!
Initializing variables...
DONE!
+-----------+----------+------------+
| Epoch=0 | Cost | Accuracy |
+===========+==========+============+
| Train | 0.354964 | 0.832031 |
+-----------+----------+------------+
| Test | 0.137772 | 0.96875 |
+-----------+----------+------------+

vivanov879 · 2016-12-23T09:26:46Z

here's a full run -- it works

vivanov879 · 2016-12-23T09:27:59Z

i wonder how to keep track of initial_states in a tensorflow's summary -- can you please help me with that? That would prove it works correctly

vivanov879 · 2016-12-23T09:36:54Z

+-----------+----------+------------+
| Epoch=0 | Cost | Accuracy |
+===========+==========+============+
| Train | 0.354964 | 0.832031 |
+-----------+----------+------------+
| Test | 0.137772 | 0.96875 |
+-----------+----------+------------+
+-----------+-----------+------------+
| Epoch=1 | Cost | Accuracy |
+===========+===========+============+
| Train | 0.137392 | 0.956641 |
+-----------+-----------+------------+
| Test | 0.0635801 | 0.96875 |
+-----------+-----------+------------+

Enny1991 · 2016-12-23T10:03:24Z

Hey! Very nice job,
Can we make the initial state argument optional? I'd like to keep it such that people can also not specify the initial state.

vivanov879 · 2016-12-23T10:04:40Z

Sure. I'll do it in an hour. Just a flag that will reset it on every iteration 23.12.2016, 131:03 ПП, "Enea Ceolini" <notifications@github.com>:Hey! Very nice job, Can we make the initial state argument optional? I'd like to keep it such that people can also not specify the initial state. —You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or mute the thread.

Enny1991 · 2016-12-23T10:05:54Z

Yeah actually I can do it now! I'll merge, update with this flag and add the summary for the initial state.

vivanov879 · 2016-12-23T10:27:29Z

that's great -- looking forward to try it out

vivanov10 added 2 commits December 23, 2016 11:14

mo

a37f12b

mo

6bbed15

vivanov879 mentioned this pull request Dec 23, 2016

how to remember states with multi layer PLSTM #2

Closed

indiejoseph reviewed Dec 23, 2016

View reviewed changes

mo

373e607

mo

fc3cb34

Enny1991 merged commit 7540014 into Enny1991:master Dec 23, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remember initial state #3

remember initial state #3

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

indiejoseph Dec 23, 2016

vivanov879 Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016 •

edited

Loading

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

Enny1991 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016 via email

Enny1991 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

remember initial state #3

remember initial state #3

Conversation

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

indiejoseph Dec 23, 2016

Choose a reason for hiding this comment

vivanov879 Dec 23, 2016

Choose a reason for hiding this comment

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016 • edited Loading

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

Enny1991 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016 via email

Enny1991 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016

vivanov879 commented Dec 23, 2016 •

edited

Loading