model/get_states.lua:78: bad argument #1 to 'copy'

[tlhumphrey2]

Getting the above error when I attempt to execute get_states.lua. This is on my own data. Here is what I do before executing get_states.lua
First, I download the tiny-Shakespeare dataset in jcjohson's github repo, torch-rnn; and split it into 2 datasets: train_tiny-shakespeare.txt and validation_tiny-shakespeare.txt.

Then, I run preprocess like so:
python model/preprocess.py data/tinyshakespeare/train_tiny-shakespeare.txt
data/tinyshakespeare/validation_tiny-shakespeare.txt 50 64
data/tinyshakespeare/convert/tiny-shakespeare

Then, I run main.lua to train on this data, like so:
th model/main.lua -rnn_size 128 -word_vec_size 64-num_layers 2
-epochs 50 -data_file data/tinyshakespeare/convert/tiny-shakespeare.hdf5
-val_data_file data/tinyshakespeare/convert/tiny-shakespeareval.hdf5
-gpuid 0 -savefile cv/tinyshakespeare
| tee train-tinyshakespeare.log

Then, I run get_states.lua like so:
th model/get_states.lua
-data_file data/tinyshakespeare/convert/tiny-shakespeare.hdf5
-checkpoint_file cv/tinyshakespeare_epoch20.00_420.16.t7
-output_file data/reads/tinyshakespeare_states.h5

This is where I get the error.

[tlhumphrey2]

Note. In my above post, all my back-slashes were left out.

[sebastianGehrmann]

Hey, can you try adding a rnn_size parameter to your call to get_states.lua?

th model/get_states.lua -data_file data/tinyshakespeare/convert/tiny-shakespeare.hdf5 
-checkpoint_file cv/tinyshakespeare_epoch20.00_420.16.t7 
-output_file data/reads/tinyshakespeare_states.h5 
-rnn_size 128

[tlhumphrey2]

Thank you for your help. Adding -rnn_size 128 and -gpuid 0 fixed the problem. get_states.lua ran to completion.

[tlhumphrey2]

Still having problems. Now with server.py. I did the following:
python server.py -dir data/reads
And get the following error message:
Traceback (most recent call last): File "server.py", line 266, in <module> create_data_handlers(args.dir) File "server.py", line 252, in create_data_handlers data_handlers[dh_id] = LSTMDataHandler(directory=p_dir, config=config) File "/home/ubuntu/cat-dog-data/LSTMVis/lstmdata/data_handler.py", line 66, in __init__ self.config['word_sequence']['size'] = list(self.h5_files[ws['file']][ws['path']].shape) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-4rPeHA-build/h5py/_objects.c:2684) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-4rPeHA-build/h5py/_objects.c:2642) File "/home/ubuntu/.local/lib/python2.7/site-packages/h5py/_hl/group.py", line 166, in __getitem__ oid = h5o.open(self.id, self._e(name), lapl=self._lapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-4rPeHA-build/h5py/_objects.c:2684) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-4rPeHA-build/h5py/_objects.c:2642) File "h5py/h5o.pyx", line 190, in h5py.h5o.open (/tmp/pip-4rPeHA-build/h5py/h5o.c:3570) KeyError: "Unable to open object (Object 'tiny-shakespeare.hdf5' doesn't exist)" ubuntu@ip-10-16-6-230:~/cat-dog-data/LSTMVis$ packet_write_wait: Connection to 54.214.11.185 port 22: Broken pipe
which is saying tiny-Shakespeare.hdf5 doesn't exist, but here is the contents of the directory data/reads:
-rw-rw-r-- 1 ubuntu ubuntu 920 Jun 8 14:28 lstm.yml -rw-r--r-- 1 ubuntu ubuntu 6263320 Jun 8 13:27 tiny-shakespeare.hdf5 -rw-rw-r-- 1 ubuntu ubuntu 39981152 Jun 8 13:20 tiny-shakespeare_states.h5 -rw-r--r-- 1 ubuntu ubuntu 302129 Jun 8 13:28 tiny-shakespeare.targ.dict
I have a feeling that my lstm.yml does NOT contain the right stuff. Here is it:
`name: tinyshakespeare # project name
description: this is a small sample of shakespeare's works # little description

files: # assign files to reference name
states: tiny-shakespeare_states.h5 # HDF5 files have to end with .h5 or .hdf5 !!!
train: tiny-shakespeare.hdf5 # word ids of training set
words: tiny-shakespeare.targ.dict # dict files have to end with .dict !!

word_sequence: # defines the word sequence
file: train # HDF5 file
path: tiny-shakespeare.hdf5 # path to table in HDF5
dict_file: words # dictionary to map IDs from HDF5 to words

states: # section to define which states of your model you want to look at
file: states # HDF5 files containing the state for each position
types: [
{type: state, layer: 1, path: states1}, # type={state, output}, layer=[1..x], path = HDF5 path
{type: state, layer: 2, path: states2},
{type: output, layer: 2, path: output2}
]
`
Any help would be most appreciated.