evaluation of short sentences

[pajkossy]

currently while preparing datasets very short sentences are dropped.
even if training is not possible with them the test data could contain them so that test results are reliable

[pajkossy]

I changed line 138 of word_tagger_dataset.py (if len(word) < 3; continue to if len(word)< 1, continue), when tring to train in the resulted dataset I got the error below
(it is possible to train with the < 2 constraint)

Traceback (most recent call last):
File "hunvec/seqtag/trainer.py", line 123, in
main()
File "hunvec/seqtag/trainer.py", line 119, in main
wt.train()
File "/home/pajkossy/Proj/hunvec/hunvec/seqtag/sequence_tagger.py", line 242, in train
self.algorithm.train(dataset=self.dataset['train'])
File "/home/pajkossy/pylearn2/pylearn2/training_algorithms/sgd.py", line 455, in train
self.sgd_update(*batch)
File "/home/pajkossy/hunvec_env/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 606, in call
storage_map=self.fn.storage_map)
File "/home/pajkossy/hunvec_env/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 595, in call
outputs = self.fn()
File "/home/pajkossy/hunvec_env/local/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 672, in rval
r = p(n, [x[0] for x in i], o)
File "/home/pajkossy/hunvec_env/local/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 661, in
self, node)
File "scan_perform.pyx", line 207, in theano.scan_module.scan_perform.perform (/home/pajkossy/.theano/compiledir_Linux-3.16--amd64-x86_64-with-debian-8.2--2.7.9-64/scan_perform/mod.cpp:2172)
NotImplementedError: We didn't implemented yet the case where scan do 0 iteration
Apply node that caused the error: forall_inplace,gpu,scan_fn}(Elemwise{Composite{minimum(maximum(((i0 + i1) - i1), (i2 - i1)), i3)}}.0, GpuSubtensor{int64:int64:int64}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, GpuDimShuffle{1,0}.0)
Inputs types: [TensorType(int64, scalar), CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, matrix)]
Inputs shapes: [(), (0, 17), (2, 17), (17, 17)]
Inputs strides: [(), (17, 1), (17, 1), (1, 17)]
Inputs values: [array(0), <CudaNdarray object at 0x7f9a661fb270>, 'not shown', 'not shown']

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

[zseder]

I think Theano/Theano#3276 will solve the issue, so only an update of theano is needed, but still testing...

[zseder]

solved in #102