t2t-decoder bug when using decode_from_file

[vince62s]

As said on gitter, there is a bug when decoding (I think it appeared in 1.2.5)
In the last batch it sent an empty line after the last line of the batch, and it throws this error:

INFO:tensorflow:Inference results INPUT:
INFO:tensorflow:Inference results OUTPUT: Précisions
2017-10-24 21:19:07.312405: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration:
2017-10-24 21:19:07.312474: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration:
[[Node: PyFunc = PyFuncTin=[], Tout=[DT_INT32, DT_INT32], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[vince62s]

@lukaszkaiser @rsepassi
1.2.6 fixes the error BUT it does not fix the fact that decode_from_file with N lines output a decode_to_file with N+1 lines, because it will add an extra empty sequence at the end of the last batch.
Hope this is clear.

[vince62s]

There are actually two different issues:
The first one was introduced by the custom delimiter commit.
here: 545ec34#diff-189703c3c1f690231282073cea1cec41R499

text.split(delimiter) will add an empty element at the end of the array.

easy fix is to modify the line after as:
inputs = [record.strip() for record in records[:-1]]
this solve the issue of the extra line as in my first post.
BUT
it does not solve the issue
W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration:
which means there is another place where there is an index issue

help ....

[martinpopel]

In my case (t2t 1.2.6 translate_ende_wmt32k) the extra line is not empty, but "Herr Präsident!" (always the same line, when translating newstest 2013, 2014 or 2016).