You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
it is collecting all the finished hypothesis in to an array, and expanding the rest of the hypothesis and taking the top K - len(finished). As a result, the finished hypothesis never drops out the final returned results, and the last candidate is effectively found by greedy decoding towards the end.
I believe a more standard (and perhaps better) implementation is to still keep the finished hypothesis in the beam, expand the other hypothesis, and take the top-K hypothesis from the union of the finished hypothesis and the other candidates. In this way, sub-optimal already-finished hypothesis can be excluded.
I do not believe it would substantially change the results in the paper, though; but just to point it out. Or maybe both versions are legit, and it's just I am unaware of the other version before.
In our implementation, we do not consider any length-normalization, so the score of a given hypothesis strictly decreases as you add tokens to it.
As a consequence, if a finished hypothesis is in the top-K at some point, it will stay in the top-K until the end of decoding.
However this would not be the case if the score was not strictly decreasing. Then I believe doing what you say would indeed be a better solution. Does that make sense?
It's super minor, but I noticed that in the beam search implementation
https://github.com/ElementAI/duorat/blob/6fba4c3f08d372465780dea0a2198a650dd407f4/duorat/utils/beam_search.py#L59
it is collecting all the finished hypothesis in to an array, and expanding the rest of the hypothesis and taking the top K - len(finished). As a result, the finished hypothesis never drops out the final returned results, and the last candidate is effectively found by greedy decoding towards the end.
I believe a more standard (and perhaps better) implementation is to still keep the finished hypothesis in the beam, expand the other hypothesis, and take the top-K hypothesis from the union of the finished hypothesis and the other candidates. In this way, sub-optimal already-finished hypothesis can be excluded.
I do not believe it would substantially change the results in the paper, though; but just to point it out. Or maybe both versions are legit, and it's just I am unaware of the other version before.
(Reference: Algo 1 in this paper)
The text was updated successfully, but these errors were encountered: