Update x_ops.cc #51
Merged
Update x_ops.cc #51
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lingvo-bot
pushed a commit
that referenced
this pull request
Mar 28, 2019
PiperOrigin-RevId: 240686612
copybara-service bot
pushed a commit
that referenced
this pull request
Oct 24, 2021
…els. Highlights: 1. Introducing additional terminating condition "force_last_chunk_eoc_in_topk". This option is in spirit similar to the existing "force_eos_in_topk". If force_last_chunk_eoc_in_topk set to True, when the local score of last chunk eoc is not high enough to enter the top-(k+2) extensions of the input hyp, we force it to be included. This makes it easier to terminate by last chunk eoc. Even with the rest changes, this modification was crucial to bring down the TPU WER for voice search to 6.4% from around 17.0% (see b/199517196, #70). In the unit test of beam_search_equivalence_test.py, this option helps the voice search model to terminate on the "triumphal ornaments" utterance even for the cpu beam search helper. The implementation of this logic was recently checked in for the cpu beam search helper in cl/402971620, and here we test the equivalence with real model and data. 2. Writing a new merge_hyps operation to perform exact path merging. This is achieved by maintaining the token history for active hyps, similarly to the "prev_labels" field of the Hyp structure in cpu beam search. The proposed implementation does not use any loop and shall be time efficient. Also the history is overwritten at each step, and we do not store this info across all decoding steps. The time and memory complexity for merge_hyps is of the order O(num_beams * num_hyps_per_beam^2 * target_seq_len) and in our typical RNN-T use case, num_beams=32, num_hyps_per_beam=4 or 8, target_seq_len=256 for EMBR training; such complexity shall be affordable. Note the previous merge_hyps implementation uses hash maps to approximately maintain hyp history, and it was observed that hash map collision occur frequently (see b/199517196, #51), causing the merged score to be wrong. Other changes: 3. Providing the option to disable fast gather of float32 and bool matrices by matmul, throughout the tpu beam search helper. It was observed that matmul may cause severe loss of precision, see cl/402747554 and b/199517196 (#52). 4. Switched the order of merge_hyps and determining terminating hyps. Previously we merge hyps before terminating, and thus it is possible that a last chunk eoc (last frame blank) is merged and this extension is killed, making it harder to terminate the hyp. 5. Move last chunk eoc off the beam regardless of whether it can terminate. The reason is that if a eoc extension survives pruning, it will advance frame index for RNN-T models. And if a last chunk eoc survives pruning, it will try to move frame index out of bound (though we do have safeguard for this in steps/rnnt_features.py). 6. Due to changes 4 and 5, we now handle terminations before merging hyps, and therefore terminating scores are not merged to active hyps. I think this is the correct behavior and is consistent with CPU beam search. Another advantage with this change is that we can now perform top-k selection directly on the remaining extensions (there are k * (k+2) of them) without any more eos and eoc tokens. This allows us to completely remove the confusing 2*k pre-selection logic in previous implementation. 7. Introduced new option "combine_eos_and_eoc_strategy". When both eoc and eos can terminate the hypothesis, this option specifies whether to use the higher score (what TPU beam search was using) or the lower score (what the CPU beam search has been using). PiperOrigin-RevId: 405278044
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.