New beam-search framework: ScorerInterface, CPU/GPU float16/32/64 decoding, and new language models (SeqRNNLM and TransformerLM)

[ShigekiKarita]

This PR refactors beam search as discussed in #489

• wrap previous RNNLM as espnet.nets.pytorch_backend.lm.legacy:LegacyRNNLM
• dynamic import using LMInterface
• decoding with LMInterface abstraction instead of current hard-coded ones for RNNLM
  • Transformer ASR Model
  • Length Penalty
  • RNN ASR Model
  • RNNLM
  • CTC Prefix Score
• support more predefined/user-defined LMs, e.g., https://github.com/pytorch/examples/blob/master/word_language_model/model.py
• GPU decoding
• multi precision (float16/32/64) support
• Transformer LM
• fix wrong ppl reported in Fix multi gpu LM issues and add hdf5 LM dataset dump #1083

I will implement these essential feature not to break current features

• custom language model training with lm_train.py --model-module xxx
• ASR decoding with custom language model asr_recog.py --lm xxx (since it is not --rnnlm anymore. I will leave the existing --rnnlm as-is.)

and I will not support the other existing features with custom LMs like ASR training with custom LMs, GPU batch decoding, streaming decoding, word LM, etc in this PR.

[codecov]

Codecov Report

Merging #1092 into master will increase coverage by 2.29%.
The diff coverage is 87.11%.

@@            Coverage Diff             @@
##           master    #1092      +/-   ##
==========================================
+ Coverage   76.36%   78.65%   +2.29%     
==========================================
  Files          83       91       +8     
  Lines        7810     8045     +235     
==========================================
+ Hits         5964     6328     +364     
+ Misses       1846     1717     -129

Impacted Files	Coverage Δ
espnet/nets/scorers/ctc.py	100% <100%> (ø)
espnet/nets/pytorch_backend/transformer/decoder.py	95.74% <100%> (+0.62%)	⬆️
espnet/nets/pytorch_backend/ctc.py	96.07% <100%> (ø)	⬆️
espnet/nets/pytorch_backend/e2e_asr_transformer.py	82.52% <100%> (+12.68%)	⬆️
espnet/nets/pytorch_backend/transformer/mask.py	100% <100%> (ø)
espnet/nets/scorers/length_bonus.py	100% <100%> (ø)
espnet/nets/pytorch_backend/rnn/attentions.py	98.15% <100%> (ø)	⬆️
espnet/nets/pytorch_backend/e2e_asr.py	68.42% <100%> (+0.08%)	⬆️
...pnet/nets/pytorch_backend/transformer/attention.py	100% <100%> (ø)	⬆️
espnet/lm/chainer_backend/lm.py	34.58% <33.33%> (-0.05%)	⬇️
... and 20 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7323eaf...f74d810. Read the comment docs.

[sw005320]

and I will not support the other existing features with custom LMs like ASR training with custom LMs, GPU batch ASR decoding, etc.

Actually, the current GPU batch ASR decoding does not do special things for rnnlm. See

espnet/espnet/lm/pytorch_backend/lm.py

Lines 134 to 146 in f1bb241

	def buff_predict(self, state, x, n):
	if self.predictor.__class__.__name__ == 'RNNLM':
	return self.predict(state, x)
	new_state = []
	new_log_y = []
	for i in range(n):
	state_i = None if state is None else state[i]
	state_i, log_y = self.predict(state_i, x[i].unsqueeze(0))
	new_state.append(state_i)
	new_log_y.append(log_y)
	return new_state, torch.cat(new_log_y)

So, you could just do the same thing for now.
I want to revisit it after #980.

[ShigekiKarita]

@sw005320 As there is no breaking change, I expect master is better.

[sw005320]

@sw005320 As there is no breaking change, I expect master is better.

OK.

[ShigekiKarita]

I have finished easy ones: Transformer and length penalty because they are state-less in the current implementation. Tomorrow, I will implement the stateful ones (RNN ASR, RNN LM, CTC Prefix Score). Currently, the old and new implementations results completely same scores. However, IMO, I do not want to reimplement this local pruning with ctc_beam (its value is hard-coded)

espnet/espnet/nets/pytorch_backend/e2e_asr_transformer.py

Lines 320 to 333 in 6759e7f

	if lpz is not None:
	local_best_scores, local_best_ids = torch.topk(
	local_att_scores, ctc_beam, dim=1)
	ctc_scores, ctc_states = ctc_prefix_score(
	hyp['yseq'], local_best_ids[0], hyp['ctc_state_prev'])
	local_scores = \
	(1.0 - ctc_weight) * local_att_scores[:, local_best_ids[0]] \
	+ ctc_weight * torch.from_numpy(ctc_scores - hyp['ctc_score_prev'])
	if rnnlm:
	local_scores += recog_args.lm_weight * local_lm_scores[:, local_best_ids[0]]
	local_best_scores, joint_best_ids = torch.topk(local_scores, beam, dim=1)
	local_best_ids = local_best_ids[:, joint_best_ids[0]]
	else:
	local_best_scores, local_best_ids = torch.topk(local_scores, beam, dim=1)

I have no idea to share scores between decoder and ctc modules because every decoder runs independently in my impl.

espnet/espnet/nets/pytorch_backend/beam_search.py

Lines 130 to 133 in be342ae

	# scoring
	for k, (d, w) in dec_weights.items():
	scores[k], states[k] = d.score(hyp.yseq, hyp.states[k], x)
	wscores += w * scores[k]

Does this ctc_beam really matter? If so, I will reconsider how to do that...

[sw005320]

Does this ctc_beam really matter? If so, I will reconsider how to do that...

ctc_beam would have a big impact when we have large numbers of output units (e.g., Japanese, Chinese, and BPE). If we don't have it, the beam search would be significantly slower. Actually, the current batch beam search does not have ctc_beam and causes significant speed degradation for Japanese, Chinese, and BPE. One of the benefits of #980 is to fix this.

[ShigekiKarita]

OK. I got how to do it. For simplicity, I will add new option like pre_beam_ratio that is same to this

espnet/espnet/nets/pytorch_backend/e2e_asr_transformer.py

Line 287 in 6759e7f

ctc_beam = min(lpz.shape[-1], int(beam * CTC_SCORING_RATIO))

So first beam_search performs topk with pre-beam size on pre-decoders scores (s2s, nn-lm), then add post-decoders scores (ctc, fst-lm?), and finally topk with the full beam size.

[ShigekiKarita]

@kan-bayashi I have a suggestion for flake8-docstrings. I like this but it is not checked by our CI. So I want CI to do that as follows:

1. pip install flake8-docstrings in ci/install.sh
2. change ci/test_python.sh like

# --extend-ignore for wip files for flake8-docstrings
flake8 --extend-ignore=D espnet test utils

# white list of files that should support flake8-docstrings
flake8 \
  espnet/nets/beam_search.py \
  espnet/nets/lm_interface.py \
  espnet/nets/scorer_interface.py \
  ...

see how to disable flake8-docstrings temporally https://stackoverflow.com/questions/41860123/how-to-ignore-installed-flake8-plugin-easily-for-one-time

[kan-bayashi]

That’s a nice idea.
Maybe the blacklist is better?
Anyway, let us introduce it.

[ShigekiKarita]

I agree. I made the black list. No more invalid docstrings 🍺

Now, I need to fix ppl things in lm.py. Just a moment.

[ShigekiKarita]

I believe it is done. Additionally, I split the flake8 part in ci/test_python.sh to ci/test_flake8.sh because it also helps rapid docstring annotation in local. I also updated doc/README.md for this.

[ShigekiKarita]

Fixed!

[kan-bayashi]

LGTM

[sw005320]

Many thanks!