Option for GTN CTC mode

[brianyan918]

This PR contains an implementation of CTC using FB's GTN package, enabling another ctc_type called 'gtnctc'. This mode was tested using Voxforge Italian:

Model	CER	WER
Conformer (warpctc)	8.5	30.0
Conformer (builtin)	8.6	30.6
Conformer (gtnctc)	8.5	30.0

• training curve shows spikes for builtin (red) and gtnctc (blue), but not for warpctc (orange)
  

• training times, from single 2080 GPU trials

Model	Train Time
Conformer (warpctc)	16h 4m
Conformer (builtin)	17h 23m
Conformer (gtnctc)	19h 54m

[sw005320]

Added @awni
We are planning to support gtn: first as another CTC, then try to leverage the full gtn-based differentiable wFSTs in the near future.
We welcome any comments from you!

[awni]

Hi! This looks great, I'm glad to see GTN is working. The run time in comparison to warp CTC and builtin (PyTorch?) is not bad, I assume the timings are dominated by something else in the pipeline. I will take a look at the implementation.

For what it's worth, we have seen these kind of spikes before in CTC training though I don't recall what caused them. They haven't been an issue for us though they are somewhat concerning..

[awni]

CC @vineelpratap who may recall some issues related to the spiking behaviour

CC @shubho who is working on a much faster GTN compose which should speed things up a lot here.

[vineelpratap]

Regarding spiking behavior, is it possible that there could be samples which can trigger edge cases for CTC... Depending how the library treats them, the behavior can be different..

For example - L = 0 or T + R > L where L - label length, R - label repeats, T - time frames

[codecov]

Codecov Report

Merging #2866 (9463313) into master (0866492) will decrease coverage by 0.18%.
The diff coverage is 4.22%.

@@            Coverage Diff             @@
##           master    #2866      +/-   ##
==========================================
- Coverage   80.64%   80.45%   -0.19%     
==========================================
  Files         323      324       +1     
  Lines       28660    28730      +70     
==========================================
+ Hits        23114    23116       +2     
- Misses       5546     5614      +68     

Impacted Files	Coverage Δ
espnet/bin/asr_train.py	91.01% <ø> (ø)
espnet/nets/pytorch_backend/gtn_ctc.py	0.00% <0.00%> (ø)
espnet/nets/pytorch_backend/ctc.py	64.16% <27.27%> (-4.02%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0866492...9463313. Read the comment docs.

[sw005320]

Many thanks, @brianyan918!
I just merge this PR.
But can you make a follow-up PR to deal with the following items?

• includes the installation (pip install) of gtn (in setup.py?)
• add a test
• please confirm that it would work in both espnet1 and espnet2
• add documentation about the CTC options in https://espnet.github.io/espnet/tutorial.html#ctc-attention-and-hybrid-ctc-attention

[sw005320]

Also, k2 once you fix the installation issue.

[sw005320]

@awni and @vineelpratap, thanks for the valuable comments!
As I said, we'll add further examples of gtn.
Stay tuned!

[brianyan918]

@sw005320 got it. Thanks!