time sync decoding for asr #4792

brianyan918 · 2022-11-28T01:04:16Z

This PR supports time synchronous decoding for ASR.

Librispeech_100 results:

WER

dataset	Snt	Wrd	Corr	Sub	Del	Ins	Err	S.Err
beam20_ctc0.3/dev_clean	2703	54402	94.5	5.1	0.4	0.8	6.3	56.3
beam20_ctc0.3/dev_other	2864	50948	84.6	13.9	1.5	2.1	17.4	79.9
beam20_ctc0.3/test_clean	2620	52576	94.3	5.3	0.4	0.8	6.5	57.0
beam20_ctc0.3/test_other	2939	52343	84.7	13.7	1.6	2.0	17.3	81.6
timesync_beam20_ctc0.3/dev_clean	2703	54402	94.4	5.1	0.5	0.7	6.3	56.6
timesync_beam20_ctc0.3/dev_other	2864	50948	83.9	13.4	2.7	1.8	17.8	80.3
timesync_beam20_ctc0.3/test_clean	2620	52576	94.3	5.2	0.5	0.7	6.5	57.3
timesync_beam20_ctc0.3/test_other	2939	52343	84.1	13.4	2.4	1.8	17.7	82.2

RTF

Note on RTF:

I have not yet implemented batching for the attentional decoder's fwd pass. This means that compared to the batchified label synchronous algorithm, this time synchronous algorithm is currently slower.
This gap is widened on GPU inference, as shown above.

sw005320

can you add the result in egs2/librispeech_100/asr1/README.md?
does this support the GPU inference? (if so, please add some comments)
add some test functions. Please check some examples in the normal beam search

sw005320 · 2022-11-28T15:13:12Z

espnet2/bin/asr_inference.py

+                    sos=asr_model.sos,
+                    decoder=asr_model.decoder,
+                    lm=scorers["lm"] if "lm" in scorers else None,
+                    ctc_weight=ctc_weight,


Can you make the weight and scores to be the same interface as the normal BeamSearch?

sw005320 · 2022-11-28T15:14:05Z

espnet2/bin/asr_inference.py

+            if time_sync:
+                beam_search = TimeSyncBeamSearch(
+                    beam_size=beam_size,
+                    ctc=asr_model.ctc,


Can you add some error handling when there is no ctc?

sw005320 · 2022-11-28T15:17:55Z

espnet/nets/time_sync_beam_search.py

+            lambda: (float("-inf"), float("-inf"))
+        )  # (p_nb, p_b)
+        tmp = []
+        for l in hyps:


I recommend you add some logging.debug functions of how the hypothesis grows in each time step.

sw005320 · 2022-11-28T15:19:20Z

espnet/nets/time_sync_beam_search.py

+        beam_size: num hyps
+        sos: sos index
+        ctc: CTC module
+        pre_beam_ratio: pre_beam_ratio * beam_size = pre_beam


It is better to have more explanations of pre_beam

sw005320 · 2022-11-28T15:19:53Z

@YosukeHiguchi, could you also review this PR?

mergify · 2022-11-30T16:14:17Z

This pull request is now in conflict :(

…net-ml into time_sync_asr_pr

codecov · 2022-12-16T02:33:17Z

Codecov Report

Merging #4792 (796b000) into master (7a203d5) will increase coverage by 0.04%.
The diff coverage is 93.95%.

@@            Coverage Diff             @@
##           master    #4792      +/-   ##
==========================================
+ Coverage   80.62%   80.66%   +0.04%     
==========================================
  Files         535      536       +1     
  Lines       47379    47517     +138     
==========================================
+ Hits        38197    38329     +132     
- Misses       9182     9188       +6

Flag	Coverage Δ
test_integration_espnet1	`66.38% <ø> (ø)`
test_integration_espnet2	`48.62% <18.79%> (-0.16%)`	⬇️
test_python	`69.17% <93.95%> (+0.07%)`	⬆️
test_utils	`23.34% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
espnet2/bin/asr_inference.py	`85.18% <57.89%> (-1.21%)`	⬇️
espnet/nets/beam_search_timesync.py	`99.23% <99.23%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

brianyan918 · 2022-12-16T02:43:36Z

@sw005320 I think this passed CI except for an unrelated test

sw005320 · 2022-12-16T12:32:09Z

@sw005320 I think this passed CI except for an unrelated test

I'm working on it #4826

sw005320 · 2022-12-16T19:54:08Z

Can you add a unit test?

brianyan918 · 2022-12-21T17:15:13Z

Can you add a unit test?

Sorry for the delay. I added a unit test; it is very basic and just checks that the beam search produces some result. I do not compare to any hand computed values.

sw005320

I have two comments

naming: by following beam_search.py and beam_search_transducer.py, it would be better to name it as beam_search_timesync.py. Please also change the other names (e.g., the class name TimeSyncBeamSearch --> BeamSearchTimeSync accordingly.
I'm thinking about where is the best place to put this function. We generally want to add a new function under the espnet2 directory (e.g., espnet2/asr). But at the same time, putting it in its current place also makes sense in terms of putting all beam search-related functions in the same directory. I think we can leave it based on your idea, but I just added this remark for the future beam search design.

brianyan918 · 2022-12-24T18:04:50Z

Thanks @sw005320. This is fixed now.

sw005320 · 2022-12-24T19:35:57Z

Thanks, @brianyan918 !

time sync decoding for asr

1e505a6

mergify bot added ESPnet1 ESPnet2 labels Nov 28, 2022

black

a3e9826

sw005320 added this to the v.202211 milestone Nov 28, 2022

sw005320 added New Features ASR Automatic speech recogntion labels Nov 28, 2022

sw005320 reviewed Nov 28, 2022

View reviewed changes

mergify bot added the conflicts label Nov 30, 2022

kan-bayashi modified the milestones: v.202211, v.202301 Dec 11, 2022

Update README.md

11b3bbe

mergify bot added the README label Dec 12, 2022

Merge branch 'master' into time_sync_asr_pr

4db7ef6

mergify bot removed the conflicts label Dec 12, 2022

brianyan918 added 9 commits December 12, 2022 19:25

support time sync decoding gpu inference

2d85ebc

Merge branch 'time_sync_asr_pr' of https://github.com/brianyan918/esp…

614e303

…net-ml into time_sync_asr_pr

support time sync decoding gpu inference

2363f35

isort

48bdddf

docstrings

b64214e

docstrings

b2df313

docstrings

76428a2

docstrings

16035c4

docstrings

aed34ca

Merge branch 'master' into time_sync_asr_pr

58af7ec

ts bs unit test

d6dfc12

Brian Yan and others added 3 commits December 21, 2022 13:38

ts bs unit test

c7dac1b

Merge branch 'master' into time_sync_asr_pr

0ab198d

Merge branch 'master' into time_sync_asr_pr

0149ce2

sw005320 reviewed Dec 24, 2022

View reviewed changes

Brian Yan added 2 commits December 24, 2022 12:48

rename timesyncbeamsearch

f7ec4f9

rename timesyncbeamsearch

796b000

sw005320 added the auto-merge Enable auto-merge label Dec 24, 2022

mergify bot merged commit 2096f15 into espnet:master Dec 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

time sync decoding for asr #4792

time sync decoding for asr #4792

brianyan918 commented Nov 28, 2022 •

edited

sw005320 left a comment

sw005320 Nov 28, 2022

brianyan918 Dec 13, 2022

sw005320 Nov 28, 2022

brianyan918 Dec 13, 2022

sw005320 Nov 28, 2022

brianyan918 Dec 13, 2022

sw005320 Nov 28, 2022

brianyan918 Dec 13, 2022

sw005320 commented Nov 28, 2022

mergify bot commented Nov 30, 2022

codecov bot commented Dec 16, 2022 •

edited

brianyan918 commented Dec 16, 2022

sw005320 commented Dec 16, 2022

sw005320 commented Dec 16, 2022

brianyan918 commented Dec 21, 2022

sw005320 left a comment

brianyan918 commented Dec 24, 2022

sw005320 commented Dec 24, 2022

time sync decoding for asr #4792

time sync decoding for asr #4792

Conversation

brianyan918 commented Nov 28, 2022 • edited

WER

RTF

sw005320 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sw005320 commented Nov 28, 2022

mergify bot commented Nov 30, 2022

codecov bot commented Dec 16, 2022 • edited

Codecov Report

brianyan918 commented Dec 16, 2022

sw005320 commented Dec 16, 2022

sw005320 commented Dec 16, 2022

brianyan918 commented Dec 21, 2022

sw005320 left a comment

Choose a reason for hiding this comment

brianyan918 commented Dec 24, 2022

sw005320 commented Dec 24, 2022

brianyan918 commented Nov 28, 2022 •

edited

codecov bot commented Dec 16, 2022 •

edited