[WIP] E-Branchformer Encoder in ESPnet2 #4812

kkim-asapp · 2022-12-06T19:54:18Z

Hi,
This PR is about the E-Branchformer encoder (Kim et al., SLT 2022: will be appeared, Arxiv: link).

The module is implemented in ESPnet2 with the base recipe for Librispeech ASR.
Also, I've adding an argument in Repeat module to support LayerDrop (link).

This PR is in progress, and I will update results on README file.

pyf98 · 2022-12-06T20:26:51Z

espnet2/asr/encoder/e_branchformer_encoder.py

+            x_input (Union[Tuple, torch.Tensor]): Input tensor w/ or w/o pos emb.
+                - w/ pos emb: Tuple of tensors [(#batch, time, size), (1, time, size)].
+                - w/o pos emb: Tensor (#batch, time, size).
+            mask (torch.Tensor): Mask tensor for the input (#batch, time).


The shape of mask is (#batch, 1, time)?

Hi @kkim-asapp Is the shape correct?

Thanks, @pyf98! You're right. It was wrong. Let me update it.

I remember it was wrong in Conformer (or other encoders). If so, can you update it as well?

Sure, will do.

pyf98 · 2022-12-06T20:43:11Z

Looks good to me! I'm looking forward to seeing the new results and models.

Since the repeat function is updated to support the new Layer Drop, we may need to update the other encoders as well (probably in a later PR?).

sw005320 · 2022-12-06T21:24:14Z

@kkim-asapp, thanks a lot for the great PR!
Can you fix the format error?
https://github.com/espnet/espnet/actions/runs/3632971395/jobs/6129594548#step:8:12

kkim-asapp · 2022-12-06T21:40:47Z

Looks good to me! I'm looking forward to seeing the new results and models.

Since the repeat function is updated to support the new Layer Drop, we may need to update the other encoders as well (probably in a later PR?).

Right! Currently, I didn't touch any parts of stochastic_depth_rate which has similar purpose of this new Layer Drop.
If we agree that we don't need stochastic_depth_rate anymore, then I will replace them with this new method as well.

kkim-asapp · 2022-12-06T21:41:06Z

@kkim-asapp, thanks a lot for the great PR! Can you fix the format error? https://github.com/espnet/espnet/actions/runs/3632971395/jobs/6129594548#step:8:12

I did!

kkim-asapp · 2022-12-06T22:43:51Z

espnet/nets/pytorch_backend/conformer/contextual_block_encoder_layer.py
All lines in this file has ^M at the end. I don't think this is what we want, so I removed them.

sw005320 · 2022-12-07T01:12:32Z

You can run some of the test scripts locally instead of checking it via CI.
https://github.com/espnet/espnet/blob/master/CONTRIBUTING.md#4-unit-testing

kkim-asapp · 2022-12-07T01:15:19Z

You can run some of the test scripts locally instead of checking it via CI. https://github.com/espnet/espnet/blob/master/CONTRIBUTING.md#4-unit-testing

I did, but the error logs are different. Let me check after reinstall dependencies.

sw005320 · 2022-12-07T01:16:28Z

Oh, OK.
It also depends on the version of the test script and it would not be the exactly same.

codecov · 2022-12-07T22:04:19Z

Codecov Report

Merging #4812 (a294ca6) into master (a9f7c1e) will increase coverage by 11.38%.
The diff coverage is 54.20%.

@@             Coverage Diff             @@
##           master    #4812       +/-   ##
===========================================
+ Coverage   69.06%   80.44%   +11.38%     
===========================================
  Files         526      534        +8     
  Lines       46125    47118      +993     
===========================================
+ Hits        31858    37906     +6048     
+ Misses      14267     9212     -5055

Flag	Coverage Δ
test_integration_espnet1	`66.38% <100.00%> (?)`
test_integration_espnet2	`48.83% <12.77%> (?)`
test_python	`68.86% <54.20%> (+0.61%)`	⬆️
test_utils	`23.34% <100.00%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
espnet/nets/pytorch_backend/conformer/encoder.py	`93.57% <ø> (ø)`
...et/nets/pytorch_backend/conformer/encoder_layer.py	`93.10% <ø> (ø)`
...kend/transformer/contextual_block_encoder_layer.py	`81.30% <ø> (+6.54%)`	⬆️
espnet/nets/pytorch_backend/transformer/encoder.py	`93.60% <ø> (ø)`
.../nets/pytorch_backend/transformer/encoder_layer.py	`98.07% <ø> (+3.84%)`	⬆️
espnet2/asr/decoder/transformer_decoder.py	`94.95% <ø> (ø)`
espnet2/asr/encoder/branchformer_encoder.py	`94.14% <ø> (ø)`
espnet2/asr/encoder/conformer_encoder.py	`96.29% <ø> (ø)`
...ackend/conformer/contextual_block_encoder_layer.py	`6.08% <6.08%> (ø)`
espnet2/asr/encoder/e_branchformer_encoder.py	`95.09% <95.09%> (ø)`
... and 152 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

kkim-asapp · 2022-12-08T19:01:02Z

Hi @pyf98, is there anything I can do to fix one last failing check? Maybe I need to fix This branch is out-of-date with the base branch by Merge the latest changes from master into this branch., but I'm not sure.

sw005320 · 2022-12-08T19:07:49Z

You don't need it.
I can merge this PR.
Thanks, @kkim-asapp

sw005320 · 2022-12-08T19:08:27Z

Please make a follow-up PR to provide an updated README file etc.

kkim-asapp added 5 commits November 5, 2022 21:51

Add layer_drop and Make max_pos_emb_len configurable

0ad3da9

Add E-Branchformer module

a9885db

Remove unused code and Fix minor errors

f585dc0

Add testcase and update minors

1f89f33

Add baseline recipe for E-branchformer

164bd26

mergify bot added ESPnet1 ESPnet2 labels Dec 6, 2022

sw005320 added New Features ASR Automatic speech recogntion labels Dec 6, 2022

sw005320 added this to the v.202211 milestone Dec 6, 2022

pyf98 reviewed Dec 6, 2022

View reviewed changes

Fix format errors

09c8cc7

kkim-asapp force-pushed the ebranch branch from 34864c5 to 89ad880 Compare December 6, 2022 22:41

Fix wrong comments on the shape of mask tensors

09f5523

kkim-asapp force-pushed the ebranch branch from 89ad880 to 09f5523 Compare December 6, 2022 22:54

kkim-asapp added 2 commits December 6, 2022 15:19

Apply isort

0e0b2f4

Fix style errors

f2fda1c

kkim-asapp force-pushed the ebranch branch from 023e49e to e96c40c Compare December 7, 2022 00:42

Fix ci error

3c793b1

kkim-asapp force-pushed the ebranch branch from e96c40c to 3c793b1 Compare December 7, 2022 01:09

Set default value for argument

a294ca6

sw005320 merged commit e9a940e into espnet:master Dec 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] E-Branchformer Encoder in ESPnet2 #4812

[WIP] E-Branchformer Encoder in ESPnet2 #4812

kkim-asapp commented Dec 6, 2022

pyf98 Dec 6, 2022

pyf98 Dec 6, 2022

kkim-asapp Dec 6, 2022

pyf98 Dec 6, 2022 •

edited

kkim-asapp Dec 6, 2022

pyf98 commented Dec 6, 2022 •

edited

sw005320 commented Dec 6, 2022

kkim-asapp commented Dec 6, 2022

kkim-asapp commented Dec 6, 2022

kkim-asapp commented Dec 6, 2022

sw005320 commented Dec 7, 2022

kkim-asapp commented Dec 7, 2022

sw005320 commented Dec 7, 2022

codecov bot commented Dec 7, 2022 •

edited

kkim-asapp commented Dec 8, 2022

sw005320 commented Dec 8, 2022

sw005320 commented Dec 8, 2022

[WIP] E-Branchformer Encoder in ESPnet2 #4812

[WIP] E-Branchformer Encoder in ESPnet2 #4812

Conversation

kkim-asapp commented Dec 6, 2022

pyf98 Dec 6, 2022

Choose a reason for hiding this comment

pyf98 Dec 6, 2022

Choose a reason for hiding this comment

kkim-asapp Dec 6, 2022

Choose a reason for hiding this comment

pyf98 Dec 6, 2022 • edited

Choose a reason for hiding this comment

kkim-asapp Dec 6, 2022

Choose a reason for hiding this comment

pyf98 commented Dec 6, 2022 • edited

sw005320 commented Dec 6, 2022

kkim-asapp commented Dec 6, 2022

kkim-asapp commented Dec 6, 2022

kkim-asapp commented Dec 6, 2022

sw005320 commented Dec 7, 2022

kkim-asapp commented Dec 7, 2022

sw005320 commented Dec 7, 2022

codecov bot commented Dec 7, 2022 • edited

Codecov Report

kkim-asapp commented Dec 8, 2022

sw005320 commented Dec 8, 2022

sw005320 commented Dec 8, 2022

pyf98 Dec 6, 2022 •

edited

pyf98 commented Dec 6, 2022 •

edited

codecov bot commented Dec 7, 2022 •

edited