Add SEW CTC models #14158
Add SEW CTC models #14158
Conversation
| self.project_features = config.conv_dim[-1] != config.hidden_size | ||
| if self.project_features: | ||
| self.feature_projection = nn.Linear(config.conv_dim[-1], config.hidden_size) | ||
| self.feature_dropout = nn.Dropout(config.feat_proj_dropout) |
There was a problem hiding this comment.
@patrickvonplaten I was wrong about this previously. Both SEW and SEW-D require projections depending on the model size (larger models have different hidded_sizes), and my previous assumption was based only on the tiny checkpoints.
If this design is OK with you, I'll update the already uploaded unsupervised checkpoints as well.
There was a problem hiding this comment.
Yes this sounds good to me! Just to make sure, some checkpoints have the projection and others don't?
There was a problem hiding this comment.
Yes, both SEW and SEW-D have checkpoints with and without the projection (depending on the sizes)
There was a problem hiding this comment.
Basically, any model with config.conv_dim[-1] != config.hidden_size (see the list below) needs the projection, while the others don't. That's how it's implemented in the original sew codebase.
tests/test_modeling_sew.py
Outdated
| model = SEWForCTC.from_pretrained("anton-l/sew-tiny-100k-ft-ls100h").to(torch_device) | ||
| processor = Wav2Vec2Processor.from_pretrained("anton-l/sew-tiny-100k-ft-ls100h", do_lower_case=True) |
There was a problem hiding this comment.
feel free to do it right away :-)
| @@ -1383,12 +1364,13 @@ def forward( | |||
| extract_features = extract_features.transpose(1, 2) | |||
| extract_features = self.layer_norm(extract_features) | |||
|
|
|||
| if self.project_features: | |||
| extract_features = self.feature_projection(extract_features) | |||
There was a problem hiding this comment.
Do we need both cases here? Which checkpoitns have self.projection_features = False and which have self.projection_features =True ?
There was a problem hiding this comment.
project_features = False:
sew-tiny-100ksew-d-small-100ksew-d-mid-100ksew-d-mid-k127-100ksew-d-mid-400ksew-d-mid-k127-400k
project_features = True:
sew-small-100ksew-mid-100ksew-d-tiny-100ksew-d-base-100ksew-d-base-plus-100ksew-d-base-plus-400k
| class SEWDModel(SEWDPreTrainedModel): | ||
| def __init__(self, config: SEWDConfig): | ||
| super().__init__(config) | ||
| self.config = config | ||
| self.feature_extractor = SEWDFeatureExtractor(config) | ||
| self.layer_norm = nn.LayerNorm(config.conv_dim[-1], eps=config.layer_norm_eps) | ||
| self.feature_projection = nn.Linear(config.conv_dim[-1], config.hidden_size) | ||
|
|
||
| self.project_features = config.conv_dim[-1] != config.hidden_size |
There was a problem hiding this comment.
We need both cases I assume no?
* Add SEW CTC models * Update paths * Update paths
What does this PR do?
This adds the conversion steps and bugfixes to support finetuned SEW and SEW-D checkpoints (https://github.com/asappresearch/sew#asr-model-fine-tuned-on-librispeech-train-clean-100h)
TODO