[ASR] Support Hubert, fintuned on the librispeech dataset #3088
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
|
This pull request is now in conflict :( |
zxcd
reviewed
Mar 27, 2023
| @@ -55,6 +58,8 @@ def __init__(self, config: dict): | |||
| reduction='mean') | |||
|
|
|||
| def forward(self, wav, wavs_lens_rate, target, target_lens): | |||
| # import pdb | |||
| # pdb.set_trace() | |||
zxcd
reviewed
Mar 27, 2023
examples/librispeech/asr3/run.sh
Outdated
| @@ -19,6 +20,7 @@ audio_file=data/demo_002_en.wav | |||
|
|
|||
| avg_ckpt=avg_${avg_num} | |||
| ckpt=$(basename ${conf_path} | awk -F'.' '{print $1}') | |||
| ckpt=test6 | |||
zxcd
reviewed
Mar 27, 2023
dataset/librispeech/librispeech.py
Outdated
| @@ -133,7 +133,7 @@ def create_manifest(data_dir, manifest_path): | |||
| def prepare_dataset(url, md5sum, target_dir, manifest_path): | |||
| """Download, unpack and create summmary manifest file. | |||
| """ | |||
| if not os.path.exists(os.path.join(target_dir, "LibriSpeech")): | |||
There was a problem hiding this comment.
原先的代码似乎和librispeech解压出的结果不太一致，本地已有librispeech数据集的情况下不太方便
zxcd
reviewed
Mar 27, 2023
|
|
||
|
|
||
| task_cfg: | ||
| sample_rate: 16000 |
zxcd
reviewed
Mar 27, 2023
| # Data Augmentation # | ||
| ############################################ | ||
| audio_augment: # for raw audio | ||
| sample_rate: 16000 |
zxcd
reviewed
Mar 27, 2023
| self.mask_emb = paddle.create_parameter( | ||
| shape=[cfg.encoder_embed_dim], | ||
| dtype='float32', | ||
| default_initializer=paddle.nn.initializer.Uniform(), |
There was a problem hiding this comment.
torch和paddle对于uniform的初始化范围不一致，torch为(0,1)，paddle为(-1,1)，可以确定下是否会对训练产生影响，或者直接加上low和high参数
zxcd
reviewed
Mar 27, 2023
| self.label_embs_concat = paddle.create_parameter( | ||
| shape=[sum(self.num_classes), final_dim], | ||
| dtype='float32', | ||
| default_initializer=paddle.nn.initializer.Uniform(), |
zxcd
reviewed
Mar 27, 2023
|
|
||
| return x, mask_indices | ||
|
|
||
| def compute_nce(x, pos, negs): |
zxcd
reviewed
Mar 27, 2023
| from dataclasses import dataclass, field, is_dataclass | ||
| from copy import deepcopy | ||
|
|
||
| from omegaconf import II, MISSING, open_dict |
zxcd
reviewed
Mar 27, 2023
|
|
||
|
|
||
| class HubertBase(nn.Layer): | ||
| """Wav2vec2 model""" |
zxcd
reviewed
Mar 30, 2023
| enc_n_units: 1024 | ||
| blank_id: 0 | ||
| dropout_rate: 0.0 | ||
| hubert_params_path: "exp/hubert/pd_hubert.pdparams" |
zh794390558
reviewed
Apr 11, 2023
| fp16: True | ||
| label_rate: 50 | ||
| extractor_mode: layer_norm | ||
| encoder_layers: 24 |
examples/librispeech/asr3/path.sh
Outdated
| @@ -10,6 +10,5 @@ export PYTHONPATH=${MAIN_ROOT}:${PYTHONPATH} | |||
|
|
|||
| export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib/ | |||
|
|
|||
|
|
|||
| MODEL=wav2vec2 | |||
| MODEL=$1 | |||
There was a problem hiding this comment.
这个不需要固定，不能用传参的方式。如果是和wav2vec一个asr目录的话就单开个吧。
examples/librispeech/asr3/run.sh
Outdated
| stage=0 | ||
| stop_stage=0 | ||
| conf_path=conf/wav2vec2ASR.yaml | ||
| gpus=2 |
| logger = Log(__name__).getlog() | ||
|
|
||
|
|
||
| def clip_grad_norm_( |
There was a problem hiding this comment.
我看wav2vec2目前也用的这个接口？paddle的对应api是哪个？可以用了吗？
There was a problem hiding this comment.
这里我注释了todo，后面paddle依赖改为2.5后再改这里吧
| self.feat2tar_ratio = cfg.label_rate * feature_ds_rate / task_cfg.sample_rate | ||
|
|
||
| self.post_extract_proj = ( | ||
| nn.Linear(self.embed, cfg.encoder_embed_dim) |
There was a problem hiding this comment.
需要替换成align.Linaer，相关的都需要操作下。
| self.target_glu = None | ||
| if cfg.target_glu: | ||
| self.target_glu = nn.Sequential( | ||
| nn.Linear(final_dim, final_dim * 2), GLU()) |
| self.target_glu = nn.Sequential( | ||
| nn.Linear(final_dim, final_dim * 2), GLU()) | ||
|
|
||
| self.final_proj = nn.Linear(cfg.encoder_embed_dim, final_dim) |
| is_group_norm=False, | ||
| conv_bias=False, ): | ||
| def make_conv(): | ||
| conv = nn.Conv1D( |
| def make_conv_block(e, k, g, l): | ||
| return nn.Sequential(*[ | ||
| nn.Sequential( | ||
| nn.Conv1D( |
|
|
||
| # layer norm associated with the self attention layer | ||
| self.self_attn_layer_norm = LayerNorm(self.embedding_dim) | ||
| self.fc1 = nn.Linear(self.embedding_dim, ffn_embedding_dim) |
zxcd
reviewed
Apr 25, 2023
| @@ -0,0 +1,586 @@ | |||
| # Copyright (c) Facebook, Inc. and its affiliates. | |||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Models
Describe
support ASR Hubert