Adding WavLM implementation #3242

jiamingkong · 2023-05-15T03:41:16Z

PR types

New features

PR changes

Models

Describe

This PR implements the WavLM model as in https://arxiv.org/abs/2110.13900 for speech recognition. On Librispeech clean set, the model finetuned from wavlm-base-plus has a WER of 6.0%.

复现效果：

在Librispeech Clean 100小时数据集上，不使用语言模型：

模型	论文精度	paddle复现精度	torch复现精度
wavlm-base	5.7%	5.8%	6.8%
wavlm-base-plus	4.7%	5.6%	-

对wavlm-base模型的paddle复现可以接近论文效果，优于torch实现。

torch 复现链接：https://huggingface.co/patrickvonplaten/wavlm-libri-clean-100h-base-plus

对齐效果

如果使用上面torch复现的超参数进行训练，paddle版本可以得到6.9%左右精度，与torch一致。
为了实现论文中的效果，在paddle复现中，我们改动了模型：
- 结构上，wavlm 编码后，后面衔接了三层MLP + BatchNorm + 激活函数（和paddlespeech中的wav2vec2ASR实现一致）
- 训练上，wavlm权重有一个带有预热的更小学习率的优化器，三层MLP有学习率正常的优化器

以上两个优化可以训练出精度接近论文描述的模型。

paddle-bot · 2023-05-15T03:41:20Z

Thanks for your contribution!

examples/librispeech/asr5/README.md

paddlespeech/s2t/models/wavlm/processing/signal_processing.py

paddlespeech/s2t/models/wavlm/processing/speech_augmentation.py

zxcd · 2023-05-16T03:25:25Z

conv_layers.py空文件删一删

zxcd · 2023-05-16T03:25:48Z

examples/librispeech/asr5/README.md

+Stage 0 also downloads the pre-trained [hubert](https://paddlespeech.bj.bcebos.com/hubert/hubert-large-lv60.pdparams) model.
+```bash
+mkdir -p exp/hubert
+wget -P exp/hubert https://paddlespeech.bj.bcebos.com/hubert/hubert-large-lv60.pdparams


模型链接改掉

examples/librispeech/asr5/RESULTS.md

zh794390558 · 2023-05-16T03:39:22Z

examples/librispeech/asr5/avg_model.py

@@ -0,0 +1,18 @@
+#!/usr/bin/env python3


这个文件可以删除，应该path.sh配置路径吧

zh794390558 · 2023-05-16T03:39:48Z

examples/librispeech/asr5/compute_wer.py

@@ -0,0 +1,558 @@
+# Copyright 2021 Mobvoi Inc. All Rights Reserved.


删除，新增path.sh文件

zh794390558 · 2023-05-16T06:13:15Z

examples/librispeech/asr5/format_rsl.py

@@ -0,0 +1,143 @@
+# Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.


examples/librispeech/asr5/test.profile

zh794390558 · 2023-05-16T06:15:02Z

paddlespeech/s2t/exps/wavlm/bin/test.py

+# from paddlespeech.utils.argparse import print_arguments
+import distutils.util
+
+def add_arguments(argname, type, default, help, argparser, **kwargs):


可以复用已有的函数。

zh794390558 · 2023-05-16T06:15:09Z

paddlespeech/s2t/exps/wavlm/bin/test.py

+        help=help + ' Default: %(default)s.',
+        **kwargs)
+
+def print_arguments(args, info=None):


zh794390558 · 2023-05-16T06:15:33Z

paddlespeech/s2t/exps/wavlm/bin/train.py

+
+import distutils.util
+
+def add_arguments(argname, type, default, help, argparser, **kwargs):


zh794390558 · 2023-05-16T06:15:38Z

paddlespeech/s2t/exps/wavlm/bin/train.py

+        help=help + ' Default: %(default)s.',
+        **kwargs)
+
+def print_arguments(args, info=None):


zh794390558 · 2023-05-16T06:16:19Z

paddlespeech/s2t/models/wavlm/modules/activations.py

+}
+
+
+def get_activation(activation_string):


是否可以服用modules目录里的

zh794390558 · 2023-05-16T06:16:42Z

paddlespeech/s2t/models/wavlm/modules/conv_layers.py

jiamingkong · 2023-05-21T09:54:26Z

收到，上述的内容我会修改完成。

我先附带上权重：

链接：https://pan.baidu.com/s/1Yjv1rITAWeYv-MjRJD-PPg?pwd=wavl
提取码：wavl

zh794390558 · 2023-05-25T06:39:47Z

examples/librispeech/asr5/format_rsl.py 可以删除，在utils目录下有

jiamingkong · 2023-05-25T06:59:52Z

复现效果：

在Librispeech Clean 100小时数据集上，不使用语言模型：

模型	论文精度	paddle复现精度	torch复现精度
wavlm-base	5.7%	5.8%	6.8%
wavlm-base-plus	4.7%	5.6%	-

对wavlm-base模型的paddle复现可以接近论文效果，优于torch实现。

torch 复现链接：https://huggingface.co/patrickvonplaten/wavlm-libri-clean-100h-base-plus

对齐效果

如果使用上面torch复现的超参数进行训练，paddle版本可以得到6.9%左右精度，与torch一致。
为了实现论文中的效果，在paddle复现中，我们改动了模型：
- 结构上，wavlm 编码后，后面衔接了三层MLP + BatchNorm + 激活函数（和paddlespeech中的wav2vec2ASR实现一致）
- 训练上，wavlm权重有一个带有预热的更小学习率的优化器，三层MLP有学习率正常的优化器

以上两个优化可以训练出精度接近论文描述的模型。

paddlespeech/s2t/models/wavlm/modules/functional.py

zh794390558 · 2023-05-30T02:31:09Z

paddlespeech/s2t/models/wavlm/modules/functional.py

+    return out
+
+
+def addr(input, vec1, vec2, beta=1, alpha=1, out=None):


这是算什么的？加下doc-string.

已处理，这个是计算alpha * (vec1 * vec2.T) + beta * input的帮手函数，用于attention的QK计算

paddlespeech/s2t/models/wavlm/modules/functional.py

paddlespeech/s2t/models/wavlm/modules/modules.py

zh794390558 · 2023-05-30T02:38:19Z

paddlespeech/s2t/models/wavlm/modules/modules.py

+        # normal_(module.v_proj.weight.data)
+
+
+def quant_noise(module, p, block_size):


这个有用吗？

有的，如果要从头预训练wavlm则需要这个函数

paddlespeech/s2t/models/wavlm/wavlm_paddle.py

paddlespeech/s2t/models/wavlm/modules/modules.py

zh794390558

LGTM

zxcd · 2023-05-31T03:59:04Z

examples/librispeech/asr5/conf/preprocessor_config.json

@@ -0,0 +1,9 @@
+{
+  "do_normalize": true,
+  "feature_extractor_type": "Wav2Vec2FeatureExtractor",


这个文件是否有用到？

应该是有的，feature extractor是取出音频，然后减去了平均值，得到一个mean = 0 的[ time * 16000, 1]的tensor

zxcd · 2023-05-31T06:04:04Z

paddlespeech/s2t/exps/wavlm/bin/test_wav.py

+logger = Log(__name__).getlog()
+
+
+class Wav2vec2Infer():


zxcd · 2023-05-31T06:11:52Z

examples/librispeech/asr5/run.sh

+. ./path.sh || exit 1;
+. ./cmd.sh || exit 1;
+
+gpus=1,2,3


最好能从0开始

…in model options

zh794390558

LGTM

zh794390558

LGTM

zxcd

LGTM

zh794390558

LGTM

Adding WavLM implementation

3b6651b

paddle-bot bot added contributor status: proposed labels May 15, 2023

mergify bot added S2T asr/st Example README labels May 15, 2023

zxcd reviewed May 16, 2023

View reviewed changes

examples/librispeech/asr5/README.md Outdated Show resolved Hide resolved

zxcd reviewed May 16, 2023

View reviewed changes

paddlespeech/s2t/models/wavlm/processing/signal_processing.py Outdated Show resolved Hide resolved

zxcd reviewed May 16, 2023

View reviewed changes

paddlespeech/s2t/models/wavlm/processing/speech_augmentation.py Outdated Show resolved Hide resolved

zxcd reviewed May 16, 2023

View reviewed changes

examples/librispeech/asr5/RESULTS.md Outdated Show resolved Hide resolved

zh794390558 reviewed May 16, 2023

View reviewed changes

Code clean up according to comments in PaddlePaddle#3242

60bd7f2

jiamingkong mentioned this pull request May 22, 2023

【PaddlePaddle Hackathon 第四期】任务总览 PaddlePaddle/Paddle#51281

Closed

jiamingkong added 3 commits May 23, 2023 01:48

Changed the path for the uploaded weight

9ee1205

Adapted wavlmASR model to pretrained weights and CLI

232dcf8

Changed the MD5 of the pretrained tar file due to bug fixes

2ea0075

Deleted examples/librispeech/asr5/format_rsl.py

927c60a

zh794390558 reviewed May 30, 2023

View reviewed changes

jiamingkong and others added 2 commits May 30, 2023 11:23

Merge branch 'PaddlePaddle:develop' into develop

3ef28de

Code clean up for CIs

0e2068e

zh794390558 reviewed May 30, 2023

View reviewed changes

paddlespeech/s2t/models/wavlm/modules/modules.py Outdated Show resolved Hide resolved

Fixed the transpose usages ignored before

ba874db

zh794390558 previously approved these changes May 31, 2023

View reviewed changes

zxcd reviewed May 31, 2023

View reviewed changes

Final cleaning; Modified SSL/infer.py and README for wavlm inclusion …

8432e86

…in model options

jiamingkong dismissed zh794390558’s stale review via 8432e86 May 31, 2023 07:07

mergify bot added CLI Demo labels May 31, 2023

zh794390558 approved these changes May 31, 2023

View reviewed changes

zh794390558 previously approved these changes May 31, 2023

View reviewed changes

updating readme and readme_cn

f8b7d76

jiamingkong dismissed zh794390558’s stale review via f8b7d76 May 31, 2023 12:31

zxcd approved these changes May 31, 2023

View reviewed changes

zh794390558 approved these changes May 31, 2023

View reviewed changes

zh794390558 merged commit 2214c0d into PaddlePaddle:develop Jun 1, 2023
5 of 7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding WavLM implementation #3242

Adding WavLM implementation #3242

jiamingkong commented May 15, 2023 •

edited by zh794390558

paddle-bot bot commented May 15, 2023

zxcd commented May 16, 2023

zxcd May 16, 2023

zh794390558 May 16, 2023

zh794390558 May 16, 2023

zh794390558 May 16, 2023

zh794390558 May 16, 2023

zh794390558 May 16, 2023

zh794390558 May 16, 2023

zh794390558 May 16, 2023

zh794390558 May 16, 2023

zh794390558 May 16, 2023

jiamingkong commented May 21, 2023

zh794390558 commented May 25, 2023

jiamingkong commented May 25, 2023 •

edited

zh794390558 May 30, 2023

jiamingkong May 30, 2023

zh794390558 May 30, 2023

jiamingkong May 30, 2023

zh794390558 left a comment

zxcd May 31, 2023

jiamingkong May 31, 2023

zxcd May 31, 2023

jiamingkong May 31, 2023

zxcd May 31, 2023

jiamingkong May 31, 2023

zh794390558 left a comment

zh794390558 left a comment

zxcd left a comment

zh794390558 left a comment

		@@ -0,0 +1,558 @@
		# Copyright 2021 Mobvoi Inc. All Rights Reserved.

		@@ -0,0 +1,143 @@
		# Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved.


		import distutils.util

		def add_arguments(argname, type, default, help, argparser, **kwargs):

		return out


		def addr(input, vec1, vec2, beta=1, alpha=1, out=None):

		# normal_(module.v_proj.weight.data)


		def quant_noise(module, p, block_size):

		logger = Log(__name__).getlog()


		class Wav2vec2Infer():

Adding WavLM implementation #3242

Adding WavLM implementation #3242

Conversation

jiamingkong commented May 15, 2023 • edited by zh794390558

PR types

PR changes

Describe

复现效果：

对齐效果

paddle-bot bot commented May 15, 2023

zxcd commented May 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiamingkong commented May 21, 2023

zh794390558 commented May 25, 2023

jiamingkong commented May 25, 2023 • edited

复现效果：

对齐效果

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zh794390558 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zh794390558 left a comment

Choose a reason for hiding this comment

zh794390558 left a comment

Choose a reason for hiding this comment

zxcd left a comment

Choose a reason for hiding this comment

zh794390558 left a comment

Choose a reason for hiding this comment

jiamingkong commented May 15, 2023 •

edited by zh794390558

jiamingkong commented May 25, 2023 •

edited