[ModelZoo] Refactor ERNIE-M usage in Model Zoo #4324

Yam0214 · 2023-01-03T13:12:13Z

PR types

Function optimization

PR changes

APIs

Description

Refactor ERNIE-M usage in Model Zoo and add unittest.

…nto zoo_ernie_m

paddle-bot · 2023-01-03T13:12:19Z

Thanks for your contribution!

model_zoo/ernie-m/README.md

tests/fixtures/tests_samples/xnli/XNLI-1.0/README.md

tests/fixtures/tests_samples/xnli/XNLI-1.0/xnli.dev.jsonl

model_zoo/ernie-m/run_classifier.py

…nto zoo_ernie_m

codecov · 2023-01-31T13:12:32Z

Codecov Report

Merging #4324 (03626da) into develop (9f78fa5) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #4324      +/-   ##
===========================================
- Coverage    44.64%   44.63%   -0.01%     
===========================================
  Files          446      446              
  Lines        64361    64361              
===========================================
- Hits         28731    28729       -2     
- Misses       35630    35632       +2

Impacted Files	Coverage Δ
paddlenlp/utils/downloader.py	`65.04% <0.00%> (-0.89%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

scripts/regression/ci_case.sh

model_zoo/ernie-m/run_classifier.py

…nto zoo_ernie_m

model_zoo/ernie-m/README.md

model_zoo/ernie-m/deploy/predictor/inference.py

model_zoo/ernie-m/run_classifier.py

ZHUI · 2023-02-10T09:35:58Z

model_zoo/ernie-m/run_classifier.py

+    warmup = training_args.warmup_steps if training_args.warmup_steps > 0 else training_args.warmup_ratio
+    if training_args.do_train:
+        num_training_steps = (
+            training_args.max_steps
+            if training_args.max_steps > 0
+            else len(train_ds) // training_args.train_batch_size * training_args.num_train_epochs
+        )
    else:
-        num_training_steps = len(train_data_loader) * args.num_train_epochs
-        num_train_epochs = args.num_train_epochs
-
-    warmup = args.warmup_steps if args.warmup_steps > 0 else args.warmup_proportion
-
-    lr_scheduler = LinearDecayWithWarmup(args.learning_rate, num_training_steps, warmup)
+        num_training_steps = 10
+    lr_scheduler = LinearDecayWithWarmup(training_args.learning_rate, num_training_steps, warmup)


ZHUI · 2023-02-10T09:46:26Z

model_zoo/ernie-m/run_classifier.py

@@ -250,94 +231,103 @@ def do_train(args):
    for n, p in model.named_parameters():
        name_dict[p.name] = n

-    simple_lr_setting = partial(layerwise_lr_decay, args.layerwise_decay, name_dict, n_layers)
+    simple_lr_setting = partial(layerwise_lr_decay, model_args.layerwise_decay, name_dict, n_layers)


这里的话，不建议这么用。

https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/optimizer/AdamW_cn.html#daimashili

设置内部的 params 的 learning_rate。造一个 params list 然后再set一下即可。

trainer.set_optimizer_grouped_parameters(params_to_train)

ZHUI · 2023-02-10T09:47:00Z

model_zoo/ernie-m/run_classifier.py

+
+    trainer = Trainer(
+        model=model,
+        criterion=criterion,


不要 criterion 了，模型里面支持 label 输入

ZHUI · 2023-02-10T09:47:23Z

model_zoo/ernie-m/run_classifier.py

+        eval_dataset=eval_ds if training_args.do_eval else None,
+        tokenizer=tokenizer,
+        compute_metrics=compute_metrics,
+        optimizers=[optimizer, lr_scheduler],


Suggested change

optimizers=[optimizer, lr_scheduler],

删除

ZHUI · 2023-02-10T09:52:29Z

model_zoo/ernie-m/run_classifier.py

+    if training_args.do_eval:
+        combined = {}
+        for language in all_languages:
+            eval_ds = load_xnli_dataset(model_args, "xnli", language, split="validation")


这个可以支持一下 eval_ds dict输入

我看了HF的逻辑，如果我们支持 trianer.evaluate(eval_dataset: Union[Dict[str, Dataset], Dataset, None])，会和HF的行为不对齐。
~~如果计划支持这个功能，我另起一个PR实现下。~~

HF里 trainer.evaluate()认为传入的eval_dataset要么是datasets.Dataset，要么是torch.utils.data.IterableDataset（见evaluate和get_eval_dataloader）。如果在实例化trainer时trainer.eval_dataset传入了一个dict，那么在训练调用trainer.evaluate之前先做类型判断，拆出字典中的dataset循环送进trainer.evaluate（见_maybe_log_save_evaluate）。

对这块确实不是这个场景。应该是边训练边 eval的时候支持多个数据输入。

如果计划支持这个功能，我另起一个PR实现下。

有计划的，你可以帮忙加一下吗？张斌那边有这个场景

ZHUI · 2023-02-13T09:24:52Z

model_zoo/ernie-m/run_classifier.py

+        n_layers = model.config.num_hidden_layers
+        for static_name, param in model.named_parameters():
+            if any(nd in static_name for nd in ["bias", "norm"]):
+                params_list.append({"params": param})
+                continue
+
+            if "encoder.layers" in static_name:
+                idx = static_name.find("encoder.layers.")
+                layer = int(static_name[idx:].split(".")[2])
+                ratio = layerwise_decay ** (n_layers - layer)
+            elif "embedding" in static_name:
+                ratio = layerwise_decay ** (n_layers + 1)
+
+            params_list.append({"params": param, "learning_rate": param.optimize_attr["learning_rate"] * ratio})
+        return params_list


weight_decay 和 learning_rate 是独立设置的。不能 continue 跳过。

这里 param.optimize_attr["learning_rate"] 应该不需要吧？

ZHUI

LGTM

Yam0214 added 2 commits January 3, 2023 08:17

update model_zoo/ernie-m

b920c67

Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…

5456850

…nto zoo_ernie_m

Yam0214 added 3 commits January 3, 2023 13:12

add logging eval infomations

f90c3ef

delete unused testing code

08adf0f

add tests samples for xnli

5a788f2

Yam0214 force-pushed the zoo_ernie_m branch from df6f791 to 5a788f2 Compare January 5, 2023 06:39

Merge branch 'develop' into zoo_ernie_m

d8eae11

wawltor reviewed Jan 5, 2023

View reviewed changes

model_zoo/ernie-m/README.md Show resolved Hide resolved

tests/fixtures/tests_samples/xnli/XNLI-1.0/README.md Outdated Show resolved Hide resolved

tests/fixtures/tests_samples/xnli/XNLI-1.0/xnli.dev.jsonl Outdated Show resolved Hide resolved

wawltor reviewed Jan 5, 2023

View reviewed changes

model_zoo/ernie-m/run_classifier.py Outdated Show resolved Hide resolved

model_zoo/ernie-m/run_classifier.py Show resolved Hide resolved

Yam0214 and others added 9 commits January 5, 2023 08:21

fewer tests samples for xnli

9857eed

add predict and export

4bf8875

add predictor and serving

292d74a

Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…

f6e2fae

…nto zoo_ernie_m

fix export error when training with data parallelism

342002d

fix mkdir error in distributed training and use ERNIEMHandler

764daa6

Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…

aaea2d5

…nto zoo_ernie_m

adjust predictor

d8e952b

Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…

c3d524a

…nto zoo_ernie_m

Yam0214 force-pushed the zoo_ernie_m branch from dce6a92 to 3d0e080 Compare January 31, 2023 11:21

modify ci accroding to PaddlePaddle#4398 and adjust predictor

bb48adb

Yam0214 force-pushed the zoo_ernie_m branch from 3d0e080 to bb48adb Compare January 31, 2023 12:54

Yam0214 added 3 commits February 3, 2023 11:27

enable load tiny dataset for ci

3aa1e7d

Merge branch 'develop' into zoo_ernie_m

26e975c

support testing infer with precision_mode fp16

9a1e8d2

wj-Mcat reviewed Feb 8, 2023

View reviewed changes

scripts/regression/ci_case.sh Outdated Show resolved Hide resolved

model_zoo/ernie-m/run_classifier.py Outdated Show resolved Hide resolved

Yam0214 added 3 commits February 9, 2023 02:48

modify ci script

58fdaf7

Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…

82856d8

…nto zoo_ernie_m

Update ci_case.sh

b258a8e

Yam0214 force-pushed the zoo_ernie_m branch from 12ad91d to b258a8e Compare February 9, 2023 07:40

enable to test inputs_embeds for enire-m

986a285

ZHUI self-requested a review February 9, 2023 08:12

Yam0214 added 2 commits February 9, 2023 09:11

Merge branch 'develop' of https://github.com/PaddlePaddle/PaddleNLP i…

08b0e29

…nto zoo_ernie_m

fix ci script

4e34df7

ZHUI reviewed Feb 10, 2023

View reviewed changes

fix using fast tokenizer

bcbf2ee

ZHUI reviewed Feb 13, 2023

View reviewed changes

using set_optimizer_grouped_parameters instead

4a0d661

Yam0214 force-pushed the zoo_ernie_m branch from c59c313 to 4a0d661 Compare February 13, 2023 10:08

Yam0214 added 2 commits February 17, 2023 06:45

consturct optimizer with layerwise_lr_decay out of Trainer

171fb53

merge and fix conflicts

03626da

ZHUI approved these changes Feb 20, 2023

View reviewed changes

joey12300 merged commit dd376ce into PaddlePaddle:develop Feb 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ModelZoo] Refactor ERNIE-M usage in Model Zoo #4324

[ModelZoo] Refactor ERNIE-M usage in Model Zoo #4324

Yam0214 commented Jan 3, 2023

paddle-bot bot commented Jan 3, 2023

codecov bot commented Jan 31, 2023 •

edited

ZHUI Feb 10, 2023

ZHUI Feb 10, 2023

ZHUI Feb 10, 2023

ZHUI Feb 10, 2023

ZHUI Feb 10, 2023

Yam0214 Feb 13, 2023 •

edited

ZHUI Feb 13, 2023

ZHUI Feb 13, 2023

ZHUI left a comment

[ModelZoo] Refactor ERNIE-M usage in Model Zoo #4324

[ModelZoo] Refactor ERNIE-M usage in Model Zoo #4324

Conversation

Yam0214 commented Jan 3, 2023

PR types

PR changes

Description

paddle-bot bot commented Jan 3, 2023

codecov bot commented Jan 31, 2023 • edited

Codecov Report

ZHUI Feb 10, 2023

Choose a reason for hiding this comment

ZHUI Feb 10, 2023

Choose a reason for hiding this comment

ZHUI Feb 10, 2023

Choose a reason for hiding this comment

ZHUI Feb 10, 2023

Choose a reason for hiding this comment

ZHUI Feb 10, 2023

Choose a reason for hiding this comment

Yam0214 Feb 13, 2023 • edited

Choose a reason for hiding this comment

ZHUI Feb 13, 2023

Choose a reason for hiding this comment

ZHUI Feb 13, 2023

Choose a reason for hiding this comment

ZHUI left a comment

Choose a reason for hiding this comment

codecov bot commented Jan 31, 2023 •

edited

Yam0214 Feb 13, 2023 •

edited