-
Notifications
You must be signed in to change notification settings - Fork 2.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add English Documents for UTC (#4476)
* [UTC] Add English documents * [UTC] Add English documents * [utc] update readmes * [utc] add xclue link
- Loading branch information
Showing
4 changed files
with
400 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
253 changes: 253 additions & 0 deletions
253
applications/zero_shot_text_classification/README_en.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,253 @@ | ||
[简体中文](README.md) | English | ||
|
||
# Zero-shot Text Classification | ||
|
||
**Table of contents** | ||
- [1. Zero-shot Text Classification Application](#1) | ||
- [2. Quick Start](#2) | ||
- [2.1 Code Structure](#21) | ||
- [2.2 Data Annotation](#22) | ||
- [2.3 Finetuning](#23) | ||
- [2.4 Evaluation](#24) | ||
- [2.5 Inference](#25) | ||
- [2.6 Deployment](#26) | ||
- [2.7 Experiments](#27) | ||
|
||
<a name="1"></a> | ||
|
||
## 1. Zero-shot Text Classification | ||
|
||
This project provides an end-to-end application solution for universal text classification based on Universal Task Classification (UTC) finetuning and goes through the full lifecycle of **data labeling, model training and model deployment**. We hope this guide can help you apply Text Classification techniques with zero-shot ability in your own products or models. | ||
|
||
<div align="center"> | ||
<img width="700" alt="UTC模型结构图" src="https://user-images.githubusercontent.com/25607475/212268807-66181bcb-d3f9-4086-9d4a-de4d1d0933c2.png"> | ||
</div> | ||
|
||
Text Classification refers to assigning a set of categories to given input text. Despite the advantages of tuning, applying text classification techniques in practice remains a challenge due to domain adaption and lack of labeled data, etc. This PaddleNLP Zero-shot Text Classification Guide builds on our UTC from the Unified Semantic Matching (USM) model series and provides an industrial-level solution that supports universal text classification tasks, including but not limited to **semantic analysis, semantic matching, intention recognition and event detection**, allowing you accomplish multiple tasks with a single model. Besides, our method brings good generation performance through multi-task pretraining. | ||
|
||
**Highlights:** | ||
|
||
- **Comprehensive Coverage**🎓: Covers various mainstream tasks of text classification, including but not limited to semantic analysis, semantic matching, intention recognition and event detection. | ||
|
||
- **State-of-the-Art Performance**🏃: Strong performance from the UTC model, which ranks first on [ZeroCLUE](https://www.cluebenchmarks.com/zeroclue.html)/[FewCLUE](https://www.cluebenchmarks.com/fewclue.html) as of 01/11/2023. | ||
|
||
- **Easy to use**⚡: Three lines of code to use our Taskflow for out-of-box Zero-shot Text Classification capability. One line of command to model training and model deployment. | ||
|
||
- **Efficient Tuning**✊: Developers can easily get started with the data labeling and model training process without a background in Machine Learning. | ||
|
||
<a name="2"></a> | ||
|
||
## 2. Quick start | ||
|
||
For quick start, you can directly use ```paddlenlp.Taskflow``` out-of-the-box, leveraging the zero-shot performance. For production use cases, we recommend labeling a small amount of data for model fine-tuning to further improve the performance. | ||
|
||
<a name="21"></a> | ||
|
||
### 2.1 Code structure | ||
|
||
```shell | ||
. | ||
├── deploy/simple_serving/ # model deployment script | ||
├── utils.py # data processing tools | ||
├── run_train.py # model fine-tuning script | ||
├── run_eval.py # model evaluation script | ||
├── label_studio.py # data format conversion script | ||
├── label_studio_text.md # data annotation instruction | ||
└── README.md | ||
``` | ||
<a name="22"></a> | ||
|
||
### 2.2 Data labeling | ||
|
||
We recommend using [Label Studio](https://labelstud.io/) for data labeling. You can export labeled data in Label Studio and convert them into the required input format. Please refer to [Label Studio Data Labeling Guide](./label_studio_text_en.md) for more details. | ||
|
||
Here we provide a pre-labeled example dataset `Medical Question Intent Classification Dataset`, which you can download with the following command. We will show how to use the data conversion script to generate training/validation/test set files for fine-tuning. | ||
|
||
Download the medical question intent classification dataset: | ||
|
||
```shell | ||
wget https://bj.bcebos.com/paddlenlp/datasets/utc-medical.tar.gz | ||
tar -xvf utc-medical.tar.gz | ||
mv utc-medical data | ||
rm utc-medical.tar.gz | ||
``` | ||
|
||
Generate training/validation set files: | ||
|
||
```shell | ||
python label_studio.py \ | ||
--label_studio_file ./data/label_studio.json \ | ||
--save_dir ./data \ | ||
--splits 0.8 0.1 0.1 \ | ||
--options ./data/label.txt | ||
``` | ||
|
||
For multi-task training, you can convert data with script seperately and move them to the same directory. | ||
|
||
<a name="23"></a> | ||
|
||
### 2.3 Finetuning | ||
|
||
Use the following command to fine-tune the model using `utc-large` as the pre-trained model, and save the fine-tuned model to `./checkpoint/model_best/`: | ||
|
||
Single GPU: | ||
|
||
```shell | ||
python run_train.py \ | ||
--device gpu \ | ||
--logging_steps 10 \ | ||
--save_steps 100 \ | ||
--eval_steps 100 \ | ||
--seed 1000 \ | ||
--model_name_or_path utc-large \ | ||
--output_dir ./checkpoint/model_best \ | ||
--dataset_path ./data/ \ | ||
--max_seq_length 512 \ | ||
--per_device_train_batch_size 2 \ | ||
--per_device_eval_batch_size 2 \ | ||
--gradient_accumulation_steps 8 \ | ||
--num_train_epochs 20 \ | ||
--learning_rate 1e-5 \ | ||
--do_train \ | ||
--do_eval \ | ||
--do_export \ | ||
--export_model_dir ./checkpoint/model_best \ | ||
--overwrite_output_dir \ | ||
--disable_tqdm True \ | ||
--metric_for_best_model macro_f1 \ | ||
--load_best_model_at_end True \ | ||
--save_total_limit 1 | ||
``` | ||
|
||
Multiple GPUs: | ||
|
||
```shell | ||
python -u -m paddle.distributed.launch --gpus "0,1" run_train.py \ | ||
--device gpu \ | ||
--logging_steps 10 \ | ||
--save_steps 100 \ | ||
--eval_steps 100 \ | ||
--seed 1000 \ | ||
--model_name_or_path utc-large \ | ||
--output_dir ./checkpoint/model_best \ | ||
--dataset_path ./data/ \ | ||
--max_seq_length 512 \ | ||
--per_device_train_batch_size 2 \ | ||
--per_device_eval_batch_size 2 \ | ||
--gradient_accumulation_steps 8 \ | ||
--num_train_epochs 20 \ | ||
--learning_rate 1e-5 \ | ||
--do_train \ | ||
--do_eval \ | ||
--do_export \ | ||
--export_model_dir ./checkpoint/model_best \ | ||
--overwrite_output_dir \ | ||
--disable_tqdm True \ | ||
--metric_for_best_model macro_f1 \ | ||
--load_best_model_at_end True \ | ||
--save_total_limit 1 | ||
``` | ||
|
||
Parameters: | ||
|
||
* `device`: Training device, one of 'cpu' and 'gpu' can be selected; the default is GPU training. | ||
* `logging_steps`: The interval steps of log printing during training, the default is 10. | ||
* `save_steps`: The number of interval steps to save the model checkpoint during training, the default is 100. | ||
* `eval_steps`: The number of interval steps to save the model checkpoint during training, the default is 100. | ||
* `seed`: global random seed, default is 42. | ||
* `model_name_or_path`: The pre-trained model used for few shot training. Defaults to "utc-large". | ||
* `output_dir`: Required, the model directory saved after model training or compression; the default is `None`. | ||
* `dataset_path`: The directory to dataset; defaults to `./data`. | ||
* `train_file`: Training file name; defaults to `train.txt`. | ||
* `dev_file`: Development file name; defaults to `dev.txt`. | ||
* `max_seq_len`: The maximum segmentation length of the text and label candidates. When the input exceeds the maximum length, the input text will be automatically segmented. The default is 512. | ||
* `per_device_train_batch_size`: The batch size of each GPU core/CPU used for training, the default is 8. | ||
* `per_device_eval_batch_size`: Batch size per GPU core/CPU for evaluation, default is 8. | ||
* `num_train_epochs`: Training rounds, 100 can be selected when using early stopping method; the default is 10. | ||
* `learning_rate`: The maximum learning rate for training, UTC recommends setting it to 1e-5; the default value is 3e-5. | ||
* `do_train`: Whether to perform fine-tuning training, setting this parameter means to perform fine-tuning training, and it is not set by default. | ||
* `do_eval`: Whether to evaluate, setting this parameter means to evaluate, the default is not set. | ||
* `do_export`: Whether to export, setting this parameter means to export static graph, and it is not set by default. | ||
* `export_model_dir`: Static map export address, the default is `./checkpoint/model_best`. | ||
* `overwrite_output_dir`: If `True`, overwrite the contents of the output directory. If `output_dir` points to a checkpoint directory, use it to continue training. | ||
* `disable_tqdm`: Whether to use tqdm progress bar. | ||
* `metric_for_best_model`: Optimal model metric, UTC recommends setting it to `macro_f1`, the default is None. | ||
* `load_best_model_at_end`: Whether to load the best model after training, usually used in conjunction with `metric_for_best_model`, the default is False. | ||
* `save_total_limit`: If this parameter is set, the total number of checkpoints will be limited. Remove old checkpoints `output directory`, defaults to None. | ||
|
||
<a name="24"></a> | ||
|
||
### 2.4 Evaluation | ||
|
||
Model evaluation: | ||
|
||
```shell | ||
python evaluate.py \ | ||
--model_path ./checkpoint/model_best \ | ||
--test_path ./data/test.txt \ | ||
--per_device_eval_batch_size 2 \ | ||
--max_seq_len 512 \ | ||
--output_dir ./checkpoint_test | ||
``` | ||
|
||
Parameters: | ||
|
||
- `model_path`: The path of the model folder for evaluation, which must contain the model weight file `model_state.pdparams` and the configuration file `model_config.json`. | ||
- `test_path`: The test set file for evaluation. | ||
- `per_device_eval_batch_size`: Batch size, please adjust it according to the machine situation, the default is 8. | ||
- `max_seq_len`: The maximum segmentation length of the text and label candidates. When the input exceeds the maximum length, the input text will be automatically segmented. The default is 512. | ||
|
||
<a name="25"></a> | ||
|
||
### 2.5 Inference | ||
|
||
You can use `paddlenlp.Taskflow` to load your custom model by specifying the path of the model weight file through `task_path`. | ||
|
||
```python | ||
>>> from pprint import pprint | ||
>>> from paddlenlp import Taskflow | ||
>>> schema = ["病情诊断", "治疗方案", "病因分析", "指标解读", "就医建议", "疾病表述", "后果表述", "注意事项", "功效作用", "医疗费用", "其他"] | ||
>>> my_cls = Taskflow("zero_shot_text_classification", schema=schema, task_path='./checkpoint/model_best', precision="fp16") | ||
>>> pprint(my_cls("中性粒细胞比率偏低")) | ||
``` | ||
|
||
<a name="26"></a> | ||
|
||
### 2.6 Deployment | ||
|
||
We provide the deployment solution on the foundation of PaddleNLP SimpleServing, where you can easily build your own deployment service with three-line code. | ||
|
||
``` | ||
# Save at server.py | ||
from paddlenlp import SimpleServer, Taskflow | ||
schema = ["病情诊断", "治疗方案", "病因分析", "指标解读", "就医建议"] | ||
utc = Taskflow("zero_shot_text_classification", | ||
schema=schema, | ||
task_path="../../checkpoint/model_best/", | ||
precision="fp32") | ||
app = SimpleServer() | ||
app.register_taskflow("taskflow/utc", utc) | ||
``` | ||
|
||
``` | ||
# Start the server | ||
paddlenlp server server:app --host 0.0.0.0 --port 8990 | ||
``` | ||
|
||
It supports FP16 (half-precision) and multiple process for inference acceleration. | ||
|
||
<a name="27"></a> | ||
|
||
### 2.7 Experiments | ||
|
||
The results reported here are based on the development set of KUAKE-QIC. | ||
|
||
| | Accuracy | Micro F1 | Macro F1 | | ||
| :------: | :--------: | :--------: | :--------: | | ||
| 0-shot | 28.69 | 87.03 | 60.90 | | ||
| 5-shot | 64.75 | 93.34 | 80.33 | | ||
| 10-shot | 65.88 | 93.76 | 81.34 | | ||
| full-set | 81.81 | 96.65 | 89.87 | | ||
|
||
where k-shot means that there are k annotated samples per label for training. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.