使用transformer构建的机器翻译系统
介绍:
本项目为语言分析与机器翻译作业,
数据集:iwslt’14 de-en
训练模型及预处理后训练集下载地址:https://www.aliyundrive.com/s/d9PJMWNQh8Z
运行环境:
pytorch_gpu:1.10
nltk
目录结构:
├── data
│ ├── 2k #bpe大小为2k的数据集
│ ├── 5k #bpe大小为5k的数据集
│ ├── de.txt #对原始的数据集进行预处理
│ ├── en.txt
│ └── origin_data_set #原始的数据集
├── model
│ └── Transformer.py #Transformer模型文件
├── save #训练结果过程和训练结果保存目录
├── tool
│ ├── DataTool.py #数据载入工具类
│ ├── Global.py #Transformer模型参数及 数据集目录配置
│ └── TrainTool.py # Warm-up函数
├── 数据处理 #以下三个文件,按顺序执行,预处理数据
│ ├── S0_修改原始数据格式.py
│ ├── S1_子词切分.py
│ └── S2_数据集生成.py
├── 测试.py #模型参数
└── 训练.py #模型训练
实现部分:
使用pytorch框架搭建Transformer模型(Pre-Norm结构)
实现BPE子词切分算法
实现Beam Search算法
对模型参数(超参数)进行设置、Warm-up、学习率初始化等
对实验结果计算BLEU得分
-
配置模型参数
vim Globay.py
-
修改train.py
init_model_number=0 #表示从头开始训练
-
开始训练
python 训练.py
-
配置模型参数,确保与训练时的模型参数一致
vim Globay.py
-
修改train.py
init_model_number=10 #表示载入epoch=10的模型文件,继续训练 model_path = './save/de2en_5k_%04d.pt' % init_model_number #de2en_5k为模型名 step = 2182 * init_model_number # 2182为一轮的batch数量
-
开始执行
python 训练.py
此处训练参数设置为3层,4头,256维度,bpe大小为2k
gpu模式
载入已有字典
encoder_chars: 2004
decoder_chars: 2004
max_enc_seq_length: 363
max_dec_seq_length: 278
数据集大小:172530
数据集大小:872
batch num:1120
Epoch:0001 prog:100.0017% batch:1120/1120 batch_size:180 mean_loss=5.996997 mean_accu=11.41% lr=0.000098
进行测试中...batch num:44
valid accuracy:22.38 % test loss:4.4169
......
......
batch num:1120
Epoch:0020 prog:100.0017% batch:1120/1120 batch_size:200 mean_loss=2.138918 mean_accu=68.88% lr=0.000418
进行测试中...batch num:44
valid accuracy:70.41 % test loss:1.4048
cat save/de2en_2k.txt
n_layers3 n_heads:4 d_model:256 d_ff:1024 batch_size:20 encoder_len:363 decoder_len:278
Epoch:0001 batch:1120 loss=5.996997 train_accu=11.412931 valid_accu=22.383728 lr=0.000098
Epoch:0002 batch:1120 loss=4.755715 train_accu=24.517178 valid_accu=36.618428 lr=0.000196
Epoch:0003 batch:1120 loss=3.974497 train_accu=35.328851 valid_accu=45.392770 lr=0.000293
Epoch:0004 batch:1120 loss=3.470239 train_accu=43.350113 valid_accu=52.148458 lr=0.000391
Epoch:0005 batch:1120 loss=3.135638 train_accu=49.315405 valid_accu=56.918749 lr=0.000489
Epoch:0006 batch:1120 loss=2.912801 train_accu=53.593547 valid_accu=59.448717 lr=0.000587
Epoch:0007 batch:1120 loss=2.763752 train_accu=56.515822 valid_accu=62.279953 lr=0.000685
Epoch:0008 batch:1120 loss=2.642074 train_accu=58.967584 valid_accu=63.859193 lr=0.000660
Epoch:0009 batch:1120 loss=2.534271 train_accu=61.111251 valid_accu=65.178845 lr=0.000623
Epoch:0010 batch:1120 loss=2.453718 train_accu=62.700495 valid_accu=66.578826 lr=0.000591
Epoch:0011 batch:1120 loss=2.393994 train_accu=63.887431 valid_accu=67.546647 lr=0.000563
Epoch:0012 batch:1120 loss=2.344394 train_accu=64.858695 valid_accu=67.881313 lr=0.000539
Epoch:0013 batch:1120 loss=2.303971 train_accu=65.646550 valid_accu=68.559155 lr=0.000518
Epoch:0014 batch:1120 loss=2.270668 train_accu=66.321382 valid_accu=68.881203 lr=0.000499
Epoch:0015 batch:1120 loss=2.240943 train_accu=66.875999 valid_accu=69.428752 lr=0.000482
Epoch:0016 batch:1120 loss=2.216345 train_accu=67.376881 valid_accu=69.645483 lr=0.000467
Epoch:0017 batch:1120 loss=2.193870 train_accu=67.818207 valid_accu=69.951239 lr=0.000453
Epoch:0018 batch:1120 loss=2.173374 train_accu=68.200069 valid_accu=70.102368 lr=0.000440
Epoch:0019 batch:1120 loss=2.155479 train_accu=68.574355 valid_accu=70.356942 lr=0.000428
Epoch:0020 batch:1120 loss=2.138918 train_accu=68.877916 valid_accu=70.409328 lr=0.000418
-
配置模型参数,确保与训练时的模型参数一致
vim Globay.py
-
修改测试.py
model = Transformer(len(encoder_chars), len(decoder_chars), d_model, d_ff, num_layers, num_heads, device, 0, 0, 0.1) m_state_dict = torch.load('./save/de2en_2k_0020.pt', map_location="cuda:{}".format(map_gpu_index)) model.load_state_dict(m_state_dict) k = 3
de2en_2k_0020.pt为模型文件
map_gpu_index设置使用的gpu序号
k为书搜索宽度
-
运行测试
python 测试.py 原文:nun , das bringt ihnen keinen doktortitel in harvard , aber es ist viel interessanter als stamina zu zählen . 参考译文:now , that 's not going to get you a ph.d. at harvard , but it 's a lot more interesting than counting stamens . -12.349,now , it doesn 't get you a doctor in harvard , but it 's much more interesting to count than stamina . -12.397,now , this doesn 't get you a doctor in harvard , but it 's much more interesting to count than stamina . -13.909,now , it doesn 't get you a doctor in harvard , but it 's much more interesting to count as a stamina . ...... ...... 原文:es war auf jeden fall eine herausforderung . 参考译文:so , certainly the challenge was there . -4.524,it was a challenge anyway . -2.915,it was certainly a challenge . -3.620,it was definitely a challenge . 原文:wir haben bis jetzt wirklich nicht die richtigen worte , sie zu beschreiben . 参考译文:we don 't have words , really , to describe it yet . -4.718,we really don 't have the right words to describe them . -5.074,we really don 't have the right words to describe it . -5.088,we don 't really have the right words to describe them . 原文:ich spreche oft darüber , situationen zu " entdenken " . 参考译文:i talk about unthinking situations all the time . -2.874,i often talk about finding situations . -2.882,i often talk about discovering situations . -4.159,i 'm often talking about discovering situations . bleu:0.5828,0.3548,0.2298,0.1537 mean bleu:0.3303