关于ontonote,weibo和resume数据集上实验结果问题

[gloria0108]

作者您好：
我用ontonote,weibo,resume,msr数据集（BMEOS模式）在您的原始代码（参数等均未改动）上跑了使用average strategy的实验，其中在msr数据集上能完全复现您论文汇报的结果。其余数据集上比您论文汇报的结果要低一些，请问是由于随机种子的原因还是需要额外的调参？
我跑出的实验结果如下：
ontonote:73.36（论文73.98）
weibo nm:63.76（论文64.17）
weibo all:57.52（论文58.67）
resume:94.21（论文95.03）
对应的log如下。谢谢！

log.train.note4.txt
log.train.resume.txt
log.train.weibo.nm.txt
log.train.weibo.all.txt

[liuwei1206]

hi，您好

可能是因为随机种子的原因，其他超参数应该不用调；事实上，我的超参数大部分是参考Lattice LSTM的，也没有经过微调。

不过，您说效果偏低，我表示很疑惑，因为后面代码是我跑过之后传上去的，效果大部分比论文还要好。但我是分开跑的，不知道是不是整合的时候出现了一些问题。

我看了你发的日志文件，Ontonote4里面差别挺大的，我将Ontonote4的average策略的代码上传到云盘，你可以再试一试！
链接：https://pan.baidu.com/s/1CXnUwvtPbkjMDUIIWNNU3w
提取码：gv2y

希望对比有帮助！

[liuwei1206]

另外，我还从github上直接clone下来代码，跑了一下weibo的实验，用average策略，我只跑了23大的epoch，效果就已经很明显了，下面是结果的截图：
Epoch: 19 training finished. Time: 139.70s, speed: 9.66st/s, total loss: 2160.899757385254
gold_num = 389 pred_num = 364 right_num = 235
Dev: time: 2.02s, speed: 134.20st/s; acc: 0.9595, p: 0.6456, r: 0.6041, f: 0.6242
Exceed previous best f score: 0.6231884057971016
gold_num = 418 pred_num = 357 right_num = 232
Test: time: 1.84s, speed: 147.24st/s; acc: 0.9558, p: 0.6499, r: 0.5550, f: 0.5987
Epoch: 20/50
Learning rate is setted as: 0.005377288836128128
Instance: 1350; Time: 142.40s; loss: 2099.8514; acc: 72292.0/73778.0=0.9799
Epoch: 20 training finished. Time: 142.40s, speed: 9.48st/s, total loss: 2099.851402282715
gold_num = 389 pred_num = 347 right_num = 221
Dev: time: 2.09s, speed: 129.88st/s; acc: 0.9586, p: 0.6369, r: 0.5681, f: 0.6005
gold_num = 418 pred_num = 330 right_num = 218
Test: time: 2.21s, speed: 122.77st/s; acc: 0.9549, p: 0.6606, r: 0.5215, f: 0.5829
Epoch: 21/50
Learning rate is setted as: 0.005108424394321722
Instance: 1350; Time: 139.88s; loss: 2050.2969; acc: 72306.0/73778.0=0.9800
Epoch: 21 training finished. Time: 139.88s, speed: 9.65st/s, total loss: 2050.2968673706055
gold_num = 389 pred_num = 339 right_num = 228
Dev: time: 2.05s, speed: 132.50st/s; acc: 0.9604, p: 0.6726, r: 0.5861, f: 0.6264
Exceed previous best f score: 0.6241699867197875
gold_num = 418 pred_num = 337 right_num = 224
Test: time: 1.85s, speed: 146.41st/s; acc: 0.9568, p: 0.6647, r: 0.5359, f: 0.5934
Epoch: 22/50
Learning rate is setted as: 0.004853003174605635
Instance: 1350; Time: 141.37s; loss: 2057.7770; acc: 72323.0/73778.0=0.9803
Epoch: 22 training finished. Time: 141.37s, speed: 9.55st/s, total loss: 2057.776954650879
gold_num = 389 pred_num = 347 right_num = 233
Dev: time: 2.14s, speed: 126.53st/s; acc: 0.9603, p: 0.6715, r: 0.5990, f: 0.6332
Exceed previous best f score: 0.6263736263736264
gold_num = 418 pred_num = 329 right_num = 225
Test: time: 2.04s, speed: 132.74st/s; acc: 0.9565, p: 0.6839, r: 0.5383, f: 0.6024
Epoch: 23/50
Learning rate is setted as: 0.004610353015875353
Instance: 1350; Time: 141.84s; loss: 1966.1277; acc: 72389.0/73778.0=0.9812
Epoch: 23 training finished. Time: 141.84s, speed: 9.52st/s, total loss: 1966.1277313232422
gold_num = 389 pred_num = 327 right_num = 218
Dev: time: 2.13s, speed: 127.30st/s; acc: 0.9596, p: 0.6667, r: 0.5604, f: 0.6089
gold_num = 418 pred_num = 314 right_num = 220
Test: time: 1.93s, speed: 140.35st/s; acc: 0.9565, p: 0.7006, r: 0.5263, f: 0.6011

[gloria0108]

好的，我再试一下，非常感谢您的耐心回复！
另外，请问您用github上clone下来的代码跑的这个weibo的实验是哪个数据集？all,nm还是ne？

[liuwei1206]

all的数据集

[gloria0108]

您好，抱歉再次打扰您，我在weibo.all数据集上还是没能重现出您论文汇报的结果。我对比了我的log日志和您的日志，发现我test数据集和您test数据集的gold_num不同，因此怀疑数据集存在差异。下面是我拿到的weibo.all数据集，请问是否方便分享一下您使用的weibo all数据集，以对比一下是否有哪些不同。非常感谢！
train.all.bmes.txt
dev.all.bmes.txt
test.all.bmes.txt

[liuwei1206]

我的weibo数据集已经传到github了呀

[gloria0108]

哦哦好的。抱歉，之前没注意。。。