Some question about the Result

[ShomyLiu]

Hi,
Thanks for your released code. I have run the code and got the Accuracy is 0.779 and no F1-score(In fact the F1-score will be smaller about 4-5% than the accuracy); However the F1-score in the paper is almost 80-82% (except the WordNet lexcial features).
So I wonder that whether there are some tricks in the paper? And have you reach the result in the paper?

Thanks.

[FrankWork]

@ShomyLiu Using the official src/scorer.pl, I got f1 about 82 for 50d word embed, and 83.6 for 300d word embed. My results were saved in log directory. How do you get your f1 score?

[ShomyLiu]

@FrankWork Yeah, At the beginning, I just use the f1-metric in sklearn and the result is quite lower than the paper. Further, I use the scorer.pl to evaluate, it is right as you say: 82 for 50d
Maybe it is because the Other relation that causes the difference.
Thanks.

[JankinXu]

Hi, Frank. I run your code get the 82.4 with 50d, but i can't get the same F1 when I re-implemented your model with pytorch. My F1 is only 80 and My data processing is same as yours. I also try save your vector and run in my model and also can not get the same F1. I think the difference is in the weight initiation, but I can't find quite good weight initiation. Can I contact you and ask for some advice.

[ShomyLiu]

@JankinXu Hi, I have re-implemented the model with pytorch as well. However I can not reproduce the 82.4 F1-Score ; a little lower than yours, about 79%, which confused me a long time.
And my email is shomyliu@gmail.com, so feel free to contact me. If necessary, we can communicate via Wechat.

[FrankWork]

@ShomyLiu @JankinXu may i take a look at your code? do you use L2 normal regularization?

[ShomyLiu]

@FrankWork my model code is here https://paste.ubuntu.com/26385600/
And using Adam to optimze the CrossEntrop Loss.

[FrankWork]

@ShomyLiu do you normalize your pretrained word embedding? I didn't normalize it in my code. And make sure you add L2 regularization to your out_linear.

[ShomyLiu]

@FrankWork Yeah, I have tried to add L2 regularization only in out_linear layer, however, it does not improve the performance.... it's really confusing..

[FrankWork]

@ShomyLiu In my experience, L2 regularization in out_linear layer helps a lot.

[ShomyLiu]

@FrankWork Thanks. I use the following code in pytorch to add L2 regularization:

params=[]
for k, v in dict(model.named_parameters()).items():
      if k.startswith('out_linear'):
          param += [{'params': [v], 'lr':0.001, 'weight_decay': 0.01}]
      else:
          param += [ {'params': [v], 'lr': 0.001}]
optimizer = torch.optim.Adam(params=param)

[tangm00]

Hello, i want to know how to deal with the imbalance in SemEval-2010 task8 data? thank you.

[JankinXu]

@ShomyLiu hello, do you solve your problem, I ams sorry that i just see your message, I achieve 81.7% finally. I don't use L2 but add two dropout, and one is after embedding layer and one is before the last linear network.

[ShomyLiu]

@JankinXu Hi.
Two dropout layers indeed can accelerate convergence and improve performace.
At the begining, there is only one filter in my code, so the performance is lower. After using filters [3,4,5] and some fine tuning, the F1 score can be almost 82% (81.5% ~ 82.3%) in 50-d word embedding.