Not able to reproduce the results

[fufrank5]

I changed the codes to be compatible with TF1.3 and run the training and test.
The performance I got is lower than reported. Any clue?

Evaluating P@N for iter 11000
Evaluating P@N for one
P@100:
0.73
P@200:
0.645
P@300:
0.6
Evaluating P@N for two
P@100:
0.77
P@200:
0.69
P@300:
0.636666666667
Evaluating P@N for all
P@100:
0.77
P@200:
0.695
P@300:
0.66
2017-10-24T19:14:49.392740
Evaluating all test data and save data for PR curve
saving all test result...
PR curve area:0.347798385666
2017-10-24T19:17:38.815224
P@N for all test data:
P@100:
0.78
P@200:
0.72
P@300:
0.69

Thanks,
Lisheng

[rishabhjoshi]

Were you able to reproduce the results?

[fufrank5]

@rishabhjoshi NO. not by these codes.

[rishabhjoshi]

Can you share an updated code? Or tell me what changes you had to make? What hyperparameters you had to set? Did you need to make any changes to the architecture itself?
Thanks

[fufrank5]

I was using TF0.11 to run their codes without change

[pvthuy]

My results are similar. I couldn't reproduce the results shown in this repository for BGRU+2ATT.

Evaluating P@N for iter 17500
Evaluating P@N for one
P@100:
0.71
P@200:
0.685
P@300:
0.623333333333
Evaluating P@N for two
P@100:
0.72
P@200:
0.665
P@300:
0.663333333333
Evaluating P@N for all
P@100:
0.75
P@200:
0.67
P@300:
0.686666666667
Evaluating all test data and save data for PR curve
saving all test result...
PR curve area:0.34952666998054976
2018-06-04T10:03:39.969078
P@N for all test data:
P@100:
0.83
P@200:
0.745
P@300:
0.693333333333

I use the original code with tf r0.11.
Can anyone reproduce the results of BGRU+2ATT?

[jufengada]

hello～ I was using tf1.8 to run this code, and I get some wrong.

[pvthuy]

This code requires tf r0.11.

To use tf 1.x: https://github.com/frankxu2004/TensorFlow-NRE
(I tried with this code but very slow, don't know why??)

BTW, what kind of errors did you get?

[THUCSTHanxu13]

We have reconstructed the code and provide some training examples. You can try them again...

[fufrank5]

@THUCSTHanxu13 Thanks for effort to update the repository.
However, it seems the codes are only working with model pcnn_att. I've tried several others. All of them are reporting syntax errors.
BTW, there seems to be no validation set in this setup. Are you just reporting the best epoch on the test set directly in both test.py and in the original paper?

Thanks.

[fufrank5]

@THUCSTHanxu13 BTW, the number of examples in train.txt is 570088, which is different from 522611 from the paper and https://github.com/thunlp/NRE. Any reason?
Thanks.

[gaotianyu1350]

@fufrank5 Thanks for your feedback. The syntax errors will be fixed soon. As for the dataset, there are some repeat sentences in it so we have deleted them.