a relationship extraction issue

[karlhugle]

for this list https://github.com/sebastianruder/NLP-progress/blob/master/english/relationship_extraction.md

I would like to point out a data issue

a new model of Distantly Supervised Relationship Extraction using the same training dataset (522611 ) is be able to compare with the same results of models (PCNN+ATT, PCNN+ONE etc.) reported Lin's paper (Lin et al., 2016).
(the cleaned dataset was updated by Lin and could be downloaded from https://github.com/thunlp/NRE)

The problem is that, some new papers (e.g. two in EMNLP 2018 and one in AAAI2019) ) used the unprocessed data (570088), which contains duplicated instances in the test set. the unclean data will give higher unreliable results.

issues already have been discussed in
thunlp/NRE#16
thunlp/OpenNRE#27

the unclean data was tested and has effects on the results.

[sebastianruder]

Thanks for this note! Could you add a note to the relevant section and indicate with a symbol the methods that use a different setup?

[weilonghu]

I also noticed this problem and emailed some authors, but they have different opinions on this issue.