Zero-shot Tacred Relation Classification

[mahrahimi1]

For your zero-shot Tacred relation classification results with 1% dev (see the above picture), the paper says you did 100 runs. Did you use the following set as the 1% dev set for all 100 runs? If not, how did you create 100 1% dev sets?

Ask2Transformers/resources/tacred_splits/dev/dev.0.01.split.txt

[osainz59]

Hi @mahrahimi1 ,
To obtain the results showed in that table we performed an stratified kfold on the development set, with k being 100. As they are zero-shot results, i.e. no training data is used, the only thing that changes across runs is the development set used for threshold-estimation. You can find the script I used to generate the 100 folds here, however it should give you some errors as the code has changed drastically since then.

All the code from /scripts and /a2t/legacy is outdated and only exists for reproducibility purposes.

[mahrahimi1]

Thank you very much!