999 distractor snippets

[ncshrikanth]

Thank You for CodeBERT! A quick question, wherein CodeSearchNet did you find a fixed set
of 999 distractor codes say If I wish to evaluate (MRR) for a model built (Code Search space)?

[fengzhangyin]

Yes. We follow the official evaluation metric to calculate the Mean Reciprocal Rank (MRR) for each pair of test data over a fixed set of 999 distractor codes.

[ncshrikanth]

Thank You for your reply! Perhaps I was not clear but I was asking for the source ie., "Where is the fixed set of 999 distractor codes"? Could you share the link or dataset?

[fengzhangyin]

You can use the script we provide to get the test data.
The script link is https://github.com/microsoft/CodeBERT/blob/master/codesearch/process_data.py.

[ncshrikanth]

Thank You for the quick reply, will try this!