Baselines in Code Defect Detection task

[wasiahmad]

Hi,

The following two baseline methods are listed for the code defect (vulnerability) detection task (REF).

RoBERTa | 61.05
CodeBERT | 62.08

Which RoBERTa model is used here? Also, CodeBERT is trained on CodeSearchNet which doesn't have C++ functions. So, is CodeBERT directly fine-tuned on the Devign dataset? Please provide more information.

[guoday]

1. The paper of RoBERTa is here and we use RoBERTa base model. The architecture of RoBERTa is same as CodeBERT but Roberta is pre-trained on text. You can easily use RoBERTa model that we used by changing tokenizer_name and model_name_or_path from microsoft/codebert-base to roberta-base.

2. Yes, the CodeBERT is directly fine-tuned on the Devign dataset.

[wasiahmad]

Instead of RoBERTa, shouldn't RoBERTa (code) be used in the defect detection task? I saw RoBERTa (code) is used in the translation task. I want to confirm that RoBERTa is used in defect detection, instead of RoBERTa (code).

[guoday]

RoBERTa is used in defect detection. You are right and RoBERTa (code) should be used in the defect detection task. Maybe I can do the experiment later.