Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to reproduce results of the paper? #96

Open
Devil-Ideal opened this issue Dec 10, 2022 · 1 comment
Open

How to reproduce results of the paper? #96

Devil-Ideal opened this issue Dec 10, 2022 · 1 comment

Comments

@Devil-Ideal
Copy link

I read the paper and downloaded the AG news dataset,and tested PET model on it,but there is a great margin between my results and the author's results. I set parameters as below. To be specifically, I just used 10 examples for train(10 shot),model type is roberta,model_name_or_path is roberta-large,and I used all patterns for AG news. I did not change other parameters. And here is my results:
==============my results===============
acc-p0: 0.6450877192982456 +- 0.053859095516898825
acc-p1: 0.7874561403508772 +- 0.01841603941808791
acc-p2: 0.5642543859649123 +- 0.06912621607498706
acc-p3: 0.6119298245614034 +- 0.09528808314997761
acc-p4: 0.7537719298245614 +- 0.07473549078343446
acc-all-p: 0.6725 +- 0.10462149351553651
===============parameters setting===========
parser.add_argument("--train_examples", default=10, type=int,
help="The total number of train examples to use, where -1 equals all examples.")
parser.add_argument("--method", required=False, default='pet', choices=['pet', 'ipet', 'sequence_classifier'],
help="The training method to use. Either regular sequence classification, PET or iPET.")
parser.add_argument("--data_dir", default="./agnews/", type=str, required=False,
help="The input data dir. Should contain the data files for the task.")
parser.add_argument("--model_type", default="roberta", type=str, required=False, choices=MODEL_CLASSES.keys(),
help="The type of the pretrained language model to use")
parser.add_argument("--model_name_or_path", default="roberta-large", type=str, required=False,
help="Path to the pre-trained model or shortcut name")
parser.add_argument("--task_name", default="agnews", type=str, required=False, choices=PROCESSORS.keys(),
help="The name of the task to train/evaluate on")
parser.add_argument("--output_dir", default="./output/", type=str, required=False,
help="The output directory where the model predictions and checkpoints will be written")

# PET-specific optional parameters
parser.add_argument("--wrapper_type", default="mlm", choices=WRAPPER_TYPES,
                    help="The wrapper type. Set this to 'mlm' for a masked language model like BERT or to 'plm' "
                         "for a permuted language model like XLNet (only for PET)")
parser.add_argument("--pattern_ids", default=[0,1,2,3,4], type=int, nargs='+',
                    help="The ids of the PVPs to be used (only for PET)")
@Devil-Ideal
Copy link
Author

Sorry,I missed the appendix. According to the details in appendixD, I retrain the model with some parameters changed,and here is the results, there is still a margin, but very close, maybe I still missed somethings.
acc-p0: 0.818859649122807 +- 0.02375440378086349
acc-p1: 0.8622368421052632 +- 0.011842105263157876
acc-p2: 0.8199122807017544 +- 0.044753157499628125
acc-p3: 0.7920175438596491 +- 0.01292178509375676
acc-p4: 0.8303508771929825 +- 0.032106970762756155
acc-all-p: 0.8246754385964913 +- 0.03328311578066454

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant