-
Notifications
You must be signed in to change notification settings - Fork 282
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to choose unlabelled data #26
Comments
Hi @Punchwes, there are two options for limiting the number of unlabeled examples:
For our experiments on AG's News, we chose the second option (that is, I hope this answers your question! |
Hi @timoschick , thanks for your quick reply. I think the method your describe in the paper corresponds to the second option, what confuses me is that in the code it seems that As your code comment in tasks.py:
and in the example loading part in cli.py:
there's no num_examples_per_label parameter passing to unlabeled_data loading. This is the reason why I am confused it seems that you would always choose the first option for unlabeled data.
and unlabeled data seems not be involved in the split_examples_evenly part as I could see. Or I missed something in the code where the |
Oh right, my mistake, you are absolutely correct!
Sorry for the confusion! |
Thanks very much for this clarification, very helpful and it makes sense to remove the option for unlabelled data. One last question I have is about seed. You mentioned in the paper that:
After checking the code, it seems that the seed parameter passed by command line (args.seed) is not used to choose data examples,
seed in the load_examples function is fixed as 42:
So I wonder when you run the model 3 times with different seeds, do you also change the seed in load_example() manually? |
For our experiments in Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference, we use the same set of examples for all three runs. The different seeds only affect the initialization of model parameters (for regular supervised training), dropout and the shuffling of training examples (i.e., the order in which they are presented to the model), which happens here. If you're interested in how different sets of training examples affect performance, you might find Table 6 in this paper useful. |
Thanks very much! |
Hi @timoschick, thanks very much for your work, I have a question about how you decide the unlabelled data for each task.
In the paper you say
Taking agnews as an example, I assume it means you take 40,000 examples (it has 4 classes in total) from training in total and there will be 10,000 examples for each class. However, in your code, it seems that you are not following the 10,000 examples per label thing by just shuffling and picking first 40,000 examples.
I am little bit confused about this, any clarification would be helpful.
The text was updated successfully, but these errors were encountered: