-
Notifications
You must be signed in to change notification settings - Fork 421
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Measure the model performance #42
Comments
Hi, Have you deal with such problem? Thank you, |
@HongduanTian Yes, also it is happening every 500 episodes. Did you use original code from here. Does that have model performance part ? Thank you, |
Yep, but in MAML original code, you have to set --train=False, --test_set=True after finishing training the model so that you can measure the performance. Now I am trying to modify this pytorch version in sight of MAML. Thank you. |
Yes. Sure. I will give a try and update you. Thank you, |
@HongduanTian So in the original code, were you able to fine-tune your trained model with less number of samples (lets say only 5 samples for that new class) ? Thank you, |
Yes, sure. In MAML, just modify the related parameters in main.py file and you can make it.
|
@HongduanTian Thank you so much for the reply. I am using this repository (dragen1860 pytorch implementation), trying to write a inference code to predict the new images using the saved model. I am saving the complete model Have you tried the similar approach in pytorch version? Thank you, |
Hi,
Now I am also trying to modify this repository. And, I have successfully predicted the images. My reference codes are the same as that of evaluation in training, which just modify the number of the x_qry and y_qry pairs. I don't think it necessary to separate the two phase like MAML, and inference phase can be run after training phase completed in a single main.py file.
For saving model, I haven't tried it, and I will try in the future.
Best,
Tian
|
Thank you for your response. And apologies for the delay in response. Could you please tell me how did you predict the new images ? Also lets say if my complete training dataset has 100 classes, and I am using Thank you, |
In main function, after you have trained the model, the parameters in model are already well trained. Thus, generate the tasks from test dataset folder, put them into maml model, and predict the test data with For the 2nd question, yes, you're right. If |
In main function, after you have trained the model, the parameters in model are already well trained. Thus, generate the tasks from test dataset folder, put them into maml model, and predict the test data with maml.fine_tuning() will be okay. For the 2nd question, yes, you're right. If n_way=4, then you will obtain a tensor with the size of ( ,4)output. Thank you, |
Well, The theta after being trained is the model parameters which can adapt new tasks fast. You can view the In MAML, the test tasks number is 600, and I choose this number as well. And I am confused with your words ''I trained a model with 100 classes. And I get a model with (n_way = 4) 4 outputs.'' Do you mean you train the model with 100 classes classification tasks and the output is 4? If you just trained the model with 4 classes classification tasks, the output of the network is 4. When you test your model, you just need to generate tasks from test dataset with the same structure(like 4 classes k shot) in training tasks, and images predictions can be performed. English isn't my native language, so my explanations may sometimes confuse you or I misunderstand your questions. I am sorry for that if I don't express my answer clearly. If your questions are not well solved, you can ask me again. Best, |
@HongduanTian Sorry for the delayed response. Actually I understood the concept wrongly, you have explained it correctly. So initial training is to get only the common theta model which can adapt to new classes. And in testing we pass only k samples to fine-tune the inner loop. And get new theta model which is capable to predicting classes in the new task. And how about the query_set during the testing phase ? Do I need to have only k number of samples of a class or should I have in n_query number of samples for the same class ? Thank you, |
Tasks in testing phase have the same structures as those in training phase. So, in testing phase, query_set should be generated as what in training does. The optimal theta will adapt on support data in testing tasks and the performance will be measured on query data in testing tasks. Just treat the task as data in common supervised learning. The structure of the 'data' should keep consistent. |
@HongduanTian Thank you again for your explanation and patience. I really appreciate that. Thank you, |
Yes, you can email me instead of mentioning me on github if you have questions on MAML.
Hongduan Tian
邮箱:hongduan.tian@gmail.com
Signature is customized by Netease Mail Master
[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "#42?email_source=notifications\u0026email_token=AKIWXIPETM7TG6JZUUMHS7TQVTBQNA5CNFSM4JP7XHC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFE5P5A#issuecomment-558487540",
"url": "#42?email_source=notifications\u0026email_token=AKIWXIPETM7TG6JZUUMHS7TQVTBQNA5CNFSM4JP7XHC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFE5P5A#issuecomment-558487540",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]
|
@HongduanTian Hi, have you ever tried saving the model, and load model weights in an extra evaluate code? I have tried, I got ~47% accuracy when training with this repo(miniImagenet_train.py), but when I tried the operation mentioned above, I can only get ~44% accuracy. Can you provide some insights to me? Many thx! |
Hi,
Thanks for the simpler implementation of MAML.
As per the MAML paper
At the end of meta-training, new tasks are sampled fromp(T),and meta-performance is measured by the model’s perfor-mance after learning fromKsamples. Generally, tasksused for meta-testing are held out during meta-training.
Anybody has tried fine-tuning the model with few number (0 to 10) of samples for a new class which was not there in the training dataset and measured the performance?
Is that part of the code already available in this repository?
Thank you,
KK
The text was updated successfully, but these errors were encountered: