Measure the model performance

[kk2491]

Hi,

Thanks for the simpler implementation of MAML.

As per the MAML paper At the end of meta-training, new tasks are sampled fromp(T),and meta-performance is measured by the model’s perfor-mance after learning fromKsamples. Generally, tasksused for meta-testing are held out during meta-training.

Anybody has tried fine-tuning the model with few number (0 to 10) of samples for a new class which was not there in the training dataset and measured the performance?

Is that part of the code already available in this repository?

Thank you,
KK

[HongduanTian]

Hi,
I also notice that there is no measure of the model performance. And in the code, there is another problem that the evaluation is performed on the test set, which is performed on val set in MAML.

Have you deal with such problem?

Thank you,
Tian

[kk2491]

@HongduanTian Yes, also it is happening every 500 episodes. Did you use original code from here. Does that have model performance part ?

Thank you,
KK

[HongduanTian]

Yep, but in MAML original code, you have to set --train=False, --test_set=True after finishing training the model so that you can measure the performance. Now I am trying to modify this pytorch version in sight of MAML.
If you have dealt with the problem, could you please share your version with me ?

Thank you.
Tian

[kk2491]

Yes. Sure. I will give a try and update you.

Thank you,
KK

[kk2491]

@HongduanTian So in the original code, were you able to fine-tune your trained model with less number of samples (lets say only 5 samples for that new class) ?

Thank you,
KK

[HongduanTian]

Yes, sure. In MAML, just modify the related parameters in main.py file and you can make it.

[kk2491]

@HongduanTian Thank you so much for the reply.

I am using this repository (dragen1860 pytorch implementation), trying to write a inference code to predict the new images using the saved model.

I am saving the complete model self.net in meta.py.

Added a condition in miniimagenet_train.py to train and test the model.

However I am not successful in predicting a image from the save model.

I am getting NotImplementedError.

Have you tried the similar approach in pytorch version?

Thank you,
KK

[HongduanTian]

Hi, Now I am also trying to modify this repository. And, I have successfully predicted the images. My reference codes are the same as that of evaluation in training, which just modify the number of the x_qry and y_qry pairs. I don't think it necessary to separate the two phase like MAML, and inference phase can be run after training phase completed in a single main.py file.  For saving model, I haven't tried it, and I will try in the future. Best, Tian

[kk2491]

Thank you for your response. And apologies for the delay in response.

Could you please tell me how did you predict the new images ?

Also lets say if my complete training dataset has 100 classes, and I am using n_way=4. In this case neural network model would be having only 4 output signals right? Kindly correct me if I am wrong.
How can we use this model to predict the images ?

Thank you,
KK

[HongduanTian]

In main function, after you have trained the model, the parameters in model are already well trained. Thus, generate the tasks from test dataset folder, put them into maml model, and predict the test data with maml.fine_tuning() will be okay.

For the 2nd question, yes, you're right. If n_way=4, then you will obtain a tensor with the size of ( ,4)output.

[kk2491]

In main function, after you have trained the model, the parameters in model are already well trained. Thus, generate the tasks from test dataset folder, put them into maml model, and predict the test data with maml.fine_tuning() will be okay.
After I train the model, I get a model with common theta right ? And when I do maml.fine_tuning() theta of the model would be tuned as per new task right ? Are we giving less number of samples in fine_tuning(). I see that even test_dataset contains huge number of samples. (Kindly correct me if I am wrong)

For the 2nd question, yes, you're right. If n_way=4, then you will obtain a tensor with the size of ( ,4)output.
I am just wondering, I trained a model with 100 classes. And I get a model with (n_way = 4) 4 outputs. How can I use this model for predicting the images?

Thank you,
KK

[HongduanTian]

Well, The theta after being trained is the model parameters which can adapt new tasks fast. You can view the theta as the optimal. When you fine-tuning the model on new tasks, the theta should only be changed during inner loop phase so that it can adapt the new task. But the initial theta for each new task is same and will not be changed after predicting the tasks.

In MAML, the test tasks number is 600, and I choose this number as well.

And I am confused with your words ''I trained a model with 100 classes. And I get a model with (n_way = 4) 4 outputs.'' Do you mean you train the model with 100 classes classification tasks and the output is 4? If you just trained the model with 4 classes classification tasks, the output of the network is 4. When you test your model, you just need to generate tasks from test dataset with the same structure(like 4 classes k shot) in training tasks, and images predictions can be performed.

English isn't my native language, so my explanations may sometimes confuse you or I misunderstand your questions. I am sorry for that if I don't express my answer clearly. If your questions are not well solved, you can ask me again.

Best,
Tian

[kk2491]

@HongduanTian Sorry for the delayed response.

Actually I understood the concept wrongly, you have explained it correctly.

So initial training is to get only the common theta model which can adapt to new classes. And in testing we pass only k samples to fine-tune the inner loop. And get new theta model which is capable to predicting classes in the new task.

And how about the query_set during the testing phase ? Do I need to have only k number of samples of a class or should I have in n_query number of samples for the same class ?

Thank you,
KK

[HongduanTian]

Tasks in testing phase have the same structures as those in training phase. So, in testing phase, query_set should be generated as what in training does. The optimal theta will adapt on support data in testing tasks and the performance will be measured on query data in testing tasks.

Just treat the task as data in common supervised learning. The structure of the 'data' should keep consistent.

[kk2491]

@HongduanTian Thank you again for your explanation and patience. I really appreciate that.
I am trying to understand the MAML architecture and how actually the code works.
I will be back with more doubts. Hope that is ok?

Thank you,
KK

[HongduanTian]

Yes, you can email me instead of mentioning me on github if you have questions on MAML. Hongduan Tian 邮箱：hongduan.tian@gmail.com Signature is customized by Netease Mail Master [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "#42?email_source=notifications\u0026email_token=AKIWXIPETM7TG6JZUUMHS7TQVTBQNA5CNFSM4JP7XHC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFE5P5A#issuecomment-558487540", "url": "#42?email_source=notifications\u0026email_token=AKIWXIPETM7TG6JZUUMHS7TQVTBQNA5CNFSM4JP7XHC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFE5P5A#issuecomment-558487540", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[zzpustc]

Tasks in testing phase have the same structures as those in training phase. So, in testing phase, query_set should be generated as what in training does. The optimal theta will adapt on support data in testing tasks and the performance will be measured on query data in testing tasks.

Just treat the task as data in common supervised learning. The structure of the 'data' should keep consistent.

@HongduanTian Hi, have you ever tried saving the model, and load model weights in an extra evaluate code? I have tried, I got ~47% accuracy when training with this repo(miniImagenet_train.py), but when I tried the operation mentioned above, I can only get ~44% accuracy. Can you provide some insights to me? Many thx!