Is there any difference between 5-way 1-shot and 5-way 5-shot in the code?

[yang-yk]

Hi, Shtok, sorry to bother you again.

When I run the code of 5-way 1-shot, the experiment results are close to what the paper reports. However, when I run the code with the command 'python fpn/few_shot_benchmark.py --test_name=RepMet_inloc --Nshot=5 --Nway=5 --Nquery_cat=10 --Nepisodes=500' (5-way 5-shot), the AP is much smaller (48.3%) than what the paper reports (68.8%). I wonder why it happens. Can you help me to solve this problem. Thanks very much!

Sincerely,
Yukuan Yang

[yang-yk]

@duynn912 Do you have any idea about my questions above. Thanks!

[duynn912]

Hi @yang-yk,

I also got a result like you for 5 way 1 shot. Actually, I did not experiment 5 way 5 shot so I did not know exactly how many accuracy is. However, according to the paper, when we experiment 5 way 1 shot, it means we have one image from each class, totally 5 images for 5 classes. And you have 5 way 5 shot, you will have 5 images from each class of 5 classes, totally 25 images for 5 classes. At this time, you have more images for your experiment, I think it will have a higher accuracy in comparison to 5 way 1 shot and it will be similar to the paper. So, I was also wondering why it is?

[jshtok]

Hi Yukuyan, I need to run the test to reconstruct the problem you are describing, and I will get back to you. Regards, Joseph

…

On Sun, Jan 19, 2020 at 2:01 PM Yukuan Yang ***@***.***> wrote: Hi, Shtok, sorry to bother you again. When I run the code of 5-way 1-shot, the experiment results are close to what the paper reports. However, when I run the code with the command 'python fpn/few_shot_benchmark.py --test_name=RepMet_inloc --Nshot=5 --Nway=5 --Nquery_cat=10 --Nepisodes=500' (5-way 5-shot), the AP is much smaller (48.3%) than what the paper reports (68.8%). I wonder why it happens. Can you help me to solve this problem. Thanks very much! Sincerely, Yukuan Yang — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#22?email_source=notifications&email_token=ACOBU6VZGT2TWMD5V4VVO4TQ6Q6JLA5CNFSM4KIY2XK2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IHFQPBQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACOBU6QVFCWSF4HDSWAOUVLQ6Q6JLANCNFSM4KIY2XKQ> .

[jshtok]

Hi @yang-yk ,
I just executed the line
python fpn/few_shot_benchmark.py --test_name=RepMet_inloc --Nshot=5 --Nway=5 --Nquery_cat=10 --Nepisodes=500

and I am getting good results (see .log file below).
Sorry I couldn't reconstruct your error.

Regards,
Joseph

few_shot_benchmark_RepMet_inloc_5shot_5way_10qpc_500epi_aug_0%3A10_lr_1.0e-04_nms%3A0.30_iou%3A0.70_clust%3A1_exact%3A1_2020-01-27-15-05.log

[yang-yk]

Hi @jshtok ,
Thanks for your reply.
I think the reason why I have met the problem lies in the episode file generated by the code after comparing your log file. My log file and episode file is as follows. I think my episode file (.npz) is different from yours. I wonder whether this is caused by the random seed setting or the NumPy version (1.16.5).

To be clear, I use the original code from Github and the original model parameters (downloaded from the google drive). Can you give me your 1-shot 5-shot and 10-shot episode file (.npz file in the output directory).

Thanks very much and hope to hear from you soon!

Best regards,
Yukuan Yang

5-shot result.zip

[jshtok]

Hi Yukuan, Sorry for the late reply. Please see the file attached:

…

On Sat, Feb 8, 2020 at 5:15 PM Yukuan Yang ***@***.***> wrote: Hi @jshtok <https://github.com/jshtok> , Thanks for your reply. I think the reason why I have met the problem lies in the episode file generated by the code after comparing your log file. My log file and episode file is as follows. I think my episode file (.npz) is different from yours. I wonder whether this is caused by the random seed setting or the numpy version (1.16.5). To be clear, I use the original code from github and the original model parameters (downloaded from the google drive). Can you give me your 5-shot and 10-shot episode file (.npz file in the output directory). Thanks very much and hoping hearing from you soon! Best regards, Yukuan Yang Uploading few_shot_benchmark_RepMet_inloc_5shot_5way_10qpc_500epi_aug_0_10_lr_1.0e-04_nms_0.30_iou_0.70_clust_1_exact_1_2020-01-19-01-24.txt… — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22?email_source=notifications&email_token=ACOBU6TRIKCO3AKQ7AS74M3RB3ECLA5CNFSM4KIY2XK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELFUAIA#issuecomment-583745568>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACOBU6XC2CCUMUPD5WPWST3RB3ECLANCNFSM4KIY2XKQ> .