Reported results and reproduced results

[zhigao2017]

Hi authors, many thanks for your released code, it helps me better to understand your excellent work. A question is that reported results in your paper on the MiniImagenet dataset 1-shot 5-way using 4 Cov is 54.43, while I got less than 40 using your code. Maybe there are some mistakes in my operations, could you tell me some crucial steps when carrying out this code? Thanks very much!

[RongKaiWeskerMA]

Do you use exactly the same hyperparameters in their paper?

[RongKaiWeskerMA]

I've tried all the hyperparameters reported in the paper, however, got 40%

[zhigao2017]

I've tried all the hyperparameters reported in the paper, however, got 40%

Yes, I use the mentioned hyperparameters in the paper, and the final results are about 40%.

[KhrulkovV]

@ZhiGaomcislab @RongKaiWeskerMA Hi, thanks for raising this issue. My guess is that the problem is caused by --dim argument which is 64 by default and we used 1600 in the paper. We apologize if we haven't mentioned it in the paper, this is the default value used by many few-shot learning papers. Concretely, we are able to reproduce the results on the 1-shot 5-way MiniImageNet task with the following command:
python train_protonet.py --dataset MiniImageNet --way 30 --shot 1 --lr 0.005 --step 80 --gamma 0.5 --c 0.01 --model convnet --hyperbolic --not-riemannian --dim 1600
where the parameters are as reported in the paper (--not-riemannian is a dummy argument to be removed soon) and note that for training --way is set to 30 as commonly done.
Please tell me if this helps!

We will add the training scripts for complete reproducibility to the repo asap.

[RongKaiWeskerMA]

Thanks @KhrulkovV a lot, the performance improved a lot for 5 way 1 shot testing.
Can I ask a question that for 5 way testing, why would we use 30 way in training phase instead?
In terms of the result for 5 way 1 shot MiniImageNet task, the final test accuracy is around 51% which is slightly off with 54% with 4 layers ConvNet, how can I resolve this issue?

[sumo8291]

@ZhiGaomcislab @RongKaiWeskerMA Hi, thanks for raising this issue. My guess is that the problem is caused by --dim argument which is 64 by default and we used 1600 in the paper. We apologize if we haven't mentioned it in the paper, this is the default value used by many few-shot learning papers. Concretely, we are able to reproduce the results on the 1-shot 5-way MiniImageNet task with the following command:
python train_protonet.py --dataset MiniImageNet --way 30 --shot 1 --lr 0.005 --step 80 --gamma 0.5 --c 0.01 --model convnet --hyperbolic --not-riemannian --dim 1600
where the parameters are as reported in the paper (--not-riemannian is a dummy argument to be removed soon) and note that for training --way is set to 30 as commonly done.
Please tell me if this helps!

We will add the training scripts for complete reproducibility to the repo asap.

What are the values of dim and way for ResNet-18?

[sumo8291]

MINI-R18-1SHOT-5WAY.log
I have tried to run the same for ResNet-18 on the MiniImageNet dataset. The results I obtained is around 29%. I have tried for 30 Way too .. but the results are same.

Can you please help regarding this?

[KhrulkovV]

@sumo8291 Hey, sorry for the late response, I will look into it tomorrow.

[sumo8291]

MINI-R18-1SHOT-5WAY.log
I have tried to run the same for ResNet-18 on the MiniImageNet dataset. The results I obtained is around 29%. I have tried for 30 Way too .. but the results are same.

Can you please help regarding this?

I have used the Pretrained models https://drive.google.com/drive/folders/19TdjthkqMKLKSVHrbT5pVEmKvu-6-6iM and have achieved 56.2% accuracy on Resnet18 for 1-SHOT and 30WAY.

[KhrulkovV]

Hello!
The following script should be able to obtain the result from the paper for resnet18 miniImageNet 1-shot task (using the reported params from the paper).

python train_protonet.py --dataset MiniImageNet --way 30 --shot 1 --lr 0.001 --step 80 --gamma 0.5 --c 0.01 --model resnet18 --hyperbolic --not-riemannian --dim 512

[KhrulkovV]

@sumo8291 @RongKaiWeskerMA @ZhiGaomcislab hey, we've uploaded the scripts to reproduce all the results to https://github.com/leymir/hyperbolic-image-embeddings/tree/master/examples/fewshot/configs

Closing now.

[greeneggsandyaml]

Hello, sorry to follow up on this closed issue, but I'm having problems reproducing the results, even with the new scripts. I have not modified anything -- I literally just ran the scripts.

Here are is what I ran and what I got for CUB 1s5w (conv4):

# CUB 1s5w (conv4) 0.6402 +- 0.002
python train_protonet.py \
  --dataset CUB \
  --shot 1 \
  --lr 0.001 \
  --step 50 \
  --gamma 0.8 \
  --c 0.05 \
  --model convnet \
  --hyperbolic \
  --not-riemannian \
  --dim 1600

### OUTPUT ###
batch 9998: 62.48(49.33)
batch 9999: 62.47(54.67)
batch 10000: 62.47(54.67)
Val Best Acc 0.6809, Test Acc 0.6247
Test Acc 0.6247 + 0.0024

Here are is what I ran and what I got for MiniImageNet 1s5w (conv4):

# MiniImageNet 1s5w (conv4) 0.5443 +- 0.002
python train_protonet.py \
  --dataset MiniImageNet \
  --way 30 \
  --shot 1 \
  --lr 0.005 \
  --step 80 \
  --gamma 0.5 \
  --c 0.01 \
  --model convnet \
  --hyperbolic \
  --not-riemannian \
  --dim 1600

### OUTPUT ###
batch 9998: 52.15(44.00)
batch 9999: 52.15(60.00)
batch 10000: 52.15(42.67)
Val Best Acc 0.5362, Test Acc 0.5215
Test Acc 0.5215 + 0.0020