unable to reproduce the result

[tangshixiang]

Dear Yang:

I am really impressed with your work. The work provides me with a new angle and significantly raise the benchmark of few-shot learning tasks. However, when I tried to reproduce your result with the public code, I found the test accuracy for 5way 5shot miniimagenet (Convnet) tasks is around 78%, and the final test accuracy is about 76%. I guess there must be some tricks, could you kindly help me?

Thank you.

[AlienceGG]

Hi Shixiang,

We appreciate your interest in our DPGN work.

As you know, different backbones (ConvNet/ResNet) should be cooperated with different training settings. In DPGN, a 5way-5shot task on miniImageNet using the ConvNet backbone, the specific parameter settings are introduced as follows. Due to the length limitations of paper, we didn't list out all the details in the paper. You could have a try and modify them if you like.

from collections import OrderedDict

config = OrderedDict()

config['dataset_name'] = 'mini-imagenet'
config['num_generation'] = 5
config['num_loss_generation'] = 3
config['generation_weight'] = 0.5
config['point_distance_metric'] = 'l1'
config['distribution_distance_metric'] = 'l1'


config['emb_size'] = 128
config['backbone'] = 'convnet'

train_opt = OrderedDict()
train_opt['num_ways'] = 5
train_opt['num_shots'] = 5
train_opt['batch_size'] = 40
train_opt['iteration'] = 100000
train_opt['lr'] = 1e-3
train_opt['weight_decay'] = 1e-6
train_opt['dec_lr'] = 15000
train_opt['lr_adj_base'] = 0.5
train_opt['dropout'] = 0.1
train_opt['loss_indicator'] = [1, 0, 0]

eval_opt = OrderedDict()
eval_opt['num_ways'] = 5
eval_opt['num_shots'] = 5
eval_opt['batch_size'] = 10
eval_opt['iteration'] = 1000
eval_opt['interval'] = 1000

config['train_config'] = train_opt
config['eval_config'] = eval_opt

Please note that config files and number/type of GPU(s) may affect experiment results (We didn't test the codebase on other environments). For most of our experiment results, we used 1~3 v100(32GB) card(s) to launch the model. Considering the hardware limitation for some people/lab, we released a config of 5way-1shot mini-ImageNet (ResNet12) that is guaranteed to be fed into a single 2080ti.

Let us know if you encounter any difficulty when reproducing the DPGN.

Yours,
DPGN Team