Performance discrepancy in ResNet50/101 evaluation on ImageNette (IPC=10)

[FizzQAQ]

Dear Authors,

First of all, thank you for your impressive work on $CaO_2$ and for sharing the code/data. I have been studying your paper and attempting to reproduce the results on ImageNette with IPC=10.

While my reproduction on ResNet18 closely matches the reported results in Table 1, I have observed a significant performance gap when evaluating on ResNet50 and ResNet101. This discrepancy persists even when I evaluate the distilled images provided by you.

Here are the detailed comparisons:

Model	Paper Reported (Table 1)	My Eval on Author's Data	My Reproduction (Re-generated)
ResNet18	65.0 ± 0.7%	66.8%	64.8% (Matches well)
ResNet50	67.5 ± 0.8%	62.8%	59.8%
ResNet101	66.3 ± 1.3%	55.2%	58.4%

Observations:

1. ResNet18 Alignment: The reproduction for ResNet18 is very successful, which suggests the environment and generation pipeline are likely correct.

2. Evaluation Gap: The fact that evaluating the provided distilled data locally yields significantly lower results for ResNet50 (-4.7%) and ResNet101 (-11.1%) suggests the issue might be related to the evaluation protocol rather than the distillation process itself.

Questions:

• Could you please tell me what points should be noted during the evaluation on Resnet50/101?

I would appreciate any guidance or suggestions you could provide to help close this gap.

Thank you for your time and help!

[FizzQAQ]

IMAGENET_TRAIN_PATH=/data/imagenette2/train
IMAGENET_VAL_PATH=/data/imagenette2/val

IPC=10
SPEC=imagenet-nette
N_CLASS=10

python main_validate.py --subset "$SPEC" --arch-name "resnet18" --factor 2
--num-crop 5 --mipc 300 --ipc $IPC --stud-name "resnet18" --re-epochs 300
--train-dir $IMAGENET_TRAIN_PATH --val-dir $IMAGENET_VAL_PATH --repeat 1
--syn-data-path $IMAGENET_PATH

python main_validate.py --subset "$SPEC" --arch-name "resnet18" --factor 2
--num-crop 5 --mipc 300 --ipc $IPC --stud-name "resnet50" --re-epochs 300
--train-dir $IMAGENET_TRAIN_PATH --val-dir $IMAGENET_VAL_PATH --repeat 1
--syn-data-path $IMAGENET_PATH

python main_validate.py --subset "$SPEC" --arch-name "resnet18" --factor 2
--num-crop 5 --mipc 300 --ipc $IPC --stud-name "resnet101" --re-epochs 300
--train-dir $IMAGENET_TRAIN_PATH --val-dir $IMAGENET_VAL_PATH --repeat 1
--syn-data-path $IMAGENET_PATH

Here are my evaluation settings

[x2x5]

Dear Authors,

First of all, thank you for your impressive work on C a O 2 and for sharing the code/data. I have been studying your paper and attempting to reproduce the results on ImageNette with IPC=10.

While my reproduction on ResNet18 closely matches the reported results in Table 1, I have observed a significant performance gap when evaluating on ResNet50 and ResNet101. This discrepancy persists even when I evaluate the distilled images provided by you.

Here are the detailed comparisons:

Model Paper Reported (Table 1) My Eval on Author's Data My Reproduction (Re-generated)
ResNet18 65.0 ± 0.7% 66.8% 64.8% (Matches well)
ResNet50 67.5 ± 0.8% 62.8% 59.8%
ResNet101 66.3 ± 1.3% 55.2% 58.4%
Observations:

1. ResNet18 Alignment: The reproduction for ResNet18 is very successful, which suggests the environment and generation pipeline are likely correct.
2. Evaluation Gap: The fact that evaluating the provided distilled data locally yields significantly lower results for ResNet50 (-4.7%) and ResNet101 (-11.1%) suggests the issue might be related to the evaluation protocol rather than the distillation process itself.

Questions:

• Could you please tell me what points should be noted during the evaluation on Resnet50/101?

I would appreciate any guidance or suggestions you could provide to help close this gap.
Thank you for your time and help!

I also encountered this problem.