Reproducing CIFAR10 supervised results

[wtqin382]

I am attempting to reproduce your CIFAR10 supervised results from Table 1 (https://arxiv.org/pdf/1806.05236.pdf) using code from this repository. I cannot get within 0.5% of the following results:

• PreActResNet18, Manifold Mixup, 2.89% error
• PreActResNet152, Manifold Mixup, 2.76% error
• PreActResNet152, Manifold Mixup All Layers, 2.38% error

For example, the paper is vague on details such as initial learning rate, batchsize, Nesterov or not, and other settings. Could you kindly provide command line invocations to reproduce those results?

Also when running your training code I see test error variation of around 0.3-0.5% from epoch to epoch. Do you report results over multiple seeds or are these figures single-seed estimates of the test error?

[alexmlamb]

Hello,

Thanks for reaching out to us. How many epochs were you running for? I think that this has been the most common issue.

We can also give an exact command line command, at least for PreActResNet18.

We also submitted a new draft to arxiv which makes the experimental setup clearer and now reports mean/standard deviation across 5 trials for these experiments.

Best,

Alex.

[alexmlamb]

Hello,

Can you give us more details on what you ran?

Best,

Alex.

[vikasverma1077]

@wtqin382 if you can paste the output log, I can see what is going wrong.

"For example, the paper is vague on details such as initial learning rate, batchsize, Nesterov or not, and other settings." All these details are present in the Appendix of the paper (since the first version), except the Nesterov ( which is "True" by default in the code)

[vikasverma1077]

Hi @wtqin382, we have uploaded the new code with the command line argument to run each of the experiments in the paper. It will be great if you can try to reproduce the results and let us know if there is some issues.

[vikasverma1077]

@wtqin382 Hi there! were you able to reproduce the results?