Accuracy of Resnet50 is much higher than reported!

[Erotemic]

EDIT: Originally I title this issue "What epoch are reported results from?", but after further results have come to light I've renamed it to: "Accuracy of Resnet50 is much higher than reported!".

I've been reproducing some of these experiments and my output numbers don't exactly line up. At the moment I'm currently assuming its due to different random number seeds for the Kaiming Normal initialization.

Is the accuracy you report always the accuracy of the test set on the 350th epoch? Or are you reporting the accuracy of the best epoch?

In my reproduction of the DPN92 experiment I my measured accuracy of the last epoch is 94.92%, but the highest overall accuracy was 95.10% on epoch 275.

Surprisingly, when I ran the Resnet50 example I got an accuracy of 95.72%! (but this is likely some issue on my end) (Edit: Actually it doesn't seem to be; see bellow)

[Erotemic]

Even more surprising is that that resnet50 example is reproducible:

I get the following error rates for densenet: 5.33%, dpn92: 5.08%, resnet50: 4.46%. Subtract these from 100 to get the accuracy.

[Erotemic]

I've gone as far as to reproduce the Resnet50 results using (close to) the original training scripts. I only modified them to add logging, and so they used the same learning rate schedule as reported here without manually changing the learning rate.

Using essentially the original code I achieve an accuracy of 95.370% with Resent50. AFAIK, this is a state-of-the-art result on CIFAR-10 for a single network trained from scratch with basic data augmentation. I'm interested in digging into this a bit further. I find it very strange (but not inconcevable) that a simple Resnet50 architecture outperforms DPN92. Why is this the case now, but not before? Did kuangliu make an error in measuring accuracy? Perhaps the manual learning rate schedule was different than what was reported? Did something in torch change? Before digging in too much, it would be useful if someone else can reproduce my findings.

This is the diff between master and the code I used. This is most is boilerplate junk. The only important bit is at the end, which defines the non-manual learning rate schedule.

diff --git a/main.py b/main.py
index 26a4e98..44be9b7 100644
--- a/main.py
+++ b/main.py
@@ -15,14 +15,19 @@ import argparse
 
 from models import *
 from utils import progress_bar
+import ubelt as ub
 
+# parser = argparse.ArgumentParser(description='PyTorch CIFAR10 Training')
+# parser.add_argument('--lr', default=0.1, type=float, help='learning rate')
+# parser.add_argument('--resume', '-r', action='store_true', help='resume from checkpoint')
+# args = parser.parse_args()
 
-parser = argparse.ArgumentParser(description='PyTorch CIFAR10 Training')
-parser.add_argument('--lr', default=0.1, type=float, help='learning rate')
-parser.add_argument('--resume', '-r', action='store_true', help='resume from checkpoint')
-args = parser.parse_args()
+# device = 'cuda' if torch.cuda.is_available() else 'cpu'
+
+model = ub.argval('--model')
+gpu_num = int(ub.argval('--gpu', default=0))
+device = torch.device('cuda', gpu_num)
 
-device = 'cuda' if torch.cuda.is_available() else 'cpu'
 best_acc = 0  # best test accuracy
 start_epoch = 0  # start from epoch 0 or last checkpoint epoch
 
@@ -51,7 +56,18 @@ classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship'
 # Model
 print('==> Building model..')
 # net = VGG('VGG19')
-net = ResNet18()
+# net = ResNet18()
+
+model = ub.argval('--model')
+
+if model == 'resnet50':
+    net = ResNet50()
+elif model == 'dpn92':
+    net = DPN92()
+elif model == 'densenet121':
+    net = DenseNet121()
+else:
+    raise KeyError(model)
 # net = PreActResNet18()
 # net = GoogLeNet()
 # net = DenseNet121()
@@ -66,17 +82,21 @@ if device == 'cuda':
     net = torch.nn.DataParallel(net)
     cudnn.benchmark = True
 
-if args.resume:
-    # Load checkpoint.
-    print('==> Resuming from checkpoint..')
-    assert os.path.isdir('checkpoint'), 'Error: no checkpoint directory found!'
-    checkpoint = torch.load('./checkpoint/ckpt.t7')
-    net.load_state_dict(checkpoint['net'])
-    best_acc = checkpoint['acc']
-    start_epoch = checkpoint['epoch']
+# if args.resume:
+#     # Load checkpoint.
+#     print('==> Resuming from checkpoint..')
+#     assert os.path.isdir('checkpoint'), 'Error: no checkpoint directory found!'
+#     checkpoint = torch.load('./checkpoint/ckpt.t7')
+#     net.load_state_dict(checkpoint['net'])
+#     best_acc = checkpoint['acc']
+#     start_epoch = checkpoint['epoch']
 
 criterion = nn.CrossEntropyLoss()
-optimizer = optim.SGD(net.parameters(), lr=args.lr, momentum=0.9, weight_decay=5e-4)
+# optimizer = optim.SGD(net.parameters(), lr=args.lr, momentum=0.9, weight_decay=5e-4)
+optimizer = optim.SGD(net.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
+
+logfile = open('./logfile_{}.txt'.format(model), 'a')
+
 
 # Training
 def train(epoch):
@@ -98,8 +118,10 @@ def train(epoch):
         total += targets.size(0)
         correct += predicted.eq(targets).sum().item()
 
-        progress_bar(batch_idx, len(trainloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
-            % (train_loss/(batch_idx+1), 100.*correct/total, correct, total))
+        msg = 'Loss: %.3f | Acc: %.3f%% (%d/%d)' % (train_loss/(batch_idx+1), 100.*correct/total, correct, total)
+        progress_bar(batch_idx, len(trainloader), msg)
+    logfile.write('Train epoch {}. {}'.format(epoch, msg))
+
 
 def test(epoch):
     global best_acc
@@ -118,8 +140,9 @@ def test(epoch):
             total += targets.size(0)
             correct += predicted.eq(targets).sum().item()
 
-            progress_bar(batch_idx, len(testloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
-                % (test_loss/(batch_idx+1), 100.*correct/total, correct, total))
+            msg = 'Loss: %.3f | Acc: %.3f%% (%d/%d)' % (test_loss/(batch_idx+1), 100.*correct/total, correct, total)
+            progress_bar(batch_idx, len(testloader), msg)
+        logfile.write('Test epoch {}. {}'.format(epoch, msg))
 
     # Save checkpoint.
     acc = 100.*correct/total
@@ -136,6 +159,20 @@ def test(epoch):
         best_acc = acc
 
 
-for epoch in range(start_epoch, start_epoch+200):
-    train(epoch)
-    test(epoch)
+schedule = [
+    (0, 150, .1),
+    (150, 250, .01),
+    (250, 350, .001),
+]
+
+for start, end, lr in schedule:
+
+    # Set learning rate
+    for param_group in optimizer.param_groups:
+        param_group['lr'] = lr
+
+    # Run for awhile
+    for epoch in range(start, end):
+        train(epoch)
+        if epoch > 200:
+            test(epoch)

[Erotemic]

I've finished runs of kuangliu's code for ResNet50, DenseNet121, and DPN92.

I did recently find a bug in netharn, that would cause the learning rate schedule to be slightly off, which I assume accounts for the discrepancy between my netharn numbers and when I reran kuangliu's scripts.
However, the discrepancy between my runs of ResNet50 and kuangliu's is still very real.

Here is my results table:

          model |  kuangliu  | rerun-kuangliu  |  netharn |
    -------------------------------------------------------
    ResNet50    |    93.62%  |         95.370% |  95.72%  |  <- how did that happen?
    DenseNet121 |    95.04%  |         95.420% |  94.47%  |
    DPN92       |    95.16%  |         95.410% |  94.92%  |

The first column is kuangliu's reported accuracy, the second column is me
running kuangliu's code, and the final column is using my own training
harness (handles logging and whatnot) called netharn.

EDIT: I recently learned that CuDNN is non-determenistic by default, so that can also account for some minor discrepancy between results, but I don't think its enough to explain away these findings.

[AlphaQi]

我使用作者的code进行训练，使用20层的resnet，竟然达到了95.04%的准确率，表示震惊，这还没有用其他技巧

[willyqin]

Accuracy of ResNet18 is much higher than reported too.

[willyqin]

我使用作者的code进行训练，使用20层的resnet，竟然达到了95.04%的准确率，表示震惊，这还没有用其他技巧

我用ResNet18跑出来的效果也超过了95%.

[jingege315]

我使用作者的code进行训练，使用20层的resnet，竟然达到了95.04%的准确率，表示震惊，这还没有用其他技巧

我用ResNet18跑出来的效果也超过了95%.

Me too. The val accuracy is 94.1% using ResNet18. But I get a so bad result (87%) using my code with the same structure in torch

[willyqin]

the ResNet structure of this code is not the standard structure. More channels are used. Hui <notifications@github.com> 于2018年11月28日周三 下午6:47写道：

…

我使用作者的code进行训练，使用20层的resnet，竟然达到了95.04%的准确率，表示震惊，这还没有用其他技巧 我用ResNet18跑出来的效果也超过了95%. Me too. The val accuracy is 94.1% using ResNet18. But I get a so bad result (87%) using my code with the same structure in torch — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#45 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKMDCur0apz2IywzwzY1wVJnxfp-x8a9ks5uzmmvgaJpZM4UlGI9> .

[Juicechen95]

In Torch7, the Resnet structure only has 16 channels in the first layer, but this has 64 channels.

[michaelrzhang]

Yeah, you can get 95% with resnet18.

[xxllp]

我还以为遇到鬼了，这个acc是分类acc的平均值还是总体的

[wangyongjie-ntu]

But In fact, I just clone this repo and run the code with "python main.py". I don't modify anything of the original code. The accuracy is just about 80~85%. Quite confusing. Can anybody share your training details?

[Erotemic]

Odd, perhaps its important to have the particular checkout I used (bf78d3b) and maybe apply the diff that I specify in an earlier post?

Also note that cudnn is non-deterministic by default, so there could be issues there. So many little things to consider with these deep models that can cause large differences!

[triangleCZH]

My observation: he doesn't have the first conv with kernel-7, and the following maxpooling layer. In total it's downsampled by three times 3232 -> 44, while for a normal ResNet it will be 3232 -> 11.
It increases FLOPS, reduces (a little bit) params, and enhance the acc a lot.