Alexnet only get 48.5% Top 1 accuracy

[yiqianglee]

Hi,
After download the chainer alexnet pre-trained model, we only get 48.5% Top 1 accuracy, is it expected?

[knorth55]

After download the chainer alexnet pre-trained model, we only get 48.5% Top 1 accuracy, is it expected?

What do you mean by Top 1 accuracy?
Can you put a code which can reproduct your issue?

[yiqianglee]

We just use Imagenet's validation set to count the accuracy, totally, it should be 50,000 images, maybe too large for me to upload, but i think you can download it from official imagenet website.

One more question, is this chainer model trained by chainer directly? What hyper-parameter do you use? Liker learning rate, momentum rate, batch size, epoch ??

[knorth55]

this chainer model is converted from alexnet model trained by caffe,
and the original caffe model is downloaded from http://dl.caffe.berkeleyvision.org/bvlc_alexnet.caffemodel

[knorth55]

We just use Imagenet's validation set to count the accuracy, totally, it should be 50,000 images, maybe too large for me to upload, but i think you can download it from official imagenet website.

this is weird. let me have a look to compare the chainermodel is correctly converted from original caffemodel.
can you put your validation code for imagenet?
you dont need to upload imagenet dataset, just the code you wrote.

[wkentaro]

Maybe we need evaluate.py to reproduce the evaluation in the original work.

[yiqianglee]

Thanks for you reply, and hope for your result.

Actually, i'm more interesting to find a chainer model trained by chainer directly, not converted from caffe, but still not find any model zoo, is there any comment from your guys? Thanks!

[knorth55]

Actually, i'm more interesting to find a chainer model trained by chainer directly, not converted from caffe, but still not find any model zoo, is there any comment from your guys? Thanks!

I've never seen Alexnet model trained by chainer because chainer can load caffemodel by chainer.links.caffe.CaffeFunction.
But if there is a big difference between the original caffemodel and model trained by chainer, it would be quite interesting.

[knorth55]

Actually, i'm more interesting to find a chainer model trained by chainer directly, not converted from caffe, but still not find any model zoo, is there any comment from your guys? Thanks!

I've never seen Alexnet model trained by chainer because chainer can load caffemodel by chainer.links.caffe.CaffeFunction.
But if there is a big difference between the original caffemodel and model trained by chainer, it would be quite interesting.

[knorth55]

this is weird. let me have a look to compare the chainermodel is correctly converted from original caffemodel.

I compared weight of each layers and calculated diff of original caffemodel and original chainermodel.
result is as below and there seems no big diff between the two models.

linkname: conv1
    diff norm: 0.0
linkname: conv2
    diff norm: 0.0
linkname: conv3
    diff norm: 0.0
linkname: conv4
    diff norm: 0.0
linkname: conv5
    diff norm: 0.0
linkname: fc6
    diff norm: 0.0
linkname: fc7
    diff norm: 0.0
linkname: fc8
    diff norm: 0.0

checking script is as below:

import chainer
from chainer.links import caffe
from model import AlexNet
import numpy as np
import os.path as osp

filepath = osp.abspath(osp.dirname(__file__))


caffemodel_path = osp.join(filepath, './data/bvlc_alexnet.caffemodel')
chainermodel_path = osp.join(filepath, './data/bvlc_alexnet.chainermodel')

# chainer
chainermodel = AlexNet()
chainer.serializers.load_hdf5(chainermodel_path, chainermodel)


# caffe
caffemodel = caffe.CaffeFunction(caffemodel_path)

linknames = sorted([x.name for x in caffemodel.children()])

for linkname in linknames:
    print('linkname: {}'.format(linkname))
    caffe_layer = getattr(caffemodel, linkname)
    chainer_layer = getattr(chainermodel, linkname)
    diff_norm = np.linalg.norm(caffe_layer.W.data - chainer_layer.W.data) 
    print('    diff norm: {}'.format(diff_norm))

[yiqianglee]

Thanks for your kindly reply.

We use the similar script to load chainer model. Have you tried in whole validation set, and get the overall accuracy?

[knorth55]

We use the similar script to load chainer model. Have you tried in whole validation set, and get the overall accuracy?

We've never done before.
Please check if you dont forget to preprocess images before you infer the data.