The accuracy report during test phase is not correct

[lqshixinlei]

I have a very strange problem. I used caffe to do image classification. There are seven classes. All the images I used is 50*250. At the end of training phase, caffe report the accuracy on test images is almost 99.9%+. But when I used python interface to do predict on the test data, I find the accuracy which was computed by the python script is about 65%. This is not correct. Does any one has some hint?

This is my train prototxt:

name: "capnumNet"
layer {
name: "capnum"
type: "Data"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
scale: 0.00390625
}
data_param {
source: "examples/captcha_number/capnum_train_lmdb"
batch_size: 64
backend: LMDB
}
}
layer {
name: "capnum"
type: "Data"
top: "data"
top: "label"
include {
phase: TEST
}
transform_param {
scale: 0.00390625
}
data_param {
source: "examples/captcha_number/capnum_val_lmdb"
batch_size: 64
backend: LMDB
}
}
layer {
name: "conv1"
type: "Convolution"
bottom: "data"
top: "conv1"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 2
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "pool1"
type: "Pooling"
bottom: "conv1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv2"
type: "Convolution"
bottom: "pool1"
top: "conv2"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
convolution_param {
num_output: 2
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "pool2"
type: "Pooling"
bottom: "conv2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "ip1"
type: "InnerProduct"
bottom: "pool2"
top: "ip1"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
inner_product_param {
num_output: 50
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu1"
type: "ReLU"
bottom: "ip1"
top: "ip1"
}
layer {
name: "ip2"
type: "InnerProduct"
bottom: "ip1"
top: "ip2"
param {
lr_mult: 1
}
param {
lr_mult: 1
}
inner_product_param {
num_output: 7
weight_filler {
type: "gaussian"
std: 0.001
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "accuracy"
type: "Accuracy"
bottom: "ip2"
bottom: "label"
top: "accuracy"
include {
phase: TEST
}
}
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "ip2"
bottom: "label"
top: "loss"
}

This is the python script that I used to predict and computing accuracyimport numpy as np

import matplotlib.pyplot as plt

caffe_root = '../../'
import sys
sys.path.insert(0, caffe_root + 'python')

import caffe

MODEL_FILE = caffe_root+'examples/captcha_number/deploy.prototxt'
PRETRAINED = caffe_root+'examples/captcha_number/bak/capnumNet_iter_1000.caffemodel'
caffe.set_mode_gpu()
net = caffe.Classifier(MODEL_FILE, PRETRAINED, raw_scale=255, image_dims=(50, 250))
import os
imgs = os.listdir('/search/proj/caffe/data/validCode/captcha_number/val/')
dir_prefix = '/search/proj/caffe/data/validCode/captcha_number/val/'
ldic = {'16':0, '15':1, '17':2, '23':3, '18':4, '28':5, '19':6}
cnt = 0
err_cnt = 0
for img in imgs:
cls = img.split('-')[0]
IMAGE_FILE = dir_prefix+img
input_image = caffe.io.load_image(IMAGE_FILE)
prediction = net.predict([input_image])
p = prediction[0].argmax()
if p != ldic[cls]:
err_cnt = e_cnt + 1
if cls == '16':
print img,'class:', p
cnt = cnt + 1
print float(err_cnt)/cnt

[seanbell]

Your scales are different: [0, 1] at training time but [0, 255] in the Python interface.

At training time, you set scale: 0.00390625 to make the input scale in the range [0, 1]. Thus, you should set raw_scale=1 at test time in the Python interface to keep the same scale.

Also, you should be asking these kinds of question on the caffe users group.

[lqshixinlei]

@seanbell , thank you very much, I found this error. But even I have fixed the scaling problem, the accuracy of my python script is also not correct. I used 'caffe test' to test my model, the accuracy is about 99.5%. But the python script is about 90.5%. Are there still have some errors?

[c437yuyang]

Have you solved this
problem ？I find this question too,and cannot find any way to solve...if I use digits's classify many ,it can gets the same accuracy as I trained the model,but when I use Python interface to classify,the accuracy is much lower