Wrong classification model output #29

AIGirl10 · 2021-04-07T12:37:39Z

Converted model from keras to NCNN. Pre-processing used in keras(python) is as follows

# reading image in cv Mat format
orig = cv2.imread('black.png')
# Converting image from BGR to RGB
image = cv2.cvtColor(orig, cv2.COLOR_BGR2RGB)
# Resizing image to 128x128
image = cv2.resize(image, (128, 128))
# Convert image to float by and normalize by dividing it by 255
image = image.astype("float") / 255.0
# Convert image to array
image = img_to_array(image)
# Expand numpy array dimentions
image = np.expand_dims(image, axis=0)
# Load keras model
model = tf.keras.models.load_model('cnn128.h5')
# Give input to keras model
pred = model.predict(image)[0]

I tried to replicate the steps in c++ for NCNN model

// reading image in cv Mat format
Mat image = imread("eyes.jpg"); //read image
// Converting image from BGR to RGB
cv::cvtColor(image, image, cv::COLOR_BGR2RGB);
// Resizing image to 128x128
Mat img;   
resize(image, img, Size(128, 128));
// Convert image to float by and normalize by dividing it by 255
img.convertTo(img,CV_32F);
img /= 255.0;
// Load ncnn model
ncnn::Net smallCNN;
smallCNN.load_param("cnn128.param");
smallCNN.load_model("cnn128.bin");
// converting cv::Mat image to ncnn::Mat
ncnn::Mat in = ncnn::Mat::from_pixels(img.data, ncnn::Mat::PIXEL_RGB, img.cols, img.rows);
// Give input to ncnn model
ncnn::Extractor ex = smallCNN.create_extractor();
ex.input("conv2d_8_input_blob", in);

when run the code, as an output I am getting this value

8 = 1.000000
7 = 0.000000
6 = 0.000000

where 8 is the label and 1.000000 is the probability. correct label is 0 and not 8

can anyone please help to understand what went wrong in input processing ?

MarsTechHAN · 2021-04-08T04:46:35Z

Can you provide the model for debugging?

AIGirl10 · 2021-04-08T06:01:43Z

@MarsTechHAN

Model link: https://drive.google.com/drive/folders/1--ZzTFtdDkgRxsezZfW65is5KVBfJh8I?usp=sharing

Full c++ inference program

#include <vector>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>

#include <opencv2/ml.hpp>
#include <iterator>
#include <fstream>
#include "dirent.h"
#include "net.h"

using namespace cv::ml;
using namespace std;
using namespace cv;

/* Function is taken from one of NCNN examples */
static int print_topk(const std::vector<float>& cls_scores, int topk)
{
    // partial sort topk with index
    int size = cls_scores.size();
    std::vector<std::pair<float, int> > vec;
    vec.resize(size);
    for (int i = 0; i < size; i++)
    {
        vec[i] = std::make_pair(cls_scores[i], i);
    }

    std::partial_sort(vec.begin(), vec.begin() + topk, vec.end(),
                      std::greater<std::pair<float, int> >());

    // print topk and score
    for (int i = 0; i < topk; i++)
    {
        float score = vec[i].first;
        int index = vec[i].second;
        fprintf(stderr, "%d = %f\n", index, score);
    }

    return 0;
}

int main()
{
    	// reading image in cv Mat format
	Mat image = imread("eyes.jpg"); //read image
	// Converting image from BGR to RGB
	cv::cvtColor(image, image, cv::COLOR_BGR2RGB);
	// Resizing image to 128x128
	Mat img;   
	resize(image, img, Size(128, 128));
	// Convert image to float by and normalize by dividing it by 255
	img.convertTo(img,CV_32F);
	img /= 255.0;
	// Load ncnn model
	ncnn::Net smallCNN;
	smallCNN.load_param("cnn128.param");
	smallCNN.load_model("cnn128.bin");
	// converting cv::Mat image to ncnn::Mat
	ncnn::Mat in = ncnn::Mat::from_pixels(img.data, ncnn::Mat::PIXEL_RGB, img.cols, img.rows);
	// Give input to ncnn model
	ncnn::Extractor ex = smallCNN.create_extractor();
	ex.input("conv2d_8_input_blob", in);

        ncnn::Mat out;
    	ex.extract("dense_5_Softmax_blob", out);

	vector<float> v1;
	std::vector<float> cls_scores;
	cls_scores.resize(out.w);
    	for (int j = 0; j < out.w; j++)
    	{
       	 	cls_scores[j] = out[j];
		std::cout<<"score: "<<out[j];
    	}
	print_topk(cls_scores, 3);
    	return 0;
}

MarsTechHAN · 2021-04-08T07:03:07Z

Thanks! Can you also provide the original keras model comparing?

AIGirl10 · 2021-04-08T07:53:35Z

Uploaded kerass model same folder here: https://drive.google.com/drive/folders/1--ZzTFtdDkgRxsezZfW65is5KVBfJh8I?usp=sharing

MarsTechHAN · 2021-04-08T14:57:16Z

Here is the keras2ncnn debug mode output (by inputing random data), it seems like the network forward path is completely identical, so the problem may be in the pre-processing stage of the network. Can you try to extract the conv2d_8_input and conv2d_8 layers and comparing them to the image in the line pred = model.predict(image)[0]

conv2d_8_input
==================================
Layer Name: conv2d_8, Layer Shape: keras->(1, 43, 43, 16) ncnn->(16, 43, 43)
Max:    keras->0.819 ncnn->0.819        Min: keras->-0.964 ncnn->-0.964
Mean:   keras->-0.083 ncnn->-0.083      Var: keras->0.215 ncnn->0.215
Cosine Similarity: 0.00000
Keras Feature Map:      [ 0.084  0.128 -0.216 -0.194  0.106 -0.197 -0.051  0.18  -0.159 -0.058]
Ncnn Feature Map:       [ 0.084  0.128 -0.216 -0.194  0.106 -0.197 -0.051  0.18  -0.159 -0.058]
==================================
Layer Name: max_pooling2d_8, Layer Shape: keras->(1, 22, 22, 16) ncnn->(16, 22, 22)
Max:    keras->0.819 ncnn->0.819        Min: keras->-0.693 ncnn->-0.693
Mean:   keras->0.097 ncnn->0.097        Var: keras->0.178 ncnn->0.178
Cosine Similarity: 0.00000
Keras Feature Map:      [0.128 0.016 0.106 0.18  0.16  0.01  0.062 0.162 0.175 0.135]
Ncnn Feature Map:       [0.128 0.016 0.106 0.18  0.16  0.01  0.062 0.162 0.175 0.135]
==================================
Layer Name: conv2d_9, Layer Shape: keras->(1, 11, 11, 32) ncnn->(32, 11, 11)
Max:    keras->0.994 ncnn->0.994        Min: keras->-1.331 ncnn->-1.331
Mean:   keras->-0.115 ncnn->-0.115      Var: keras->0.370 ncnn->0.370
Cosine Similarity: -0.00000
Keras Feature Map:      [0.594 0.511 0.582 0.539 0.646 0.273 0.217 0.661 0.864 0.311]
Ncnn Feature Map:       [0.594 0.511 0.582 0.539 0.646 0.273 0.217 0.661 0.864 0.311]
==================================
Layer Name: max_pooling2d_9, Layer Shape: keras->(1, 6, 6, 32) ncnn->(32, 6, 6)
Max:    keras->0.994 ncnn->0.994        Min: keras->-0.900 ncnn->-0.900
Mean:   keras->0.034 ncnn->0.034        Var: keras->0.345 ncnn->0.345
Cosine Similarity: 0.00000
Keras Feature Map:      [0.594 0.721 0.724 0.661 0.864 0.175]
Ncnn Feature Map:       [0.594 0.721 0.724 0.661 0.864 0.175]
==================================
Layer Name: conv2d_10, Layer Shape: keras->(1, 6, 6, 64) ncnn->(64, 6, 6)
Max:    keras->1.508 ncnn->1.508        Min: keras->-2.416 ncnn->-2.416
Mean:   keras->-0.259 ncnn->-0.259      Var: keras->0.535 ncnn->0.535
Cosine Similarity: 0.00000
Keras Feature Map:      [-0.803 -0.622 -0.666 -0.458 -0.862 -0.529]
Ncnn Feature Map:       [-0.803 -0.622 -0.666 -0.458 -0.862 -0.529]
==================================
Layer Name: max_pooling2d_10, Layer Shape: keras->(1, 3, 3, 64) ncnn->(64, 3, 3)
Max:    keras->1.508 ncnn->1.508        Min: keras->-1.635 ncnn->-1.635
Mean:   keras->0.068 ncnn->0.068        Var: keras->0.479 ncnn->0.479
Cosine Similarity: 0.00000
Keras Feature Map:      [-0.622 -0.458 -0.529]
Ncnn Feature Map:       [-0.622 -0.458 -0.529]
==================================
Layer Name: conv2d_11, Layer Shape: keras->(1, 3, 3, 64) ncnn->(64, 3, 3)
Max:    keras->2.364 ncnn->2.364        Min: keras->-3.498 ncnn->-3.498
Mean:   keras->-0.388 ncnn->-0.388      Var: keras->0.852 ncnn->0.852
Cosine Similarity: -0.00000
Keras Feature Map:      [-0.083  0.35   1.077]
Ncnn Feature Map:       [-0.083  0.35   1.077]
==================================
Layer Name: max_pooling2d_11, Layer Shape: keras->(1, 2, 2, 64) ncnn->(64, 2, 2)
Max:    keras->2.364 ncnn->2.364        Min: keras->-2.076 ncnn->-2.076
Mean:   keras->-0.044 ncnn->-0.044      Var: keras->0.713 ncnn->0.713
Cosine Similarity: 0.00000
Keras Feature Map:      [0.35  1.077]
Ncnn Feature Map:       [0.35  1.077]
==================================
Layer Name: flatten_2, Layer Shape: keras->(1, 256) ncnn->(1, 1, 256)
Max:    keras->2.364 ncnn->2.364        Min: keras->-2.076 ncnn->-2.076
Mean:   keras->-0.044 ncnn->-0.044      Var: keras->0.713 ncnn->0.713
Cosine Similarity: 0.00000
Keras Feature Map:      [ 0.35   0.873 -0.533 -0.701 -1.271  0.158 -0.739 -0.816  0.883 -0.247]
Ncnn Feature Map:       [ 0.35   0.873 -0.533 -0.701 -1.271  0.158 -0.739 -0.816  0.883 -0.247]
Top-k:
Keras Top-k:    88:2.364, 174:1.651, 127:1.642, 195:1.427, 25:1.396
ncnn Top-k:     88:2.364, 174:1.651, 127:1.642, 195:1.427, 25:1.396
==================================
Layer Name: dense_4, Layer Shape: keras->(1, 256) ncnn->(1, 1, 256)
Max:    keras->6.627 ncnn->6.627        Min: keras->0.000 ncnn->0.000
Mean:   keras->0.997 ncnn->0.997        Var: keras->1.479 ncnn->1.479
Cosine Similarity: 0.00000
Keras Feature Map:      [0.    0.    1.482 1.711 2.979 5.397 0.    0.248 0.    1.833]
Ncnn Feature Map:       [0.    0.    1.482 1.711 2.979 5.397 0.    0.248 0.    1.833]
Top-k:
Keras Top-k:    82:6.627, 12:6.439, 90:5.565, 129:5.425, 201:5.404
ncnn Top-k:     82:6.627, 12:6.439, 90:5.565, 129:5.425, 201:5.404
==================================
Layer Name: dense_5_Softmax, Layer Shape: keras->(1, 9) ncnn->(1, 1, 9)
Max:    keras->1.000 ncnn->1.000        Min: keras->0.000 ncnn->0.000
Mean:   keras->0.111 ncnn->0.111        Var: keras->0.314 ncnn->0.314
Cosine Similarity: -0.00000
Keras Feature Map:      [0. 0. 0. 0. 0. 0. 0. 0. 1.]
Ncnn Feature Map:       [0. 0. 0. 0. 0. 0. 0. 0. 1.]
Top-k:
Keras Top-k:    8:1.000, 2:0.000, 5:0.000, 1:0.000, 0:0.000
ncnn Top-k:     8:1.000, 2:0.000, 5:0.000, 7:0.000, 6:0.000
Done!

MarsTechHAN · 2021-04-09T03:21:54Z

Something like this happened before... but I just not able to find where the accuracy mismatched.

AIGirl10 · 2021-04-09T04:03:00Z

yes, same for me too. But will try to print the output of first layer in both keras and NCNN, then we might be able to solve it. Will let you know if I get any success ! Thanks though

MarsTechHAN · 2021-04-09T04:49:45Z

Maybe you can also give me a test picture and expected output

AIGirl10 · 2021-04-09T05:22:23Z

@MarsTechHAN

Image is updated in the same drive link: https://drive.google.com/drive/folders/1--ZzTFtdDkgRxsezZfW65is5KVBfJh8I?usp=sharing

Expected output: 0

where 0 is the label index.

MarsTechHAN · 2021-04-09T05:37:17Z

Cool, let me have a look.

AIGirl10 · 2021-04-09T12:49:36Z

@MarsTechHAN

How did you print these values ?

If possible can you please help me with the program which prints it ?

MarsTechHAN · 2021-04-10T12:30:55Z

@MarsTechHAN

How did you print these values ?

If possible can you please help me with the program which prints it ?

It's keras2ncnn's build-in debug mode, by running:

python3 -mkeras2ncnn -i MODEL_FILE.h5 -d

However, it still contains a lot of bug. I am thinking of using docker for it. You can refer this code snap for your program:

keras2ncnn/keras2ncnn/keras_debugger.py

Lines 32 to 51 in e3f90c4

    
           static int dump_mat(const ncnn::Mat& m, const char *m_name) 
        
           { 
        
               char filename[1000] = ""; 
        
               sprintf(filename, "%s-%d-%d-%d-layer_dump.dat", m_name, m.c, m.h, m.w); 
        
               FILE* fp = fopen(filename, "w+"); 
        
               if(fp == NULL){ 
        
                   return -1; 
        
               } 
        
               for (int q=0; q<m.c; q++) 
        
               { 
        
                   const float* ptr = m.channel(q); 
        
                   fwrite(ptr, sizeof(float), m.w * m.h, fp); 
        
               } 
        
               fclose(fp); 
        
               return 0; 
        
           }

and this for loading:

keras2ncnn/keras2ncnn/keras_debugger.py

Lines 308 to 309 in e3f90c4

    
           ncnn_det_out[mat_sp[0]] = np.fromfile(os.path.join(self.tmp_dir, 'ncnn', 'build', 'benchmark', mat_file), 
        
                                                 dtype='float32').reshape(*list(map(int, mat_sp[1:4])))

AIGirl10 · 2021-04-19T18:11:24Z

@MarsTechHAN

One thing I noticed is this, in keras the dimensions of input are type: float32[?,128,128,3] and in NCNN is type: float32[128,128,3]

Keras is 4D and NCNN is 3D, do you know how to convert the NCNN image to 4D ?

AIGirl10 · 2021-04-20T06:14:38Z

@MarsTechHAN

While using the debugger I am getting this error

python3 -mkeras2ncnn -i cnn128.h5 -d

error

ValueError: ('Unrecognized keyword arguments:', dict_keys(['ragged']))

MarsTechHAN · 2021-04-23T09:26:56Z

@MarsTechHAN

One thing I noticed is this, in keras the dimensions of input are type: float32[?,128,128,3] and in NCNN is type: float32[128,128,3]

Keras is 4D and NCNN is 3D, do you know how to convert the NCNN image to 4D ?

ncnn does not have 4D tensor and the batch dim. The dim of the keras is 4-D because of the batch, you dont need that in the ncnn. So no worrying, it's correct.

@MarsTechHAN

While using the debugger I am getting this error
python3 -mkeras2ncnn -i cnn128.h5 -d
error
ValueError: ('Unrecognized keyword arguments:', dict_keys(['ragged']))

Can you paste the full ouptut log? It seems like not the issue from the debugger

MarsTechHAN added the model file required label Apr 8, 2021

MarsTechHAN added result unmatch and removed model file required labels Apr 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wrong classification model output #29

Wrong classification model output #29

AIGirl10 commented Apr 7, 2021

MarsTechHAN commented Apr 8, 2021

AIGirl10 commented Apr 8, 2021

MarsTechHAN commented Apr 8, 2021

AIGirl10 commented Apr 8, 2021

MarsTechHAN commented Apr 8, 2021

MarsTechHAN commented Apr 9, 2021

AIGirl10 commented Apr 9, 2021

MarsTechHAN commented Apr 9, 2021

AIGirl10 commented Apr 9, 2021 •

edited

MarsTechHAN commented Apr 9, 2021

AIGirl10 commented Apr 9, 2021

MarsTechHAN commented Apr 10, 2021

AIGirl10 commented Apr 19, 2021 •

edited

AIGirl10 commented Apr 20, 2021

MarsTechHAN commented Apr 23, 2021

Wrong classification model output #29

Wrong classification model output #29

Comments

AIGirl10 commented Apr 7, 2021

MarsTechHAN commented Apr 8, 2021

AIGirl10 commented Apr 8, 2021

MarsTechHAN commented Apr 8, 2021

AIGirl10 commented Apr 8, 2021

MarsTechHAN commented Apr 8, 2021

MarsTechHAN commented Apr 9, 2021

AIGirl10 commented Apr 9, 2021

MarsTechHAN commented Apr 9, 2021

AIGirl10 commented Apr 9, 2021 • edited

MarsTechHAN commented Apr 9, 2021

AIGirl10 commented Apr 9, 2021

MarsTechHAN commented Apr 10, 2021

AIGirl10 commented Apr 19, 2021 • edited

AIGirl10 commented Apr 20, 2021

MarsTechHAN commented Apr 23, 2021

AIGirl10 commented Apr 9, 2021 •

edited

AIGirl10 commented Apr 19, 2021 •

edited