C++ and the phase_train node

[jamesFerris]

I've been trying to see if I can convert the classifier.py program into a c++ equivalent. As far as I can tell I've been able to accurately simulate everything to get the graph outputs, but I've been getting strange values. If I set the phase_train node to false, as is done in the python code, my output tensor is made entirely of Nan values. If I set it to true, then I get actual values in the output that appear to match with what the python code gives.
Changing the phase_train node to true in the python code doesn't change its outputs.

What exactly does the phase_train do in this case? Why would setting it to false give such vastly different results in a C++ implementation? I gather this should not be the case, so what could be causing this?

Here is the relevant section of what I'm working on:

string input_layer = "input:0";
string phase_train_layer = "phase_train:0";
string output_layer = "embeddings:0";

tensorflow::Tensor input_tensor(tensorflow::DT_FLOAT, tensorflow::TensorShape({input_Images.size(), height, width, channels}));
auto input_tensor_mapped = input_tensor.tensor<float, 4>();        

for (int i = 0; i < input_Images.size(); i++) {
    Mat image = input_Images[i];
    const float * source_data = (float*) image.data;
    for (int h = 0; h < image.rows; ++h) {
        const float* source_row = source_data + (h * image.cols * image.channels());
        for (int w = 0; w < image.cols; ++w) {
            const float* source_pixel = source_row + (w * image.channels());
            for (int c = 0; c < image.channels(); ++c) {
                const float* source_value = source_pixel + c;
                //std::cout << *source_value << std::endl;
                input_tensor_mapped(i, h, w, c) = *source_value;
            }
        }
    }
}

tensorflow::Tensor phase_tensor(tensorflow::DT_BOOL, tensorflow::TensorShape());
phase_tensor.scalar<bool>()() = false;

    cout << phase_tensor.DebugString() << endl;
    cout << input_tensor.DebugString() << endl;
std::vector<tensorflow::Tensor> outputs;

std::vector<std::pair<string, tensorflow::Tensor>> feed_dict = {
    {input_layer, input_tensor},  
    {phase_train_layer, phase_tensor},
};    

Status run_status = session->Run(feed_dict,
                            {output_layer}, {} , &outputs);
if (!run_status.ok()) {
    LOG(ERROR) << "\tRunning model failed: " << run_status << "\n";
    return -1;
}

cout << outputs[0].DebugString() << endl;

[davidsandberg]

Hi,
Not sure if you got this to work already but the phase_train is does two things:

1. Dropout should be disabled during evaluation.
2. Batch norm should use statistics calculated during the training phase.
   I haven't used the C++ interface so I cannot say what's wrong with your code though.

[jamesFerris]

Hi,
Unfortunately I've had to put this particular exercise on hold, so I haven't made any significant progress since I made this query.
Just to confirm, that is what a phase_train of false should do, right?
I'll try to analyse more closely what tensorflow is doing vs. what I think it is doing.

[anshan-XR-ROB]

Hi, Ferris, Have you work out the C++ code?

[jamesFerris]

@AnshanTJU
Not to my satisfaction. The code I've posted "works" if its given a phase_train value of true, otherwise it all outputs are null. I am no closer to working out why this is the case, or if this is actually a bad thing.
However with a phase_train of true I can still get a classifier working - it doesn't work as well as the python code, though I think this might be more to do with how simply I'm ingesting data.

Works pulled me away from this for now, so I haven't been able to do any decent development. Hopefully things will spin back around sooner or later and I can work more on it.

[lingbao00]

Hi，Ferri，i am also doing transfer the facenet to c++,and i has a question,how you realize the svm training code and how to read the .pkl model file,i hope you can help me ,thank you

[jamesFerris]

@lingbao00
There may be a way to translate pkl files in c++, but I found it easier to build my own svm with OpenCV.
OpenCV has some Source Vector Machine modules, and these can be saved and loaded much like the .pkl model file (as long as it is a linear svm). The actions to take with that are very similar to that in Davids python code.

[lingbao00]

OK，thanks，by the way，i had get the output tensor using read image by for loop,But i found the library example using decode to get image data to graph,why you do not use this way.

[jamesFerris]

@lingbao00
Mainly because I want to be able to distort images using OpenCV (rotate, blur, etc) to get as much use as possible out of my dataset, and I'm not sure if I can do that and use tensorflows decode.
Again, this may not be the best possible way, it's just the way that made the most sense to me - I am by no means an expert with tensorflow.

[lingbao00]

OK，i know,any way thank you very much.

[lingbao00]

i am sorry to disturb you again, i had a problem,when i use opencv svm to train, i need the traindata ,a Mat,but Mat data defaultly is uchar, how you transfer the tensorflow data,the float data to uchar.

[jamesFerris]

@lingbao00
After you get the tensor outputs, define an appropriately sized cv::Mat of data type CV_32FC1 (something like height = number of output tensors, width = length of an output tensor). This will specify the contents of the Mat to be float. Then you can iterate over the tensor outputs and add them to the Mat using the .at(x,y) operator.
Something like this:

for (int i = 0; i < tenosr_output.size(); i++) {
      const auto output_mapped = tensor_output[i].tensor<float, 2>();
      for (int j = 0; j < tensor_output[0].shape().dim_sizes()[1]; j++) {
        outputMat.at<float>(i,j) = output_mapped(0, j);
      }
 }

Basically, each row of the outputMat will be one of the output tensors. You'll also need to keep track of which labels go with which tensor, as that's a required input for the SVM.

When you get it working, can you let me know what sort of tensor outputs you get when the phase_train node is false vs. when its true? I can see no reason why I get null values, so some confirmation I've done something wrong somewhere would be good.

[lingbao00]

i had transfer the classifier.py and compare.py to c++,and get the thensor i use your way,weather i set the phase_tensor true or false ,ii can get value not null,but to do classifier i should set it to true the value is same as python get,and to do compare i should set it to false the value is same to python get.the reason i still do not know .

[lingbao00]

i am sorry t disturb you again, i am want to ask you a question: which function can set tensorflow to use cpu only.

[jamesFerris]

@lingbao00
Sorry for the delay.

So did you use my method to go from opencv matrix to tensors? Or did you get it working with the tensorflow DecodeJpeg/DecodePng functions?

I think you can do something like this to set device:
tensorflow::graph::SetDefaultDevice("/cpu:0", &graph_def);

[jamesFerris]

@lingbao00
Could you post some of the code you used for your classifier? I've finally been able to re-examine my code, but I still can't work out how to get anything other than Nan values when phase_train=false. I'd like to confirm that its just something I'm doing wrong so I can close this issue.

[mndar]

@jamesFerris
I am experimenting with OpenCV 3.3.0, Tensor Flow 1.3.0 and the code you posted.
I get a segmentation fault at input_tensor_mapped(i, h, w, c) = *source_value; when I try to input a png image.
With a jpg image, I get nan as output irrespective of whether I set phase_tensor.scalar()() = true; or phase_tensor.scalar()() = false;

This is the complete program

#include <iostream>

#include "tensorflow/cc/client/client_session.h"
#include "tensorflow/cc/ops/standard_ops.h"
#include "tensorflow/core/framework/tensor.h"
#include "tensorflow/core/public/session.h"
#include <opencv2/opencv.hpp>

using namespace std;
using namespace tensorflow;
using namespace tensorflow::ops;
using namespace cv;
string input_layer = "input:0";
string phase_train_layer = "phase_train:0";
string output_layer = "embeddings:0";
        

int main (int argc, char *argv[]) {
	tensorflow::GraphDef graphDef;
	tensorflow::ReadBinaryProto(tensorflow::Env::Default(), "/disks/storage/downloads/20170512-110547/20170512-110547.pb", &graphDef);

	tensorflow::SessionOptions options;
	std::unique_ptr<tensorflow::Session> session(tensorflow::NewSession(options));

	tensorflow::Status sessionCreateStatus = session->Create(graphDef);
	vector<Mat> input_Images;
	Mat input_image = imread (argv[1]);
	input_Images.push_back (input_image);
	cout << "Input Rows:" << input_image.rows << " Cols:" << input_image.cols << endl;
	tensorflow::Tensor input_tensor(tensorflow::DT_FLOAT, tensorflow::TensorShape({1, input_image.rows, input_image.cols, 3}));
	auto input_tensor_mapped = input_tensor.tensor<float, 4>();
	cout << "Num Images: " << input_Images.size() << endl;
	for (int i = 0; i < input_Images.size(); i++) {
		  Mat image = input_Images[i];
		  cout << "Image: " << image.cols << "x" << image.rows << " " << image.channels() << endl;
		  const float * source_data = (float*) image.data;
		  for (int h = 0; h < image.rows; ++h) {
		      const float* source_row = source_data + (h * image.cols * image.channels());
		      for (int w = 0; w < image.cols; ++w) {
		          const float* source_pixel = source_row + (w * image.channels());
		          for (int c = 0; c < image.channels(); ++c) {
		              const float* source_value = source_pixel + c;
		              //cout << i << " " << h << " "<< w << " " << c << endl;		              
		              //std::cout << *source_value << std::endl;
		              input_tensor_mapped(i, h, w, c) = *source_value;
		          }
		      }
		  }
	}
	tensorflow::Tensor phase_tensor(tensorflow::DT_BOOL, tensorflow::TensorShape());
	phase_tensor.scalar<bool>()() = true;

  cout << phase_tensor.DebugString() << endl;
  cout << input_tensor.DebugString() << endl;
	std::vector<tensorflow::Tensor> outputs;

	std::vector<std::pair<string, tensorflow::Tensor>> feed_dict = {
		  {input_layer, input_tensor},  
		  {phase_train_layer, phase_tensor},
	};    

	Status run_status = session->Run(feed_dict,
		                          {output_layer}, {} , &outputs);
	if (!run_status.ok()) {
		  LOG(ERROR) << "\tRunning model failed: " << run_status << "\n";
		  return -1;
	}

	//cout << outputs[0].DebugString() << endl;
	for (auto output: outputs) {
		cout << output.DebugString() << endl;
	}
	cout << "TensorFlow End" << endl;
	return 0;
}

Am I missing something in initializing tensorFlow which might be the reason I am getting nan irrespective of phase_train ?

Thanks

[jamesFerris]

@mndar
For some reason it never occurred to me to try different image formats...I'll need to check that.
I'm using .png images, so you shouldn't be having any trouble copying them to a tensor. My best guess is that for some reason imread() is failing for you for .pngs, thus the memory source_value is trying to access is not assigned. I'm guessing argv[1] is a filepath? Make sure that that is correct - if you're using a relative path make sure it is relative to the executable. If that's all ok, verify that the matrix created by imread() is valid.

As for getting NaN with .jpg...I'm not too sure. Thats kinda my problem as well. Start by confirming your input images are 160x160 pixels, as facenet expects. That should be throwing an error though if that was the case.
Also, the code you've posted only runs on a single image - Facenet expects to run on a batch of images, even for classification. Try pushing your image to the input_Images vector 10 times or so. Not an elegant solution, but it should work.

[mndar]

@jamesFerris
Using code posted by guillaume-michel from tensorflow/tensorflow#8033 I am able to process both jpg and png
This is what I have so far
https://gist.github.com/mndar/a93cfa77080a1a490c88556a86e6cc86

The output doesn't match that of CLASSIFY in src/classfier.py (I compared the emb_array produced by classifier.py).
Once I have SVM training and classification working, I'll be able to see if the classification works or not.

[jamesFerris]

@mndar
Hmm...then I not sure why you're getting the segfault.
Also, I don't know whether directly comparing the emb_array output from c++ and python is appropriate...

Thanks for posting those, hopefully they'll help me with my experimenting, when I can get back to it properly.

[mndar]

This is my initial effort
Would love someone else also to use it and give feedback
https://github.com/mndar/facenet_classifier

[knighthappy]

Maybe use this code ,you can get the same result with the python code.
The key points are:
1.Image = Image - cv::Vec3d(mean_pxl, mean_pxl, mean_pxl);
2.const double* source_value = source_pixel + (2-c);//RGB->BGR
3.double*

    tensorflow::SessionOptions session_options;
session_options.config.mutable_gpu_options()->set_allow_growth(true);

auto session = NewSession(session_options);
if (session == nullptr) {
	throw runtime_error("Could not create Tensorflow session.");
}

Status status;

// Read in the protobuf graph we exported
MetaGraphDef graph_def;
status = ReadBinaryProto(Env::Default(), pathToGraph, &graph_def);
if (!status.ok()) {
	throw runtime_error("Error reading graph definition from " + pathToGraph + ": " + status.ToString());
}

// Add the graph to the session
status = session->Create(graph_def.graph_def());
if (!status.ok()) {
	throw runtime_error("Error creating graph: " + status.ToString());
}

// Read weights from the saved checkpoint
Tensor checkpointPathTensor(DT_STRING, TensorShape());
checkpointPathTensor.scalar<std::string>()() = checkpointPath;
status = session->Run(
{ { graph_def.saver_def().filename_tensor_name(), checkpointPathTensor }, },
{},
{ graph_def.saver_def().restore_op_name() },
	nullptr);
if (!status.ok()) {
	throw runtime_error("Error loading checkpoint from " + checkpointPath + ": " + status.ToString());
}

std::string path = "33.jpg";
cv::Mat Image = cv::imread(path);
int height = 160;
int width = 160;
//cv::cvtColor(Image,Image, CV_RGB2BGR);

/*cv::Mat sdsad = Image + cv::Vec3b(2,2,2);
sdsad = sdsad / 2;
cv::Vec3b xx = sdsad.at<cv::Vec3b>(0, 0);
cout << xx;

cv::Mat_<cv::Vec3b>::iterator it1 = Image.begin<cv::Vec3b>();
cv::Mat_<cv::Vec3b>::iterator itend1 = Image.end<cv::Vec3b>();
for (; it1 != itend1; ++it1) {
	cout << (*it1)[0] << endl;
	cout << (*it1)[1] << endl;
	cout << (*it1)[2] << endl;
}*/
//
int depth = 3;

// creating a Tensor for storing the data
tensorflow::Tensor input_tensor(tensorflow::DT_FLOAT, tensorflow::TensorShape({ 1,height,width,depth }));
auto input_tensor_mapped = input_tensor.tensor<float, 4>();

//mean and std
cv::Mat temp = Image.reshape(1, Image.rows * 3);
cv::Mat     mean3;
cv::Mat     stddev3;
cv::meanStdDev(temp, mean3, stddev3);

double mean_pxl = mean3.at<double>(0);
double stddev_pxl = stddev3.at<double>(0);


string input_layer = "input:0";
string input_flag_layer = "phase_train:0";
std::vector<string> outputOps;
outputOps.push_back("embeddings:0");
std::vector<tensorflow::Tensor> outputTensors;
tensorflow::Tensor phase_tensor(tensorflow::DT_BOOL, tensorflow::TensorShape());
phase_tensor.scalar<bool>()() = false;

cv::Mat Image2;
Image.convertTo(Image2, CV_64FC1);
Image = Image2;
Image = Image - cv::Vec3d(mean_pxl, mean_pxl, mean_pxl);
Image = Image / stddev_pxl;

/*cv::Mat_<cv::Vec3d>::iterator it = Image.begin<cv::Vec3d>();
cv::Mat_<cv::Vec3d>::iterator itend = Image.end<cv::Vec3d>();
for (; it != itend; ++it) {
	cout << (*it)[0] << endl;
	cout << (*it)[1] << endl;
	cout << (*it)[2] << endl;
}*/
const double * source_data = (double*)Image.data;

// copying the data into the corresponding tensor
for (int y = 0; y < height; ++y) {
	const double* source_row = source_data + (y * width * depth);
	for (int x = 0; x < width; ++x) {
		const double* source_pixel = source_row + (x * depth);
		for (int c = 0; c < depth; ++c) {
			const double* source_value = source_pixel + (2-c);//RGB->BGR
			input_tensor_mapped(0, y, x, c) = *source_value;
		}
	}
}

status = session->Run({ { input_layer, input_tensor },{ input_flag_layer ,phase_tensor } }, outputOps, {}, &outputTensors);
cout << outputTensors[0].DebugString() << endl;
return 0;

[jamesFerris]

@knighthappy
I don't really understand why, but that works. By removing the mean from the image I can get non-NaN valued outputs.
Reminds me a lot of the eigenfaces method of facial recognition...

Can you explain your reasoning behind those steps?

[knighthappy]

@jamesFerris
The first point is when you reduce the mean of one 3-channels image,you should reduce the mean each channel.If you just subtract one value ,it just modify the value of first channel.
The second point is the python code is use misc to read image,if you use opencv to read image in c++,then these result is not the same,you should change the image's R and B channel,this is the reason why i use (2-c).
The last, result feature non-NaN may not be your purpose. I compared this code with the python one and got the same result feature for one same image.
My english is poor,if you cannot understand what i am saying , i can explain more...

[jamesFerris]

@knighthappy
I'm mainly wondering about the first point, as that's the one which solved my issue (for some reason my output tensor would only consist of NaN values). Although I don't understand why removing the mean would solve my problem, it does. I want to understand why you thought to remove the mean in the first place and what you were aiming to achieve by it, which will hopefully shed light on whats going on.

[knighthappy]

@jamesFerris
It just to do the prewhiten, just as the function prewhiten(x) in facenet.py.

[jamesFerris]

@knighthappy
Then I'm absolutely at loss as to why that changes anything.

Regardless, this does solve the original issue and everything is working as intended, so I suppose I should finally close this.

Thank you everyone for your assistance. If I ever work out exactly what was going on I'll give an update.

[ashokbugude]

Can I please know how to avoid nan values for .jpg images, haven't tried with .png images though.
All the values in output are nan

[jamesFerris]

@ashokbugude
The only "solution" found was to remove the mean of the image from itself in a pre-processing phase. I still can't say why this worked to avoid nan values, it just did (which is a little frustrating). There shouldn't really be much difference for jpg or png images, as long as you're reading them correctly.

[jamesFerris]

So I've finally jumped back into this little project after a long distraction. Which meant I'd forgotten why'd I'd done anything, so needed to start looking at things from the ground up. This means I'm pretty sure I've worked out why I was getting the NaN values, and why adding in the prewhiten() step solves it. It was pretty fundamental, and I'm annoyed I didn't pick up on it the first time.

All Inception graphs require the input matrices to have normalised values, [-1,1]. The prewhiten() step does this.

[Kabariya]

http://ruishu.io/2016/12/27/batchnorm/

it will give you an answer... for what is phase_train and when to keep it false and when to true.

[HackersSpirit]

@mndar : I tried to run your above program for single image and i am getting the following error:
Where run command:
Running model failed: Not found: FeedInputs: unable to find feed output input:0