Multi-view face landmark extraction

[nitheeshas]

The output screenshots seem impressive, especially the multi view landmark extraction, but I can't figure out how to run it from the code you have provided. Please help.

[uricamic]

Hi @nitheeshas,

do you want C++ example, MATLAB or Python?

[nitheeshas]

Thanks for replying. I would like to see an example in C++.

[uricamic]

Ok, I will skip the face detector part, since I now have code only using a commercial one. However, it is possible to use combination of OpenCV haarcascades for frontal and profile faces.

Lets assume that we have the bbox of the face in the image (the format of bbox is described e.g. here: #20 ). The function which jointly detects the discretized yaw angle and landmarks looks like this:

void jointmv_detector(Flandmark **flandmarkPool, int *bbox, int *viewID)
{
    const int PHIS = 5;
    fl_double_t scores[PHIS];
    fl_double_t maximum = -INFINITY;

    for (int phi=0; phi < PHIS; ++phi)
    {
        Flandmark *flandmark = flandmarkPool[phi];
        flandmark->detect_optimizedFromPool(bbox);

        // compute score
        scores[phi] = flandmark->getScore();

        if (scores[phi] > maximum)
        {
            maximum = scores[phi];
            *viewID = phi;
        }
    }
}

the viewID serves as a pointer to flandmarkPool, so we can later extract landmarks and view label.

Now how to initialize the flandmarkPool. Lets assume we have a following .txt file:

./models/PART_fixed_JOINTMV_-PROFILE.xml
./models/PART_fixed_JOINTMV_-HALF-PROFILE.xml
./models/PART_fixed_JOINTMV_FRONTAL.xml
./models/PART_fixed_JOINTMV_HALF-PROFILE.xml
./models/PART_fixed_JOINTMV_PROFILE.xml

Then we can use the following function to parse it

std::vector<std::string> readModelList(const char *file)
{
    std::vector<std::string> out;
    std::ifstream infile;
    infile.open(file);
    std::string line;
    while (std::getline(infile, line))
    {
        out.push_back(line);
    }

    return out;
}

So, in the main function we can use it as follows:

// read models from a text file
std::vector<std::string> models = readModelList(argv[3]);
Flandmark *flandmarkPool[models.size()];    // pool of Flandmark instances

for (int i=0; i < models.size(); ++i)
{
    flandmarkPool[i] = Flandmark::getInstanceOf(models[i].c_str());

    if (!flandmarkPool[i])
    {
        cerr << "Couldn't create instance of flandmark with model " << models[i] << endl;
        return -1;
    }
}
tim = timer.toc();

const int * bw_size = flandmarkPool[0]->getBaseWindowSize();
CFeaturePool * featuresPool = new CFeaturePool(bw_size[0], bw_size[1]);
featuresPool->addFeaturesToPool(
            new CSparseLBPFeatures(featuresPool->getWidth(),
                                   featuresPool->getHeight(),
                                   featuresPool->getPyramidLevels(),
                                   featuresPool->getCumulativeWidths()
                )
            );

for (unsigned int i=0; i < models.size(); ++i)
{
        flandmarkPool[i]->setNFfeaturesPool(featuresPool);
}

this initializes flandmarkPool (view dependent instances of Flandmark with corresponding models loaded) and featuresPool (the helper structure, which shares precomputed features among Flandmark instances).

Prior calling jointmv_detector function, do not forget to do this

featuresPool->updateNFmipmap(featuresPool->getWidth(), featuresPool->getHeight(), flandmarkPool[0]->getNF(frm_gray, &bbox[0])->data());

where cimg_library::CImg* frm_gray is supposed to be filled with the grayscale input image. This initiates feature computation in featuresPool a necessary step for the function jointmv_detector to work properly.

[nitheeshas]

Thanks a lot for the detailed explanation!
I had doubt regarding the face detector too for multi view since opencv's profile face detector gave only an average result. I saw in the website that you were using Eyedea face detector. Their face detection seems to be almost perfect.
Anyway, I'll try this out right away. Thanks again!

[uricamic]

Yeah, Eyedea face detector is performing really well. It implements this paper, if you would like to re-implement it.
I guess another option is to re-train the OpenCV profile detector.

[nitheeshas]

Wow, waldboost? Its actually already implemented by someone. Its available in opencv-contrib. I'll try to train it and check how good it performs.

[nitheeshas]

Hi @uricamic
I was able to build the multi view landmark extraction using Dlib's face detector. I used the jointly learned landmarks pool. But the extracted landmarks are not that proper. Is it a known problem?

[uricamic]

Hi @nitheeshas,

the models currently available are learned on a very limited training set. We are currently learning them on a bigger database.
It is also possible that since the search spaces are shrinked (in order to get the detector as fast as possible), the dlib's face detector should be corrected to match the expectations for the face detector used in training.
Hard to tell without seeing some examples, though.

[nitheeshas]

I've uploaded an example demo video of the outputs I got. Please check.
https://www.youtube.com/watch?v=25dbq7KSLsI

Sorry for the poor quality!

[uricamic]

It seems that the face detection is really suffering a huge variance in scale and position. On the other hand, when it is as one would expect, it looks quite nice, I would say.

One quick suggestion which should improve the accuracy a lot is to stabilize the face detector output by e.g. Kalman filtering.

The new models should also improve the quality a lot, however, they are not yet fully learned.

[nitheeshas]

Yes, I just started modifying the code for Kalman filter now. Will update how it works :)

[nitheeshas]

@uricamic I was not able to add kalman filter since i got caught up with some other work.
But the thing is, while testing the previous output, even when i was standing perfectly still, and the face detector output was also pretty much constant, the detected landmarks kept jumping a lot.

Maybe the best solution for this problem is to build fully learned models as you said. Are you still working on creating better models?

[uricamic]

@nitheeshas, I think in such case the problem is with a noise in the webcam input. The new models should help a bit, but depending on how severe the noise is.

New version should be learned within few days, the biggest benefit should be the better yaw estimation precision and I hope to some extent also the landmark localization accuracy. However, the accuracy is limited due to relatively small normalized frame. The idea is to have this detector as an initial phase and then for precise landmark detection or tracking use better model (either with increased size of normalized frame or using regression, to remove the systematic error introduced by transforming landmarks from the normalized frame back to image).

[nitheeshas]

In that case, it will be better to wait for the new learned model and if its still shaky, will add a Kalman filter and check again.

Hope you'll update soon.

[nitheeshas]

Hi @uricamic Can you share the dataset which you are using to train the multi view landmark detection?

[uricamic]

Hi @nitheeshas,

we are still working on that. Maybe, some smaller portion of examples could be published soon. Sorry for the delay.

[mousomer]

@uricamic : A small question:

You suggest adding a call to updateNFmipmap prior to detect_optimized.
But the latter function already includes a call to updateNFmipmap. And the static_input example you supply does not have that independent call to updateNFmipmap and yet it seems to give good results nonetheless.

Also, since the CSparseLBPFeatures class inside the CFeaturePool is protected, this cannot be done on an image-by-image basis, but only during the CFeaturePool initialization. If this is indeed a critical stage, then you should add an init_CSparseLBPFeatures function to the CFeaturePool class.

[uricamic]

Hi @mousomer,

I think there is some misunderstanding. The updateNFmipmap method of CFeaturePool is needed if you want to call the detection on multiple images. You simply exhange the image on which the detection is performed. Without costly re-initialization of the objects. Btw, the static_input example is also using it (see here).

The features are computed automatically inside the CFeaturePool class, when you call this updateNFmipmap method (see here), user is not supposed to interfere in features computation anyhow.

Maybe the names of some methods are a bit confusing, I am sorry if it is the case. However, all the important functionality is there and working. Some methods are there also just because of the MATLAB interface, especially for the purpose of the model learning, where the speed is very important.

[mousomer]

Well, you pointed into the detect_optimized function, which I suppose is the main API for extracting the features. So, am I correct in understanding that I don't need an extra call to updateFmipmap before I call detect_optimized?

[uricamic]

Hi @mousomer,

yes, for detect_optimized you really do not need to do that extra call of updateFmipmap.

However, check the post, where I was suggesting this call. It was for the jointmv_detector, which is internally calling detect_optimizedFromPool. Then, you have to call updateFmipmap prior calling the detector, because in that case there is no other way how to update the image and let the features to be computed. The reason why it is so is simply because there are multiple detectors to run, and the landmarks of the detector which has the maximal response are returned. The features are computed just once per face image and used in all detectors.

[mousomer]

I see. Thanks!
Oh, and if I haven't mentioned it before - this package is really awesome.

[uricamic]

@mousomer
No problem, it is always good to ask questions ;-)

Thanks!

[mousomer]

I re-run my sample set with optimizedFromPool instead of detect_optimized.
Got exactly the same results.
And the score is always biased towards the NegativeHalfProfile.
I've run a few thousand examples. This is the statistics I'm seeing:

	Frontal	NegProf	NefHProf	PosHProf	PosProf
Mean score	1.076	1.363	2.995	1.818	-1.523
StdDev score	0.064	0.046	0.065	0.062	0.049

[uricamic]

Hi @mousomer,

thank you for reporting this. The values you show seem to be a bit suspicious, I would expect highest score for the frontal views, since those have the highest number of landmarks.
Maybe there is a bias term missing in the code sample. I will check it soon and come back with an answer.

[mousomer]

@uricamic I've tested a few of the images. Seems that when translating scores to Z-score (subtracting means, dividing by standard deviation), the best z-score does yield the best model match. I need to verify this on a large batch of images.

[mousomer]

@uricamic
I did testing with NIST face set 18 - which has right and left face profiles:
https://catalog.data.gov/dataset/nist-mugshot-identification-database-mid-nist-special-database-18

The scoring is still bad. Even when translating into z-scores or tail scores, the results are not good.
So, basically, I need some external reference software to decide on the right model (frontal, R/L profile or half profile).

[uricamic]

Hi @mousomer,

no z-score translation should be needed. I will try to check on the database you mention and share the code with you. I hope I can manage it within a week, cannot guarantee that though.

[mousomer]

I'm adding a sniplet of the code I'm using – in case there's a problem with it. From: Michal Uřičář [mailto:notifications@github.com] Sent: Tuesday, June 20, 2017 9:02 AM To: uricamic/clandmark <clandmark@noreply.github.com> Cc: Omer Moussaffi <mousomer@gmail.com>; Mention <mention@noreply.github.com> Subject: Re: [uricamic/clandmark] Multi-view face landmark extraction (#38) Hi @mousomer <https://github.com/mousomer> , no z-score translation should be needed. I will try to check on the database you mention and share the code with you. I hope I can manage it within a week, cannot guarantee that though. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#38 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/ABLqJHt3l9nIESklnN42M6oSQZ1W6NLhks5sF2BQgaJpZM4HTbTk> . <https://github.com/notifications/beacon/ABLqJGuSHu-k1SIT37KBnrOwkCJNe83Oks5sF2BQgaJpZM4HTbTk.gif> inline void run_model_fromPool(Flandmark *model, CFeaturePool* c_featuresPool, facial_features_clandmark &ff_landmark, cimg_library::CImg<unsigned char> *frameCImg, const string baseName, const string c_extension, bool crop_faces = false, bool allocate_model = false ) { CFeaturePool * featuresPoolModel; if (allocate_model) { const int * bw_size = model->getBaseWindowSize(); featuresPoolModel = new CFeaturePool(bw_size[0], bw_size[1]); featuresPoolModel->addFeaturesToPool( new CSparseLBPFeatures(featuresPoolModel->getWidth(), featuresPoolModel->getHeight(), featuresPoolModel->getPyramidLevels(), featuresPoolModel->getCumulativeWidths())); model->setNFfeaturesPool(featuresPoolModel); featuresPoolModel->updateNFmipmap(featuresPoolModel->getWidth(), featuresPoolModel->getHeight(), model->getNF(frameCImg, ff_landmark.bbox_init)->data()); } // Run the Clandmark model detector c_featuresPool->updateNFmipmap(c_featuresPool->getWidth(), c_featuresPool->getHeight(), model->getNF(frameCImg, ff_landmark.bbox_init)->data()); model->detect_optimizedFromPool(ff_landmark.bbox_init); //model->detect_optimized(c_jpg->GetDataY(), c_jpg->Width(), c_jpg->Height(), ff_landmark.bbox_init); // Get score ff_landmark.score = model->getScore(); // Get detected landmarks fl_double_t *c_landmarks = model->getLandmarks(); // Show landmarks string outCName = baseName + "." + c_extension; fprintLandmarks(outCName, c_landmarks, model->getLandmarksCount(), false); // crop faces if (crop_faces) { outCName = baseName + "_" + c_extension; draw_face(outCName, ff_landmark.index, c_landmarks, model->getLandmarksCount(), ff_landmark.bbox_init); /* rectangle(frame_gray, Point(bbox[0], bbox[1]), Point(bbox[2], bbox[5]), Scalar(255, 0, 0)); circle(frame_gray, Point(int(landmarks[0]), int(landmarks[1])), 2, Scalar(255, 0, 0), -1); for (int i=2; i < 2*flandmark->getLandmarksCount(); i+=2) circle(frame_gray, Point(int(landmarks[i]), int(landmarks[i+1])), 2, Scalar(0, 0, 255), -1); */ } // Cleanup if (allocate_model) delete featuresPoolModel; } int main (int argc, char *argv[]) { Flandmark *CDPM = init_model("CDPM.xml"); Flandmark *FDPM = init_model("FDPM.xml"); string FRONT_MODEL = GetRunPath () + "../../" +modelDir + "PART_fixed_JOINTMV_FRONTAL.xml"; Flandmark *AFLW_F = Flandmark::getInstanceOf (FRONT_MODEL.c_str ()); CFeaturePool* AFLW_featuresPool = init_feature_pool(AFLW_F); AFLW_F->setNFfeaturesPool(AFLW_featuresPool); cout<<"initialize "<<FRONT_MODEL<<" model "<< AFLW_featuresPool->getWidth() <<" x "<< AFLW_featuresPool->getHeight() <<endl; Flandmark *AFLW_PHP = init_model(AFLW_featuresPool, "PART_fixed_JOINTMV_HALF-PROFILE.xml"); Flandmark *AFLW_NHP = init_model(AFLW_featuresPool, "PART_fixed_JOINTMV_-HALF-PROFILE.xml"); Flandmark *AFLW_PP = init_model(AFLW_featuresPool, "PART_fixed_JOINTMV_PROFILE.xml"); Flandmark *AFLW_NP = init_model(AFLW_featuresPool, "PART_fixed_JOINTMV_-PROFILE.xml"); string AFLW_extentions[5] = { "Frontal", "PosHProf", "NegHProf", "PosProf", "NegProf" }; Flandmark * FL_JA[5] = {AFLW_F, AFLW_PHP, AFLW_NHP, AFLW_PP, AFLW_NP}; string JDPM_extentions[5] = { "CDPM", "FDPM"}; Flandmark * C2F_DPM[2] = {CDPM, FDPM}; c_prof.Add(pf_classesF, 7); if (!CDPM || !FDPM || !AFLW_F || !AFLW_PHP || !AFLW_NHP || !AFLW_PP || !AFLW_NP ) { cerr << "Couldn't create instance of models. check model path "<<modelDir<< endl; return -1; } // get input image, crop faces run_faces(AFLW_featuresPool, FL_JA, 5, input_basename, AFLW_extentions, true); // for CDPM // run_faces(CDPM, FDPM, c_org.FoldersProcess[0] + "/" + c_org.c_TaskName); delete AFLW_featuresPool; delete AFLW_F; delete AFLW_NHP; delete AFLW_NP; delete AFLW_PHP; delete AFLW_PP; delete CDPM; delete FDPM; delete c_jpg; std::cout<<"job summary:"<<std::endl; return 0; } int run_faces(CFeaturePool* c_featuresPool, Flandmark* land_array[], int land_number, string baseName, string c_extensions[], bool crop_faces = false) { cimg_library::CImg<unsigned char> *frameCImg; vector<facial_features_clandmark> ff_landmarks = prepare_input(baseName, frameCImg); if (ff_landmarks.empty()) { cout << "no faceV elements" << endl; return 0; } cout << __FUNCTION__<<" Json load done. loaded: " << ff_landmarks.size() << endl; for (int i_land=0; i_land<land_number; ++i_land) { for (size_t i_f = 0; i_f < ff_landmarks.size(); ++i_f) run_model_fromPool(land_array[i_land], c_featuresPool, ff_landmarks[i_f], frameCImg, baseName, c_extensions[i_land], crop_faces, false); printoutFaces(baseName + "_" + c_extensions[i_land], ff_landmarks); } delete frameCImg; return 0; } inline void run_model_fromPool(Flandmark *model, CFeaturePool* c_featuresPool, facial_features_clandmark &ff_landmark, cimg_library::CImg<unsigned char> *frameCImg, const string baseName, const string c_extension, bool crop_faces = false, bool allocate_model = false ) { CFeaturePool * featuresPoolModel; if (allocate_model) { const int * bw_size = model->getBaseWindowSize(); featuresPoolModel = new CFeaturePool(bw_size[0], bw_size[1]); featuresPoolModel->addFeaturesToPool( new CSparseLBPFeatures(featuresPoolModel->getWidth(), featuresPoolModel->getHeight(), featuresPoolModel->getPyramidLevels(), featuresPoolModel->getCumulativeWidths())); model->setNFfeaturesPool(featuresPoolModel); featuresPoolModel->updateNFmipmap(featuresPoolModel->getWidth(), featuresPoolModel->getHeight(), model->getNF(frameCImg, ff_landmark.bbox_init)->data()); } // Run the Clandmark model detector c_featuresPool->updateNFmipmap(c_featuresPool->getWidth(), c_featuresPool->getHeight(), model->getNF(frameCImg, ff_landmark.bbox_init)->data()); model->detect_optimizedFromPool(ff_landmark.bbox_init); // Get score ff_landmark.score = model->getScore(); // Get detected landmarks fl_double_t *c_landmarks = model->getLandmarks(); // Show landmarks string outCName = baseName + "." + c_extension; fprintLandmarks(outCName, c_landmarks, model->getLandmarksCount(), false); // crop faces if (crop_faces) { outCName = baseName + "_" + c_extension; draw_face(outCName, ff_landmark.index, c_landmarks, model->getLandmarksCount(), ff_landmark.bbox_init); } // Cleanup if (allocate_model) delete featuresPoolModel; }