Advice for face alignment

[kizilkanat]

Hi @DuinoDu ,
I need a Python version of the following matlab routine, which is provided by Yandong Wen to perform a 2-D face alignment using the 5 facial points obtained by MTCNN:

% load face image, and align to 112 X 96
imgSize = [112, 96];
coord5points = [30.2946, 65.5318, 48.0252, 33.5493, 62.7299; ...
                51.6963, 51.5014, 71.7366, 92.3655, 92.2041];

image = imread('path_to_image/Jennifer_Aniston_0016.jpg');
facial5points = [105.8306, 147.9323, 121.3533, 106.1169, 144.3622; ...
                 109.8005, 112.5533, 139.1172, 155.6359, 156.3451];

Tfm =  cp2tform(facial5points', coord5points', 'similarity');
cropImg = imtransform(image, Tfm, 'XData', [1 imgSize(2)],...
                                  'YData', [1 imgSize(1)], 'Size', imgSize);

I have already tried some code with OpenCV and skimage, using the points obtained by your code; but no success:

import numpy as np
import cv2

imgSize = (112, 96)

# given a dimension size, calculate the center coordinate
calc_center_coord = lambda x: np.float32(x-1)/2 if x % 2 == 0 else np.float32((x-1)/2)

# calculate normalized coordinates
calc_norm_coord = lambda x,center,scale: (x-center)/scale

x_ = [30.2946, 65.5318, 48.0252, 33.5493, 62.7299]
y_ = [51.6963, 51.5014, 71.7366, 92.3655, 92.2041]
xc, yc = calc_center_coord(imgSize[1]), calc_center_coord(imgSize[0])
x_norm = [calc_norm_coord(x, xc, imgSize[1]) for x in x_]
y_norm = [calc_norm_coord(y, yc, imgSize[0]) for y in y_]

src = np.array( zip(x_norm,y_norm) ).astype(np.float32).reshape(1,5,2)
 
w, h = img.shape[1], img.shape[0]
img_c_x, img_c_y = calc_center_coord(img.shape[1]), calc_center_coord(img.shape[0])

# there might be more than one faces, hence
# multiple sets of points
for pset in points:
    img2 = img.copy()

    pset_x = pset[0:5]
    pset_y = pset[5:10]

    pset_norm_x = [calc_norm_coord(x,img_c_x,imgSize[1]) for x in pset_x]
    pset_norm_y = [calc_norm_coord(y,img_c_y,imgSize[0]) for y in pset_y]

    dst = np.array( zip(pset_norm_x,pset_norm_y) ).astype(np.float32).reshape(1,5,2)
    
    transmat = cv2.estimateRigidTransform( src, dst, False )

    out = cv2.warpAffine(img2, transmat, (w, h))

    cv2.imshow("result", out)
    cv2.waitKey(0)

output looks totally wrong:

[kizilkanat]

This worked fine:

import numpy as np
import cv2

imgSize = (112, 96)

x_ = [30.2946, 65.5318, 48.0252, 33.5493, 62.7299]
y_ = [51.6963, 51.5014, 71.7366, 92.3655, 92.2041]

src = np.array( zip(x_, y_) ).astype(np.float32).reshape(1,5,2)

alignedFaces = []

# there might be more than one faces, hence
# multiple sets of points
for pset in points:
    img2 = img.copy()

    pset_x = pset[0:5]
    pset_y = pset[5:10]

    dst = np.array( zip(pset_x, pset_y) ).astype(np.float32).reshape(1,5,2)
    
    transmat = cv2.estimateRigidTransform( dst, src, False )

    out = cv2.warpAffine(img2, transmat, (imgSize[1], imgSize[0]))

    alignedFaces.append(out)

[DuinoDu]

Good for you.

[xizi]

Hi @DuinoDu ,
I found transmat = cv2.estimateRigidTransform( dst, src, False ) sometimes return None when the face is side face. But transf = cp2tform(coord5point_ori, coord5point_fixed, 'similarity') can work well. Could you give some help?

[DuinoDu]

Hi, @xizi. Where do you find "transmat = cv2.estimateRigidTransform( dst, src, False ) "?
Can you provide url?

[xizi]

It is kizilkanat provide it from above and i have solved this problem by use "transmat = cv2.estimateRigidTransform( dst, src, True ) ". Thanks for your reply.

[DuinoDu]

Good for you.

[nttstar]

you can refer to my article: here

[DL-85]

It's very helpful! Thanks, nttstar.

[cannguyen275]

Hey guys, I just saw @nttstar have 25% better results with skimage SimilarityTransform than OpenCV estimateRigidTransform.

That's insane! The aligned images from 2 methods totally looks the same.
Does skimage do any difference things with its function, compare with Opencv?

I do want to use skimage SimilarityTransform in C++, Java also but skimage doesn't support. Do you guys know any which can use in C++, Java but strong liked skimage?