[ ⭐] How to change lip color using Mediapipe?

[AdityaVishwakarmaIN]

Hi all, I need help with changing the lip color of a person in a video using Mediapipe. I've used Mediapipe for facial landmark detection and tracking, but I'm not sure how to proceed with changing the lip color. I couldn't find any resources on how to achieve this in the Mediapipe documentation. Can someone please provide me with some guidance on how to do this? Thank you!

[AdityaVishwakarmaIN]

I can outline lips using facemesh, but I need assistance to fill in color with controllable transparency in them. Plus no bold lines for outlining also.

[fadiaburaid]

This has to do more with OpenCV than Mediapipe. You might want to search for how to fill a polygon using cv2.fillPoly. You will need the landmarks to define the contour you can refer to this image here to find which landmarks.

[kuaashish]

@techy-keeda,
Would you please let us know the platform using such as Python, Android, JavaScript or other available? Thank you!

[AdityaVishwakarmaIN]

I'm using Python and OpenCV. Running the code on Google Colab.
I did try the method suggested by @fadiaburaid but the result was not up to the mark. The polygons seems to dance as the coords detected by Mediapipe were continuously changing and the polygons drawn on the image seemed visibly heterogenous. I tried feathering, but it didn't bring quality of results to an acceptable level.

Any suggestion to improve and stabilize the polygon blending are welcome!!

Face Cropping

from google.colab.patches import cv2_imshow
import cv2
import mediapipe as mp
mp_face_detection = mp.solutions.face_detection
face_detection = mp_face_detection.FaceDetection()
image = cv2.imread('/content/wallpaper.png')
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
results = face_detection.process(image)
face = results.detections[0]
x1 = int(face.location_data.relative_bounding_box.xmin * image.shape[1])
y1 = int(face.location_data.relative_bounding_box.ymin * image.shape[0])
x2 = int(x1 + face.location_data.relative_bounding_box.width * image.shape[1])
y2 = int(y1 + face.location_data.relative_bounding_box.height * image.shape[0])
size = max(x2 - x1, y2 - y1)
center_x = (x1 + x2) // 2
center_y = (y1 + y2) // 2
x1_square = center_x - size // 2
y1_square = center_y - size // 2
x2_square = x1_square + size
y2_square= y1_square + size
square_face_region=image[y1_square:y2_square,x1_square:x2_square]
resized_image=cv2.resize(square_face_region,(480,480))
resized_image_bgr = cv2.cvtColor(resized_image, cv2.COLOR_RGB2BGR)
cv2.imwrite('resized_image.jpg', resized_image_bgr)

Mask Generation

import itertools
import numpy as np
mp_face_mesh = mp.solutions.face_mesh
face_mesh = mp_face_mesh.FaceMesh( static_image_mode=True,refine_landmarks=True,min_detection_confidence=0.5)
image = resized_image_bgr
upper_new = [0,267,269,270,409,291,375,321,405,314,17,84,181,91,146,61,185,40,39,37]
lower_new = [13,312,311,310,415,308,324,318,402,317,14,87,178,88,95,78,191,80,81,82]
results = face_mesh.process(image)
mask_upper = np.zeros(image.shape[:2], dtype=np.uint8)
for face_landmarks in results.multi_face_landmarks:
points_upper = []
for i in upper_new:
landmark = face_landmarks.landmark[i]
x = int(landmark.x * image.shape[1])
y = int(landmark.y * image.shape[0])
points_upper.append((x, y))
cv2.fillConvexPoly(mask_upper, np.int32(points_upper), 255)
mask_lower = np.zeros(image.shape[:2], dtype=np.uint8)
for face_landmarks in results.multi_face_landmarks:
points_lower = []
for i in lower_new:
landmark = face_landmarks.landmark[i]
x = int(landmark.x * image.shape[1])
y = int(landmark.y * image.shape[0])
points_lower.append((x, y))
cv2.fillPoly(mask_lower, np.int32([points_lower]), 255)
mask_diff = cv2.subtract(mask_upper, mask_lower)
kernel=cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(5 ,5))
mask_diff=cv2.morphologyEx(mask_diff,cv2.MORPH_OPEN,kernel)
mask_diff=cv2.morphologyEx(mask_diff,cv2.MORPH_CLOSE,kernel)
cv2_imshow(mask_diff)

Mask Blending

mask_diff_3ch = cv2.cvtColor(mask_diff, cv2.COLOR_GRAY2BGR)
image = cv2.imread('/content/resized_image.jpg')
masked_image = cv2.bitwise_and(image, mask_diff_3ch)
cv2_imshow(masked_image)
def create_colored_mask(hex_color, shape):
rgb_color = tuple(int(hex_color[i:i+2], 16) for i in (0, 2 ,4))
colored_mask = np.zeros(shape, dtype=np.uint8)
colored_mask[:,:,0] = rgb_color[2]
colored_mask[:,:,1] = rgb_color[1]
colored_mask[:,:,2] = rgb_color[0]
return colored_mask
mask_diff_3ch = cv2.cvtColor(mask_diff,cv2.COLOR_GRAY2BGR)
hex_color = input('Enter a hex color code for your mask (e.g. FF0000 for red): ')
colored_mask = create_colored_mask(hex_color, mask_diff_3ch.shape)
masked_image = cv2.bitwise_and(colored_mask,colored_mask ,mask=mask_diff)
final_image = cv2.addWeighted(image, 1 , masked_image ,1 ,0)
cv2_imshow(final_image)

How can I make improvements to the code so that lipstick color change is seamless and the mask dances a bit less on webcam imput?

[AdityaVishwakarmaIN]

@kuaashish any suggestions?

[kuaashish]

Hi @kostyaby,
Could you please look into this issue. Thank you!

[kostyaby]

Hey @techy-keeda,

Generating the lip mask from the landmarks is an important step that you've already taken, so that's good. It looks like you've gotten pretty far already! To enhance the quality, I'd recommend doing the following:

• Smooth the segmentation mask using OpenCV gaussian blur filter (tutorial). This should help to avoid sharp edges of the segmentation mask
• When blending, avoid using the sharp mask and/or any bitwise operations on the mask - instead, use the blurred mask and treat it as if it has continuous values from 0 to 1, not just {0, 1}. This means using alpha-blending (A * (1 - alpha) + B * alpha) instead of bitwise operator like bitwise_and
• As for the virtual makeup rendering itself, it's a very complex area so don't expect to have an easy perfect solution. However, for darker colors (like red or blue) a decent blending function is "multiply" (CSS spec). In other words, if the original image pixel color is (R_i, G_i, B_i) and the color of the makeup is (R_m, G_m, B_m) then the "multiply" result would be (R_i * R_m, G_i * G_m, B_i * B_m).
• If you're looking to build a more advanced solution, there's a paper that we wrote together with my colleagues back in the days: PDF

Hopefully this reply helps you to proceed forward!

[AdityaVishwakarmaIN]

Thanks @kostyaby
will work on the recommendations and post results here

[kuaashish]

@techy-keeda,
Could you please update the result here if you have succeeded to archive the mentioned use case?

[github-actions]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No