Welcome to my Face Mesh App – a solution for capturing intricate facial details and landmarks with precision. This application utilizes Google's MediaPipe, recognized as state-of-the-art technology. The app employs face mesh tessellation, pinpointing key facial features like eyes, chin, mouth, and nose, crucial for accurate face identification. Having explored alternatives such as dlib and Haar cascade, MediaPipe emerged as the optimal choice, excelling in versatility across diverse face types and lighting conditions while operating seamlessly in real-time. Notably, its efficiency with minimal system resources and compatibility with mobile edge devices positions it for future opportunities. The app, executed with a single command line, offers real-time visualization of the face mesh and landmarks. Upon completion, it captures a snapshot of the final face position and records mesh coordinates in a .csv file. While the current focus is on face mesh, the captured image, when coupled with a classification model, enables skin condition evaluation. Furthermore, the face mesh can be leveraged for Augmented Reality applications, facilitating tasks like makeup application or exploring various medical benefits. Dive into the forefront of facial analysis technology with the Face Mesh App, combining precision, efficiency, and future potential in a seamless package.
Screen.Recording.2023-12-19.at.4.39.31.PM.mov
Note: This video only shows the Face Mesh tessellation because it is more plesant to see. The application also highlights landmarks such as the eyes, nose, and face oval or chin.
After installing all the dependancies, you can run the command python3 main.py to begin the application.
The ESC button will stop the application and save the results.
There is the option to -output <filename> to give the name of the output files. If nothing is given, it will use the word output for the files name.
There is a requirement file that will help install all the dependancies. Below are the main dependancies that required and installing them will install all the subdepenacies.
This application doesn't analyze any skin conditions and issues, that seems like a logical next step. It would be helpful if this app did split up the image to regions of interest as in this tutorial which would be a good first step for preprocessing the image. I did not find pretrained skin categorization model so I belive I would have to architect and train one as a next step. I did find a Dataset on Kaggle and an architecture for a CCN on Kaggle. The MediaPipe framework also has this Model Maker which can help build a classification solution. I also came across a number of papers on this topic such as (paper and paper) which can be used as inspiration for a developing a classification CNN.
- The MediaPipe landmarks did not include a nose landmark, so I had to create my own using this diagram. Click on the image to zoom-in.
- imultils is being used for the videostream. It is primarly used as a wrapper to ensure the threading of the is started and stopped properly. Details here
- There is a Warning Message that may appear in the logs depending on the device you are running the application on. This is because of the MediaPipe library here.
- Another Warning Message appears to be created by the MediaPipe library here which is based on the access to the camera.
A few tutorials I found very helpful on ramping up:
- https://mediapipe.readthedocs.io/en/latest/solutions/face_mesh.html
- https://www.assemblyai.com/blog/mediapipe-for-dummies/
After running the application and the data is saved, you can then run a second application which will process the data into a 3D visual layout. I came across this tutorial when researching the MediaPipe and thought it was fun. It could be useful to demonstrate and analyze 3d layouts of AR filters. To use this application, run the command python3 plotData.py -data <yourCsvDataFile> with a reference to the CSV file that was created. It will take a few seconds before completeing and generating a mp4 file like the one below.