-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
If Face BlendShape is a production ready solution ? #4210
Comments
I also had a similar problem when I used this model: https://github.com/google/mediapipe/blob/master/mediapipe/tasks/testdata/vision/face_landmarker_with_blendshapes.task |
Using "https://storage.googleapis.com/mediapipe-assets/face_landmarker_with_blendshapes.task" as the task, results in the following error when I use detect. |
Many thanks for your assistance @kuaashish |
Hello @endink, @baronha, |
@yichunk Hi, thanks for your reply, sorry for late. Here is my graph:
|
It has to be said that most of the BlendShapes work well, and the problems mainly appear in the eyes. My program calibrated bs before running mediapipe.
I don't want to compare mediapipe to other things, but the same code will work fine with Nvidia AR. |
@endink hi mate, I am writing a new updated plugin for feeding iOS FaceMesh landmarks in Unity from one I did a while ago, the update being the new data that can now come through, i.e the transformation matrix and blendshapes. Is it worth taking the same approach as what you have done and manually pushing it into a legacy graph or do you think its better to wait on these new solutions? |
@creativewax Hi,I think you'd better wait google, I have done legacy solution (in Unreal Engine), but its not good enough. I also done new solution in Android, and push to unreal engine app, it's a little bit of an improvement over the legacy solution, but it's still not good enough to drive a 3D avatar. |
@endink thank you, I have explained this to my client, but they want to push ahead with the update anyway, I started putting together a new legacy graph last night, but I have not looked at the supporting bits around that. That is the main thin I was unsure about looking at the graph you put above, what is the corresponding code around it. Does the Classification BS just get passed through? |
@creativewax You can use my graph above, but you need dowload face_blendshapes.tflite file. BS is in ClassificationList. Label is bs name , value is bs weight (0-1) ,I saw google release a new version for mediapipe(0.9.3), I have not try it. A google developer let me try on another issue. I think you can try it , use C++ task api is a better choise(task api is not worked 0.9.2). |
@endink cool, ill take a look, basically all I want to do is use the same Face Mesh solution and pull the BS data into it, so atm I am unsure where to reference the face_blendshapes.tflite atm, but I saw that in the graph, so ill dig a bit deeper. I am guessing the ClassificationList. Label is exposed into the legacy FaceMesh solution? |
@endink yup, I told my client about it not being production ready and they was still really keen on getting it in, no idea why |
Just use my graph, and observe/poll "face_blendshapes" stream. 😄 |
If you want to know how mediapipe bs actually works, I recorded a video on my youtube channel: https://youtube.com/playlist?list=PL1Bnbwb6xrM-WhM0XgK6mSMpkTL73kmln At 33:20 |
@endink nice, I have had a look, but thats solely on the UE side of things, I need to understand how they stream into the iOS side of things, ill take a proper look later, thanks dude |
Hi, the updated face landmarker and face blendshapes are now ready and publicly released. and the guide here: https://developers.google.com/mediapipe/solutions/vision/face_landmarker hope this help! |
@yichunk Why C++ was abandoned? |
It is not abandoned. There is C++ API for mediapipe Tasks https://github.com/google/mediapipe/blob/master/mediapipe/tasks/cc/vision/face_landmarker/face_landmarker.h |
@yichunk Won't there be C++ documentation and code examples in the future? Is it that C++ users can only get the usage method by reading the source code |
I need C++ documentation and code examples, too. |
@yichunk facing same performance issue on even high end android devices! Is there any workaround or it's just a dead end? |
Is anyone able to crack this one? |
@emphaticaditya i have it working on iOS and Android now. I can post my graph and solution later when I am at my computer. |
It seems that Google will no longer respond to this issue, and the only can do is mix the new TaskAPI, the old solution used to do Holistic, and the TaskAPI specifically handles Face. One image send to two pipes. It is not much difference in performance. That's how I use it at this moment. |
@endink @creativewax thanks for responding!! |
I use android too, I got 30FPS (java sdk) |
@endink is it possible for you to share a screen recording by any chance? |
@creativewax Can you share the performance benchmarks? I am getting 30fps fixed with 33.8ms latency on android s10. Can anyone comment on the same regarding the performance? It would be helpful really! |
@emphaticaditya sorry for the delay, them benchmarks seem inline with what I got, and mine was piping into Unity as well, ran pretty well in the end after code optimisations etc |
There seems to be a new Mediapipe blendshape prediction model on the horizon that looks very promising. A Google Research Paper describing the new model was released in September 2023 and I have written an issue about it, linking the research paper and asking for a timeline of the new model's implementation. If this is still relevant you, please upvote the issue so that the blendshape topic gains more exposure. Thanks and greetings, Ferdinand |
I also have an error in getting Unreal Blendshape calculator. Currently, the following error occurs in the Unreal Output Log in the part of the pbtxt(endink's holistic pbtxt)in LandmarksToTensorCalculator calculator [LogMediaPipe:Error:; mediapipe::ParseTextProtomediapipe::CalculatorGraphConfig (config_str, &config)]. |
I referenced the source code of FaceBlendShapesGraph and put it into a "legacy" solution in the same way (because TaskRunner didn't work), I used the TensorsToClassificationCalculator and face_blendshapes.tflite file to caculate the ClassificationList and succeeded. After this, I brought it into Unreal Engine.
There are some things that I don't quite understand, and I hope to be able to give guidance:
In fact, the left and right eyes of the characters in these pictures are almost the same size.
Here is a part of the graph code I used to render annotation overlay:
The text was updated successfully, but these errors were encountered: