get Blendshape Weights from the Face Landmark Sample scene

[rudrjain]

Plugin Version or Commit ID

2f37866

Unity Version

2022.3.61

Your Host OS

macOS Monterary

Target Platform

UnityEditor, Android

Description

Hi @homuler . Really thanks for the plugin that you have provided, its of great help.

I was trying the Landmarker scene that you have added recently, after building the package I was able to run the scene and get the mesh overlay over the face, But now I want to get the Blendshape weights so that I can use it on my avatar so can you please help me out or point me to the function which is responsible to expose the Blendshape weights.

My query could be naive so forgive me for that I am new to Mediapipe so your little guidance will be really helpful to me.

Also can you please give some guidance/ pointers to take care so that I can use the mediapipe face landmarker to Production on Android Devices.

Thanks in advance

Code to Reproduce the issue

No response

Additional Context

No response

[homuler]

The Face Landmark Detection scene runs FaceLandmarkerRunner, which runs FaceLandmarker internally.
The FaceLandmarker class is translated from the official FaceLandmarker API.
Since I have written documentation comments for each method, please refer to those for more detailed information.

The FaceLandmarker API returns FaceLandmarkResult.
If outputBlendshapes is set to true, FaceLandmarkResult.faceBlendshapes will contain the blendshapes value.

MediaPipeUnityPlugin/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Tasks/Vision/FaceLandmarker/FaceLandmarkerResult.cs

Line 17 in 2f37866

public readonly IReadOnlyList<Classifications> faceBlendshapes;

[rudrjain]

The Face Landmark Detection scene runs FaceLandmarkerRunner, which runs FaceLandmarker internally. The FaceLandmarker class is translated from the official FaceLandmarker API. Since I have written documentation comments for each method, please refer to those for more detailed information.

The FaceLandmarker API returns FaceLandmarkResult. If outputBlendshapes is set to true, FaceLandmarkResult.faceBlendshapes will contain the blendshapes value.

MediaPipeUnityPlugin/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Tasks/Vision/FaceLandmarker/FaceLandmarkerResult.cs

Line 17 in 2f37866

public readonly IReadOnlyList<Classifications> faceBlendshapes;

Thanks @homuler

I was able to get the BS Values but I am now facing other issue that I am not able to store the values returned by the model. I am quoting my code for your reference Please help me out if I have done something wrong

public BlendShapeData blendShapeData = new BlendShapeData();
private void OnFaceLandmarkDetectionOutput(FaceLandmarkerResult result, Image image, int timestamp)
    {
      _faceLandmarkerResultAnnotationController.DrawLater(result);
      //Debug.Log(result.faceBlendshapes);
      foreach (Classifications cls in result.faceBlendshapes)
      {
        Debug.Log("bs data = " + cls);
        string jsonString = cls.ToString();
        BSData _bsData = new BSData();
        _bsData = JsonUtility.FromJson<BSData>(jsonString);
        blendShapeData.listOfBSData.Add(_bsData);
      }
    }

Following are my Classes

[SerializeField]
public class Category
{
  public int index;
  public double score;
  public string categoryName;
  public string displayName;
}

[SerializeField]
public class BSData
{
  public List<Category> categories = new List<Category>();
  public int headIndex;
  public object headName;
}

[SerializeField]
public class BlendShapeData
{
  public List<BSData> listOfBSData = new List<BSData>();
}

[homuler]

1. ToString does not guarantee the creation of valid JSON (it attempts to resemble JSON-like format, but there might be typos or errors).
2. Serializing values to strings and then deserializing them is an extremely slow operation, and there seems to be no apparent reason to do so.