Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Questions on using the face meshes to make masks #430

Closed
mantzaris opened this issue Mar 18, 2024 · 1 comment
Closed

Questions on using the face meshes to make masks #430

mantzaris opened this issue Mar 18, 2024 · 1 comment
Assignees

Comments

@mantzaris
Copy link
Sponsor

I am using within ElectronJS:

    "@tensorflow/tfjs": "^4.17.0",
    "@tensorflow/tfjs-node": "^4.17.0",
    "@vladmandic/human": "^3.2.1",

the human config object:

const human_config = {
  backend: "webgl", // Specify the backend to use, wasm alternative
  modelBasePath: model_path,
  face: {
    enabled: true,
    detector: { rotation: true, maxDetected: 5, return: true },
    mesh: { enabled: true, return: true },
    meshRaw: { enabled: true, return: true },
    iris: { enabled: true, return: true },
    distance: { enabled: true, return: true },
    description: { enabled: true },
    emotion: { enabled: false },
  },
  gesture: { enabled: true },
};

I have a question on how to use the face meshes to create 'masks'. I can get the points of the face mesh and draw them via:

const result = await human.detect(videoElement);

  // You can access various results, e.g., face landmarks
  if (result.face.length > 0) {
    const canvas = document.getElementById("face-mesh-canvas");
    const ctx = canvas.getContext("2d");
    ctx.clearRect(0, 0, canvas.width, canvas.height);

    result.face.forEach((face) => {
      drawMesh(face.mesh, ctx); 
      //drawFilledMesh(face.mesh, ctx);
    });
...

function drawMesh(mesh, ctx) {
  const scaleX = videoElement.offsetWidth / videoElement.videoWidth;
  const scaleY = videoElement.offsetHeight / videoElement.videoHeight;

  mesh.forEach((point) => {
    const size = 2;
    const x = point[0] * scaleX; 
    const y = point[1] * scaleY; 

    ctx.fillStyle = "blue"; 
    ctx.fillRect(x, y, size, size);
  });
}

This simply produces the points but not the mask of the face showcased on the right in
https://github.com/vladmandic/human-motion/blob/main/assets/screenshot-face.jpg

  1. how can I produce that face mask?

  2. There is also an image of a 468-Point Face Mesh with an image for the point markers to help 3D rendering at
    https://github.com/vladmandic/human/blob/main/assets/facemesh.png
    On the result from Human, the face has a 'mesh' of 478 points, are we supposed to produce a new map and create a mapping to a new image is there another way with some function helpers?

  3. Would it be easier to use the annotations to create mask mappings?

4 How can a mapping to another modified mapping be possible, eg, a 'cat' face? Is there process for creating the 'mesh' with the numbering of the 'keypoints' that can then be drawn from some standard approach?

any assistance is appreciated

(sponsoring)

@vladmandic
Copy link
Owner

you've posted a link to image on human-motion repo - that repo uses human library to for analysis, but then it also uses a 3d framework babylonjs to draw a mask. and yes, human does give you helper functions for masks - they are called uvmaps.
see human-motion repo for examples, e.g. https://github.com/vladmandic/human-motion/blob/8e1ccbb9b53443360a070b2f7342b7c4501976bb/src/mesh.ts#L137

for 3d framework - i choose babylonjs in human-motion, but you could go with any of your choice, for example three.js`.

regarding mesh having 468 or 478 points, it simply depends if mesh is just a normal face mesh or its augmented with details from iris model for extra points for each eye. this is covered in docs.

btw, in the future please create a discussion topic - this is a general question, not a product issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants