`tfjs-models/face-landmarks-detection`: `keypoint` `291` missing in `MEDIAPIPE_FACE_MESH_KEYPOINTS_BY_CONTOUR.lips`

[jackbdu]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS Sonoma 14.4
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: n/a
• TensorFlow.js installed from (npm or script link): n/a
• TensorFlow.js version (use command below): n/a
• Browser version: Firefox Developer Edition 125.0b3 (64-bit)
• Tensorflow.js Converter Version: n/a

Describe the current behavior
keypoint 291 (a corner of the lips, see mesh_map) is not present in MEDIAPIPE_FACE_MESH_KEYPOINTS_BY_CONTOUR.lips

Describe the expected behavior
keypoint 291 should be included in MEDIAPIPE_FACE_MESH_KEYPOINTS_BY_CONTOUR.lips

Standalone code to reproduce the issue
Lips corner not rendered correctly in face-landmarks-detection MediaPipe demo

Other info / logs
I suspect the issue comes from constants.ts. See snippets below:

lips: connectionsToIndices(LIPS_CONNECTIONS),

function connectionsToIndices(connections: PairArray) {
  const indices = connections.map(connection => connection[0]);
  indices.push(connections[connections.length - 1][1]);
  return indices;
}

const LIPS_CONNECTIONS: PairArray = [
  [61, 146],  [146, 91],  [91, 181],  [181, 84],  [84, 17],   [17, 314],
  [314, 405], [405, 321], [321, 375], [375, 291], [61, 185],  [185, 40],
  [40, 39],   [39, 37],   [37, 0],    [0, 267],   [267, 269], [269, 270],
  [270, 409], [409, 291], [78, 95],   [95, 88],   [88, 178],  [178, 87],
  [87, 14],   [14, 317],  [317, 402], [402, 318], [318, 324], [324, 308],
  [78, 191],  [191, 80],  [80, 81],   [81, 82],   [82, 13],   [13, 312],
  [312, 311], [311, 310], [310, 415], [415, 308],
];

In particular, connections.map(connection => connection[0]); eliminates keypoint 291 from the indices.
This happens because 291 appears as the second element in both [375, 291] and [409, 291] whereas 61 (the other corner of the lips) appears as the first element in both [61, 146] and [61, 185].

Hope this helps!

[lindapaiste]

There are two other keypoints which will be missing for the same reason.

Part: lips is missing indices: 291
Part: leftEye is complete.
Part: leftEyebrow is missing indices: 285
Part: leftIris is complete.
Part: rightEye is complete.
Part: rightEyebrow is missing indices: 55
Part: rightIris is complete.
Part: faceOval is complete.

It seems like the expectation is that the end of one pair will be the start of the next pair, so I ran a quick code to find instances where this is not the case. Not all of these "jumps" lead to missing indices. For the left/right eye the index which would potentially be left off (362/ 133) also appears at the end of the last connection so it gets included there. Same for index 308 in the lips.

Part: lips
Connection jump at index 10: 291 -> 61 between pairs [375,291] and [61,185].
Connection jump at index 20: 291 -> 78 between pairs [409,291] and [78,95].
Connection jump at index 30: 308 -> 78 between pairs [324,308] and [78,191].
Part: leftEye
Connection jump at index 8: 362 -> 263 between pairs [382,362] and [263,466].
Part: leftEyebrow
Connection jump at index 4: 285 -> 300 between pairs [295,285] and [300,293].
Part: leftIris
All okay
Part: rightEye
Connection jump at index 8: 133 -> 33 between pairs [155,133] and [33,246].
Part: rightEyebrow
Connection jump at index 4: 55 -> 70 between pairs [65,55] and [70,63].
Part: rightIris
All okay
Part: faceOval
All okay

Code I used for both checks:

const parts = {
  lips: LIPS_CONNECTIONS,
  leftEye: LEFT_EYE_CONNECTIONS,
  leftEyebrow: LEFT_EYEBROW_CONNECTIONS,
  leftIris: LEFT_IRIS_CONNECTIONS,
  rightEye: RIGHT_EYE_CONNECTIONS,
  rightEyebrow: RIGHT_EYEBROW_CONNECTIONS,
  rightIris: RIGHT_IRIS_CONNECTIONS,
  faceOval: FACE_OVAL_CONNECTIONS,
};

Object.entries(parts).forEach(([part, connections]) => {
  const allIndices = MEDIAPIPE_FACE_MESH_KEYPOINTS_BY_CONTOUR[part];
  const omissions = connections
    .flat()
    .filter((index) => !allIndices.includes(index));
  if (omissions.length > 0) {
    console.log(`Part: ${part} is missing indices: ${omissions.join(", ")}`);
  } else {
    console.log(`Part: ${part} is complete.`);
  }
});

Object.entries(parts).forEach(([part, connections]) => {
  const jumps = connections
    .map((current, i) => {
      const previous = connections[i - 1];
      if (previous && current[0] !== previous[1]) {
        return `Connection jump at index ${i}: ${previous[1]} -> ${current[0]} between pairs [${previous}] and [${current}].`;
      }
      return;
    })
    .filter(Boolean);
  console.log(`Part: ${part}`);
  if (jumps.length > 0) {
    console.log(jumps.join("\n"));
  } else {
    console.log("All okay");
  }
});

[gaikwadrahul8]

Hi, @jackbdu, @lindapaiste

We sincerely apologize for the delay in our response and thank you for bringing this issue to our attention, I tried face-landmarks-detection MediaPipe demo and it seems like there are total 03 missing keypoints so our relavant team will look into this issue and I believe they'll fix this issue soon

In the meantime, we welcome contributions from the community. If you have a solution to address these missing keypoints, please feel free to submit a pull request (PR). Our team will thoroughly review your PR and take appropriate action, including merging it if it meets our guidelines.

Thank you for your cooperation and patience.

[nileshau-afk]

Hello @jackbdu ,

Thank you for bringing this to our attention and for providing a comprehensive overview. We appreciate the time you invested in creating this issue.

Currently, our internal development team is concentrated on other scheduled milestones, so we cannot prioritize this task immediately. However, we would love to see this move forward as a community-led effort.

If you’d like to submit a fix, please fork the repo and open a PR following the TensorFlow Contribution Guidelines, Tfjs Contributions and be sure to link this issue in your PR to avoid duplicate work, kindly leave a comment here if you plan to take this on.

Thank you again for your interest and time.