Multiple issue facing with Face Recognization.

[mManishTrivedi]

Thanks for this pkg. It would be really helpful for lots of developers. I appreciate your efforts. I am using your pkg in my activity. Unfortunately, I am facing some issues which are listed below.

My Codebase:

Loading Image:

const buffer = fs.readFileSync(image_path);
const decoded = tf.node.decodeImage(buffer);
const casted = decoded.toFloat();
const loadedImage = casted.expandDims(0);
decoded.dispose(); //release from memory
casted.dispose(); //release from memory

Getting Descriptors:

const result = await FaceAPI
                 .detectAllFaces(loadedImage, new FaceAPI.SsdMobilenetv1Options({ minConfidence: 0.5 }))
                .withFaceLandmarks()
                .withFaceDescriptors()

Issue-1: Sometimes, I feel it's creating issue with PNG images. I don't know why? Any comment plz.
Issue-2: Some images don't have multiple faces, but still getting multiple descriptors. (let me know where I can share image.)
Issue-3: Some images have face but still get no descriptors. I have change value "minConfidence" and it's working.How to recognize right value?
Issue-4: Getting below error while executing ".withFaceDescriptors()" method for specific image. it's seems that is just configuration fine-tuning, but I am unable to identify the configuration for it. + Why it's showing UnhandledPromiseRejectionWarning while I am using try-catch in my codebase.

(node:6042) UnhandledPromiseRejectionWarning: Error: Invalid TF_Status: 3
Message: Input to reshape is a tensor with 1048576 values, but the requested shape has 786432
at NodeJSKernelBackend.executeSingleOutput (/FR/code/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:209:43)
at Object.kernelFunc (/home/manish/Desktop/MIND/FR/code/node_modules/@tensorflow/tfjs-node/dist/kernels/Reshape.js:33:27)
at kernelFunc (/FR/code/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3139:32)
at /FR/code/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3203:110
at holdResultWrapperFn (/FR/code/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:1628:23)
at NodeJSKernelBackend. (/FR/code/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:510:17)
at step (/FR/code/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:61:23)
at Object.next (/FR/code/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:42:53)
at /FR/code/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:36:71
at new Promise ()

Environment

• Ubuntu 18.04 with Node-v12.18.1
• "@tensorflow/tfjs-node": "^3.3.0"
• "@vladmandic/face-api": "^1.0.2"
• "face-api.js": "^0.22.2"

[vladmandic]

Issue-1: Sometimes, I feel it's creating issue with PNG images.

Without details, I don't know either

But you can try loading image using separate module instead of
relying on tfjs-node built-in decodeImage method

That is anyhow more common as then you can use that model to actually
draw on image and save it back to diskif desired and not just load it

For example:

// you first have to install canvas module with `npm install canvas`
const canvas = require('canvas');

const img = canvas.loadImage(inputFile);
const c = canvas.createCanvas(img.width, img.height);
const ctx = c.getContext('2d');
ctx.drawImage(img, 0, 0, img.width, img.height);

const result = await faceapi.detectAllFaces(c, new faceapi.SsdMobilenetv1Options({ minConfidence, maxResults }));
...

Issue-2: Some images don't have multiple faces, but still getting multiple descriptors.
Issue-3: Some images have face but still get no descriptors.
I have change value "minConfidence" and it's working.How to recognize right value?

I feel those two are the same thing - in one case minConfidence is too low and in another its too high
And there is no way to know what the perfect value is - if there was, it wouldn't be configurable, but set automatically.

Issue-4: Getting below error while executing ".withFaceDescriptors()" method for specific image.
it's seems that is just configuration fine-tuning, but I am unable to identify the configuration for it.

Error: Input to reshape is a tensor with 1048576 values, but the requested shape has 786432

No runtime error should be due to fine-tuning, this is a real issue

Please upload the image in question and I'll take a look

Environment
"@vladmandic/face-api": "^1.0.2"
"face-api.js": "^0.22.2"

Do you have both original face-api.js and @vladmandic/face-api loaded in your project
or just installed and then switching from one to another for testing?

They cannot be loaded at the same time as they include different versions of tfjs, so that creates a major conflict
Side-by-side install is ok, they just cannot be loaded at the same time

[mManishTrivedi]

Thanks @vladmandic for your response.

PNG related issues (ISSUE-1) and limit breaching issue (ISSUE-4) resolved while using canvas module. I am still testing with multiface detection issue.
One thing, I have noticed when I was using canvas module there is no RAM consumption here. Earlier, I was using your demo code with 2400+ images then it took 4-6 GB RAM and don't release it.

For canvas package, I applied monkey-patch before load model.

const CanvasAPI = require('canvas')
const { Canvas, Image, ImageData } = CanvasAPI;

//Apply monekyPatch
faceapi.env.monkeyPatch({Canvas, Image, ImageData});

//load image
const img = CanvasAPI.loadImage(inputFile);
// detect faces
const result = await faceapi.detectAllFaces(load, new faceapi.SsdMobilenetv1Options({ minConfidence, maxResults }));

Unfortunately, no-descriptor/"no-face" identify issue is still facing. Please find below two of testing images which are recognize as a no-face.

Last point, Why I am getting UnhandledPromiseRejectionWarning from face-API pkg while using try-catch + asyn/await properly in codebase. for eg: use below code and send loadedImage anything except HTMLImageElement | HTMLVideoElement | HTMLCanvasElement | tf.Tensor3D.

// have try/catch block already
async detectSingleFace(loadedImage) {
        const result = await faceapi
            .detectSingleFace(loadedImage, new faceapi.SsdMobilenetv1Options({ minConfidence, maxResults })
            // .detectSingleFace(loadedImage)
            .withFaceLandmarks()
            // .withFaceExpressions()
            .withFaceDescriptor()
            // .withAgeAndGender();

        return result;
    }

I can manage unhandled-exception but I am in request-response cycle so here code will be become unresponsive. Request you to please suggest right approach to handle errors with this package.

EDIT: Get face descriptors for above image when I use face-api module. I have install both module (face-api and @vladmandic/face-api), but use only one at a time.

[vladmandic]

@mManishTrivedi

Regarding detecting faces

When decoding image to tensor manually and passing tensor to FaceAPI,
it will use it as-is as it assumes user already performed any pre-processing on the image
to sufficiently enhance it before detection

And with your test image, that results in no face detections as results are just too low as input when squared distorts large face too much (I cannot help that)

# this demo uses straight decodeJpeg to tensor without any pre-processing
$ node demo/node.js /tmp/test1.jpeg
2021-03-26 08:35:48 DATA:  Image: /tmp/test1.jpeg Detected faces: 0

But when input is Canvas, FaceAPI will perform some pre-processing to input before converting it to tensor
(primarily padding to maintain correct aspect ratio of the input image when resizing it to square input as required by model)

And with your test image, that is sufficient to enhance image to the point where detection works
(note the score is still low - around 18%, but it does work!)

# this demo uses canvas module to load image and pass it to face-api which then pre-processes input
$ node demo/node-canvas.js /tmp/test1.jpeg
2021-03-26 08:36:16 DATA:  Image: /tmp/test1.jpeg Detected faces: 1
2021-03-26 08:36:16 DATA:  Detection confidence: 18% Gender: 75% male Age: 42.9 Expression: 36% neutral Box: 846,883,921,538

Regarding error handling

If I pass incorrect input to FaceAPI, it throws an error with a message:

Error: toNetInput - expected media to be of type HTMLImageElement | HTMLVideoElement | HTMLCanvasElement | tf.Tensor3D, or to be an element id

However, catching errors can seem tricky as FaceAPI performs internal class creations asynchronously and try/catch only works on code that is internally fully synchronous - that is the limit of try/catch blocks in JavaScript

So for example, this does not work as you'd expect - it still throws unhandledRejection:

  try {
    const res = await faceapi.detectAllFaces(null, optionsSSDMobileNet);
    console.log(res);
  } catch {
    console.error('error happened in try/catch of async call');
  }

but this does work as expected and does not throw an unhandledRejection as promise rejection is done correctly:

  faceapi.detectAllFaces(null, optionsSSDMobileNet)
    .then((res) => console.log(res))
    .catch(() => console.error('error happened in promise'));

or if you really want to stay with async/await, you can enable global error handler as this also works:

  process.on('unhandledRejection', () => console.error('error happened in unhandled rejection'));
  const res = await faceapi.detectAllFaces(null, optionsSSDMobileNet);
  console.log(res);

If I was starting to write FaceAPI from scratch, I'd avoid async handling inside classes
But it is what it is and you have two valid options for error handling

Hope this helps, let me know if there are any further questions.

[vladmandic]

@mManishTrivedi any updates regarding this issue?

[mManishTrivedi]

Due to Holi festival I was bit busy. But Thanks @vladmandic. I appreciate your efforts and input which provide me right direction.

I will work on your suggested solution and update you with in 1-2 days. In-meantime, you can close this ticket. Thanks Again! :)

Happy Holi

[vladmandic]

Enjoy!

[mManishTrivedi]

@vladmandic : Sorry, I am going to reopen this ticket because I found crazy thing.

As I reported some images are identified "no face/ no descriptors", because they are tilt by 90° (either left or right). When I rotate manually by image editor and upload them to my demo server then face is detected successfully.

I hope you got my points. :) So any idea about it.

One more thing, how to use below code with ".withFaceLandmarks() .withFaceDescriptor()" methods. (I have tried with various combination but failed.. :( )

faceapi.detectAllFaces(null, optionsSSDMobileNet)
    .then((res) => console.log(res))
    .catch(() => console.error('error happened in promise'));

Thanks for your all help. :)

[vladmandic]

some images are identified "no face/ no descriptors", because they are tilt by 90°

That's how models were trained. In reality, majority of face detection models are trained on mostly upright faces
I've run some tests and results are pretty consistent until 45% degrees in all cases and up to 60% in majority of cases
However, detection with face angles above 60 degrees are unreliable - and that is not likely to improve without complete retraining - and that is far outside of the scope for this library

how to use below code with ".withFaceLandmarks() .withFaceDescriptor()" methods
.catch(() => console.error('error happened in promise'));

I've spent significant time today trying to fix this code, but it's a mess as all different ".with" are asynchronously chained, so there is no easy way to propagate error hierarchically. fixing this is beyod the scope of maintaining faceapi library, it would be a rewrite - and i've done all that in my own library, so i'm not going to re-do it in a older ported library.

best solution is to validate input before passing it to faceapi so to make sure there are no errors before faceapi is even triggered.
but other than that, you can still utilize global exception handler process.on('unhandledRejection',...

on a completely separate note, check out my newer library https://github.com/vladmandic/human - and if there are any issues, i can address them there as its 100% my code instead of maintaining somebody else's code.