Error in Blazeface detection with a vertical video frame (1080x1920 resolution)

[rakadambi]

I am getting an error in landmarks detection with a vertical video frame. This is the image

This is the error:

/usr/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
return f(*args, **kwds)
/home/aditya/Python_code_learning/dev/kython_env/lib/python3.7/site-packages/face_alignment/utils.py:79: RuntimeWarning: divide by zero encountered in double_scalars
t[0, 0] = resolution / h
/home/aditya/Python_code_learning/dev/kython_env/lib/python3.7/site-packages/face_alignment/utils.py:80: RuntimeWarning: divide by zero encountered in double_scalars
t[1, 1] = resolution / h
E

ERROR: test_predict_points (main.Tester)

Traceback (most recent call last):
File "facealignment_test.py", line 33, in test_predict_points
landmarks = fa.get_landmarks_from_image(image)
File "/home/aditya/Python_code_learning/dev/kython_env/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
return func(*args, **kwargs)
File "/home/aditya/Python_code_learning/dev/kython_env/lib/python3.7/site-packages/face_alignment/api.py", line 153, in get_landmarks_from_image
inp = crop(image, center, scale)
File "/home/aditya/Python_code_learning/dev/kython_env/lib/python3.7/site-packages/face_alignment/utils.py", line 128, in crop
interpolation=cv2.INTER_LINEAR)
cv2.error: OpenCV(4.4.0) /tmp/pip-build-qct9o6da/opencv-python/opencv/modules/imgproc/src/resize.cpp:3929: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

[moncio]

I have the same error, please, could please someone help us? Thanks!

[1adrianb]

It looks like the blazeface detector failed for some reasons to detect a face (note: resizing the image may "fix" this).
While this particular hard error can be easily avoided by performing a check, I haven't trained or tested the blazeface detector personally so I am unable to be of much help on this aspect. @imadtoubal do you have any suggestions on this since you helped porting this over?
For now my suggestion is to either add a fall-back to sf3d when this happens or switch to it.

[rakadambi]

This seems to happen with vertical videos, I think.

[1adrianb]

@rakadambi Sorry for delay. Does this happens in the case of blazeface only for vertical videos? Have you tested it with sf3d?

I presume this is a duplicated of #210.

[rakadambi]

I should have clarified. Yes, this works fine with SF3D. Pretty much all the images I tested work fine with SF3D. Blazeface seems to fail now and then and it seems vertical frames fail, but not all.

If you need me to test more, I am happy to do that.

[1adrianb]

The error itself should be avoided now. The blazeface behavior remains however the same. One of the likely reason why blazeface fails more often is likely due to the fact that it operates on 128x128px images. This seam to be the default settings used in mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt Increasing the resolution should likely perform better at the cost of extra computational demand.

[imadtoubal]

@1adrianb I concur. The way BlazeFace has been implemented is it crops the frame instead of resizing. I thought this would actually help preserve the aspect ratio of the face. I have seen other implementations where people used 3 square tiles (across x or y) and then do 3 separate inferences, this way, you don't lose any parts of the frame. Perhaps @jklewis99 might be able to help since he sold this issue when we used this library before. Otherwise, I will look at solving this as soon as I get the chance! :)

[1adrianb]

@imadtoubal thanks! perhaps this should be done via resizing + padding to avoid cropping ans scale distortions ?

[imadtoubal]

That's definitely one way to do it! Might perform a little worse with 16:9 and similar aspect ratios since it will require quite a bit of resizing. Almost half of the 128x128 image will be black and faces might be too low-résolution to detect.