Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

numpy throwing error in batch mode when some images do not contain a face #224

Open
markwillowtree opened this issue Apr 24, 2024 · 6 comments

Comments

@markwillowtree
Copy link

markwillowtree commented Apr 24, 2024

venv/lib/python3.12/site-packages/facenet_pytorch/models/mtcnn.py", line 444, in select_boxes
selected_boxes = np.array(selected_boxes)
^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (316,) + inhomogeneous part.

I got this error message from this code

    mtcnn = MTCNN(112, device="cuda:0")

    container = av.open(video_path)
    frames = list(container.decode(video=0))
    images = [frame.to_image() for frame in frames]

    mtcnn(images)

It appears to be similar to the one described in this resolved issue:

#206

I made the following changes to lines 444-446 in the file 'models/mtcnn.py'. These modifications ensure that when there are no faces detected in an image, the function returns None.

        if batch_mode:
            selected_boxes = np.array(selected_boxes, dtype=object)
            selected_probs = np.array(selected_probs, dtype=object)
            selected_points = np.array(selected_points, dtype=object)

However, I'm uncertain if this fix might have adverse effects elsewhere in the codebase as I'm not familiar with it.

@florianblume
Copy link

florianblume commented May 14, 2024

I ran into the same problem. My solution is to remove the boxes that are None and delete the corresponding images. I achieved this by returning the images from this function back to forward, like so:

forward function

# Detect faces
batch_boxes, batch_probs, batch_points = self.detect(img, landmarks=True)
# Select faces
if not self.keep_all:
    img, batch_boxes, batch_probs, batch_points = self.select_boxes(
        batch_boxes, batch_probs, batch_points, img, method=self.selection_method,
    )
# Extract faces
faces = self.extract(img, batch_boxes, save_path)

if return_prob:
    return faces, batch_probs
else:
    return faces

and the select_boxes function (omitted everything before if batch_mode because nothing changed there)

if batch_mode:
    sanitized_selected_boxes = []
    sanitized_selected_probs = []
    sanitized_selected_points = []
    for idx, box in enumerate(selected_boxes):
        if box is None:
            if isinstance(imgs, list):
                imgs.pop(idx)
            elif isinstance(imgs, np.ndarray):
                imgs = np.delete(imgs, idx, 0)
            elif isinstance(imgs, torch.Tensor):
                imgs = torch.cat((imgs[:idx], imgs[idx+1:]), 0)
            else:
                raise TypeError("imgs must be a list, np.ndarray, or torch.Tensor")
        else:
            sanitized_selected_boxes.append(box)
            sanitized_selected_probs.append(selected_probs[idx])
            sanitized_selected_points.append(selected_points[idx])
    selected_boxes = np.array(sanitized_selected_boxes)
    selected_probs = np.array(sanitized_selected_probs)
    selected_points = np.array(sanitized_selected_points)
else:
    selected_boxes = selected_boxes[0]
    selected_probs = selected_probs[0][0]
    selected_points = selected_points[0]

return imgs, selected_boxes, selected_probs, selected_points

@beicodewarrior
Copy link

Hi, @florianblume

Your code fixed a certain issue when some images don't contain faces. However, it threw a new error when testing one image in the inference script.

mtcnn = MTCNN(
    image_size=160, margin=0, min_face_size=20,
    thresholds=[0.6, 0.7, 0.7], factor=0.709, post_process=True,
    device=device
)

img = Image.open('1.jpg')
img_cropped = mtcnn(img)

This code throws the error like

lib\site-packages\facenet_pytorch\models\utils\detect_face.py", line 359, in extract_face
    margin * (box[2] - box[0]) / (image_size - margin),
TypeError: 'float' object is not subscriptable

Can you check it on your side?

@florianblume
Copy link

I would've expected that this doesn't affect a single image because of the if batch_mode: check in the original code. I don't need to process a single image so I'll just use the solution I posted. But I guess it should be an easy fix if you debug the code a bit.

@beicodewarrior
Copy link

The returned image value you create seems to pass the if batch_mode:
So it doesn't recognize the single image and batch group. Could you please update your code for that?

@florianblume
Copy link

Sorry, I don't have time for that, you'll have to fix it yourself (and you seem to be on the right track).

@makea2018
Copy link

Hi, @beicodewarrior ! I faced with same problem. This is how i fixed it:

mtcnn = MTCNN(
    image_size=160, margin=0, min_face_size=20,
    thresholds=[0.6, 0.7, 0.7], factor=0.709, post_process=True,
    device=device
)

img = Image.open('1.jpg')
img_cropped = mtcnn([img])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants