Getting Error: List index out of rage. While using detect_video(). Please help

[Abid-S]

I am using py-feat 0.5.0 pypi version and using detect_video function to extract features from video.
The program started successfully and in the middle of execution, its failing with the below error.

IndexError Traceback (most recent call last)
Input In [44], in <cell line: 7>()
11 print(f"Processing: {video}")
13 # This is the line that does detection!
---> 14 fex = detector.detect_video(video)
16 fex.to_csv(out_name, index=False)

File ~.conda\envs\LIE\lib\site-packages\feat\detector.py:802, in Detector.detect_video(self, video_path, skip_frames, output_size, batch_size, num_workers, pin_memory, **detector_kwargs)
800 frames = list(batch_data["Frame"].numpy())
801 landmarks = _inverse_landmark_transform(landmarks, batch_data)
--> 802 output = self._create_fex(
803 faces, landmarks, poses, aus, emotions, batch_data["FileName"], frames
804 )
805 batch_output.append(output)
807 batch_output = pd.concat(batch_output)

File ~.conda\envs\LIE\lib\site-packages\feat\detector.py:872, in Detector._create_fex(self, faces, landmarks, poses, aus, emotions, file_names, frame_counter)
856 for j, face_in_frame in enumerate(frame):
857 facebox_df = pd.DataFrame(
858 [
859 [
(...)
868 index=[j],
869 )
871 facepose_df = pd.DataFrame(
--> 872 [poses[i][j].flatten(order="F")],
873 columns=self.info["facepose_model_columns"],
874 index=[j],
875 )
877 landmarks_df = pd.DataFrame(
878 [landmarks[i][j].flatten(order="F")],
879 columns=self.info["face_landmark_columns"],
880 index=[j],
881 )
883 aus_df = pd.DataFrame(
884 aus[i][j, :].reshape(1, len(self["au_presence_columns"])),
885 columns=self.info["au_presence_columns"],
886 index=[j],
887 )

IndexError: list index out of range

[Ben-FCC]

I'm also faced with the same problem!

[ljchang]

Hi @Abid-S and @Ben-FCC, so sorry to hear about this. Any chance either of you could share one of your video files that is producing this error so that we can look into how to fix it? Feel free to email a link if you don't want to publicly share it on github to luke.j.chang@dartmouth.edu.

Hopefully, we can resolve this quickly.

[Abid-S]

Hi @ljchang, attached and sent a video to the above email. Thank you for your help.

[ljchang]

Apologies for the delay.

This issue appears to arise when there is a discrepancy between faces detected using the face detector and pose detector when detecting image frames in a video. In the video @Abid-S sent me, there are frames in which all of the detectors find 2 faces, except for the pose detector, which only finds 1.

@ejolly and @TiankangXie, there appear to be two different fixes for this on the detect_image method that were not included in the detect_video. Neither of them seems to be a particularly great fix. I wonder if we can come up with a more robust solution.

1. In detect_image there is a threshold keyword, which is passed into the face detector, but not the pose detector. This seems like a bad idea in general as it will potentially conflict with passing a threshold using the **face_model_kwargs. I also think the default of 0.5 is entirely too low and honestly, this alone might be the main culprit for this bug.
2. In detect_image there is a kludgy _check_detections() method which will try to skip detections by default when this happens, which is not present in the detect_video method. All it seems to do is to not register any detections and pass an error message when this happens, but doesn't actually fix the problem.

This is actually a pretty strange issue, which we do not have any type of test for yet. There are a number of potential solutions that warrant discussion before we can resolve this bug.

1. We can return all detections, but fill in nans for the detector that has a discrepancy. This seems like it is happening in the pose detection more often than any other detector. We would need to figure out which of the poses that are detected match the faces found in the other detectors. This would be fairly straightforward with img2pose as we can just find overlap in the bounding boxes and set nans for non-matched detections.
2. We can pass the same threshold value (if we decide to keep this keyword) into the pose detection as well as the face. And obviously do this for both image and video detection. I doubt this will end up being a reliable fix as they might have different thresholds.
3. We could try re-running pose with different parameters if it detects a different number of faces. or even try a different pose algorithm altogether, though this would dramatically slow things down and I never like when software makes choices without the user knowing (e.g., fmriprep).
4. We could just pick the min of the faces or pose bounding box detection and use those for all other detections (e.g., landmark, au, and emotion).

Curious to hear everyone's opinions on this before we spend too much time working on this problem. I will continue to play with the threshold part in the meantime.

[ljchang]

This definitely looks like it is related to threshold=0.5 keyword, as the problematic frame I'm working on is finding a not real face, but is still returning it because the threshold is too low.

ipdb> frame
[[344.8896484375, 155.04977416992188, 499.3318684895833, 372.75858306884766, 0.9949491], [543.8634440104166, 377.5543899536133, 606.249755859375, 438.67803955078125, 0.5618871]]

[ljchang]

ok, just wanted to share an update. The issue for both @Abid-S and @Ben-FCC 's videos is very clearly caused by the default face detection threshold of 0.5. I arbitrarily moved it to 0.9. Honestly, I think we ought to entirely remove this from the package, but will leave it until I hear some discussion from @ejolly and @TiankangXie.

I have started a PR #157 addressing these issues.