Frames input folder files empty

[2909ft]

Hello! :)
I am having issues with labelling the behaviours. Whenever I try to do it, it seems like the GUI does not load the extracted frames correctly, leaving me with a blank space where the video frames should be presented. I have tried extracting the frames again, but it tells me the frames have already been extracted, despite the fact that the folders inside input folder are empty. Any ideas on how to solve this? I am using the no DLC version, and I obtained the csvs from colab.

Thank you!
Filipa

[inoejj]

How did you extract your frames? Please follow the steps below and see if it works.

1. Please make sure that the project_folder/videos/ folder contains videos.

2. Click on File --> Load project --> Load Project .ini and load the project_config.ini.

3. Under "Extract further frames into project folder", click Extract frames.

If you do not have any videos in the project_folder/videos folder, you can import the videos in the second tab of "Load project".

[2909ft]

Hello inoejj! Thanks for the reply!

Yes, I followed exactly those steps when I first tried to extract the frames. The videos folder has all the videos inside. In fact, I could even watch the videos when I attempted to label the frames, by clicking on the "open video" button on the gui.

Best regards,
Fillipa

[sgoldenlab]

Hi @2909ft! I can't recreate the issue at the moment but we should be able to solve it.

1. Do you have ffmpeg installed? (install instructions here: https://m.wikihow.com/Install-FFmpeg-on-Windows)

2. Delete the empty folder that should contain the frames (e.g., project_folder\frames\input\VideoName), and click on extract frames again. What is printed out in the SimBA main console window?

3. To extract frames for single videos, you can use the tools menu. You can try this, and if this works we do not have an ffmpeg issue. Click on Tools--> Extract frames--> Extract frames. Select the path to your video and click on extract all frames. Does this work?

Thanks!

[2909ft]

Ah yes, that might be the issue! I switched computers yesterday and forgot to install ffmpeg. That should fix the issue, I will try it tomorrow.

[sgoldenlab]

@2909ft Are you able to resolve the issue?

[2909ft]

Yes I was, thank you!
However, now I have another issue: whenever I try to train the model, I get the following error message "Error: the dataframe does not contain any target annotations. Please check the csv files in the project_folder/csv/target_inserted folder". There are in fact no csv files inside that folder. Do I have to label all the frames for the generation of the csv files to work? I labelled around 200.

[sronilsson]

Hi @2909ft - you do not have to label all of the frames. I highly recommend it, but it will depend on your specific case. If you do not look at all of the frames using the behavior labelling tool, there may be some frames that contain the behavior of interest, that will automatically labelled as not containing the behavior of interest, and this will disrupt the classifier.

When you finished labelling the 200 frames, there should be a new csv generated in the target inserted folder. This is generated when you click on the "Save and quit" button at the bottom of the label frames interface to save the data: https://github.com/sgoldenlab/simba/blob/master/docs/labelling_aggression_tutorial.md. Make sure you click this button.

Also, can you see your .csv files in the festures_extracted folder?

[2909ft]

Thank you for the reply! My features_extracted folder is empty indeed, but I am unsure why since I clicked on "extract features". I also do not have a "save and quit" button, I have a "generate/save csv", but I guess that doesn't make much of a difference.

[sronilsson]

You're correct - "Generate/save csv" is what the button should read,

An empty feature_extracted folder suggests an earlier step didn't go as planned.

In your csv folder, which subfolders do contain csv files - can you see your csv files in your "input" subfolder and your two different outlier correction subfolders?

[2909ft]

Only the input subfolder contains csv files, both of the outlier correction subfolders are empty.

[sronilsson]

That would be the issue - did you do the outlier correction step? This is required for the rest of the steps to work: https://github.com/sgoldenlab/simba/blob/master/docs/tutorial.md#step-4-outlier-correction

If you did, and it didn't work as planned, what was printed out in the main SimBA terminal window when you clicked on 'Correct outlier'?

[2909ft]

I had only selected movement criteria for correction, not location, I thought it would work with just one of them. I have now selected for both, and there are new csvs on the subfolders, apologies for the misconception :)
However, the extraction still does not output anything. On the terminal it is simply displayed "Pose-estimation body part setting for feature extraction: 8" and nothing else.

[sronilsson]

Thanks!

1. To confirm, when you created the project, did you set the 'Animal Settings" to "1 animal, 8bp" like this?

2. In the main terminal window (the terminal window you used to start SimBA and were you wrote "python Simba.py") do you see any messages being printed after clicking on "Extract Features"?

[2909ft]

Thank you!
Yes, I set the animal settings to 1 animal 8 points.

Yes, there's the following error message. However, the file exists in that directory (as you can see from the screenshot).

[sronilsson]

Thanks - the non-16 body part feature extraction scripts contained typos - I'm sorry about that. To get passed this error:

i) In he features_scripts folder in Simba and, open "extract_features_8bp.py". On line 18, you should see: vidInfPath = os.path.join(vidInfPath, 'project_folder', 'logs')

Change this line to: vidInfPath = os.path.join(vidInfPath,'logs'), i.e, remove 'project_folder' from this line.

Let me know how it goes!

[2909ft]

I have now deleted the 'project_folder' from the script, and now I get a different error message ahaha "Error: make sure all the videos that are going to be analyzed are represented in the project_folder/logs/video_info.csv file", but they are (see screenshot)
On the python terminal it says "TypeError: cannot convert the series to <class 'float'>.
I have noticed that the column corresponding to the distance is blank, but the pixels/mm was calculated correctly. Could this be the issue?

[sronilsson]

Hi @2909ft - this may be another bug in SimBA, in handling .avi's. To check if this is the case there are two options. I can't see the entire column A in your screenshot of the video_config, but if you expand it, do you see the entire filename including the file endings?

If you see the file endings, can you remove the ".avi" from each row and try again?

Alternatively, could you try and replace the .avi's with mp4 versions of the videos and try to extract features again? There is a tool in SimBA in the tools menu to convert video file format.

Thanks!

[2909ft]

Hey!

I do not see the avi ending at the end, it is simply the name of the file. I have tried with the mp4 format and it did not work either

[sronilsson]

Thanks for testing this out - the "TypeError: cannot convert the series to <class 'float'> error comes from the code finding multiple rows in the video_config file, when it should only find one row. The code looks at each of your file names in your outlier_corrected_movevement_location folder, and finds the matching row in the video_config file, using the Video column to get a match. Things could go wrong if the filenames aren't matching. To check whats happening, could you open "extract_features_8bp.py", and after this line (line 55):

currVideoSettings = vidinfDf.loc[vidinfDf['Video'] == currVidName]

insert a new line:

print(currVideoSettings)

try again, and then tell me whats printed out in the main SimBA console.

[2909ft]

Hey! Thanks for all the help! Once I did that, simba console output this. I believe the file names are not matching (as you can see from the screenshot) could that be the issue?

[sronilsson]

That's it! SimBA should fix the DLC csv filenames (i.e. remove the long part of the filename that specify the model and the iterations the file was generated by) when imported through the GUI. Try and rename the filenames to match the video_info file and re-run feature extraction and let me know if it works

[sgoldenlab]

Following up, has this resolved the issue?

[2909ft]

Yes it did, thank you! I am now trying to get my head around the hyperparameters. Thanks for all the help!

[sronilsson]

@2909ft - a heads up, the code isn't optimized for single animals and 8 body-parts yet, it's something we're working on - to have SimBA accept any body part configuration. So if you happen to bump into any issues let us know and we'll work though them.

[2909ft]

Awesome thank you!

[Qasimster]

Whenever i extract frame rates from a video dataset folder to a train_1 folder. After extracting frame rates the folder is empty where as pycharm shows it has been done successfully. Any one aware of this issue? My code
for i in tqdm(range(train.shape[0])):
count = 0
videoFile = train['video_name'][i]
cap = cv2.VideoCapture('UCF/'+videoFile.split(' ')[0].split('/')[1]) # capturing the video from the given path
frameRate = cap.get(5) #frame rate
x=1
while(cap.isOpened()):
frameId = cap.get(1) #current frame number
ret, frame = cap.read()
if (ret != True):
break
if (frameId % math.floor(frameRate) == 0):
# storing the frames in a new folder named train_1
filename =r"D:\New Projects Pycharm\Videoclassi\tttt/" + videoFile.split('/')[1].split(' ')[0] +"_frame%d.jpg" % count;count+=1
cv2.imwrite(filename, frame)
cap.release()