Batch processing not functioning properly

[dgbarson]

Hello!

Thanks for all your work on this! It's an amazing tool.

I have been having trouble getting batch processing to work. I can save multiple sets of ROIs (which saves multiple _proc.npy and _proc.mat files in my data directory), but they do not appear in a list under the process batch button. When I click the process batch button, Facemap runs only on the most recent movie and set of ROIs. However, it will do so multiple times (the same number of times as there are sets of saved ROIs - i.e. if I create ROIs for file1.avi, file2.avi, and file3.avi, it will process the ROIs for file3.avi three times).

I am using the most recent version of the Facemap using the conda environment on Windows 7 and Windows 10.

Thanks for your help with this!

Best,
Dan

[Michaelhess17]

Hi there!
I'm having an issue with Batch Processing, as well. I have the most up-to-date version of conda, and just tried installing FaceMap both through creating a new conda environment as described in the README and through pip install/pip --upgrade in another new environment, and neither way is working. I should mention that I ran into a Pip error during the conda installation (and subsequently had errors that said WARNING: Ignoring invalid distribution ~yqt5-sip (though pyqt5 did get installed). After these errors occurred I went through the environment.yml file and installed all the packages either through pip or conda as described in the .yml file.

There are separate problems I'm experiencing depending on the installation method, so I'll describe them both.

When installing with pip, it seems that it is not saving the selected ROIs in the output folder when that button is pressed, and then when loading multiple videos the Batch Process button remains grey.
When using the save ROIs button, here is the error shown in the command window:

Error reading frame
Traceback (most recent call last):
File "C:\Users\Micha\Anaconda3\envs\Lu Lab\lib\site-packages\facemap\gui.py", line 501, in go_to_frame
self.jump_to_frame()
File "C:\Users\Micha\Anaconda3\envs\Lu Lab\lib\site-packages\facemap\gui.py", line 533, in jump_to_frame
self.next_frame()
File "C:\Users\Micha\Anaconda3\envs\Lu Lab\lib\site-packages\facemap\gui.py", line 565, in next_frame
self.imgs[i][:,:,:,2] = im[i]
IndexError: list index out of range
Y:/Jui-Yen Huang/Derived Files/Data/In vivo Calcium Imaging/20210516_0528_jRGECO1a_FGFR-tri-Ncre64-2/20210516_jRGECO1a_FGFR-tri-Ncre64-2_P11/Behavior
Traceback (most recent call last):
File "C:\Users\Micha\Anaconda3\envs\Lu Lab\lib\site-packages\facemap\gui.py", line 637, in save_ROIs
'filenames': self.filenames, 'iframes': self.iframes}
AttributeError: 'MainW' object has no attribute 'iframes'
Y:/Jui-Yen Huang/Derived Files/Data/In vivo Calcium Imaging/20210516_0528_jRGECO1a_FGFR-tri-Ncre64-2/20210516_jRGECO1a_FGFR-tri-Ncre64-2_P11/Behavior
Traceback (most recent call last):
File "C:\Users\Micha\Anaconda3\envs\Lu Lab\lib\site-packages\facemap\gui.py", line 637, in save_ROIs
'filenames': self.filenames, 'iframes': self.iframes}
AttributeError: 'MainW' object has no attribute 'iframes'
Y:/Jui-Yen Huang/Derived Files/Data/In vivo Calcium Imaging/20210516_0528_jRGECO1a_FGFR-tri-Ncre64-2/20210516_jRGECO1a_FGFR-tri-Ncre64-2_P11/Behavior
Traceback (most recent call last):
File "C:\Users\Micha\Anaconda3\envs\Lu Lab\lib\site-packages\facemap\gui.py", line 637, in save_ROIs
'filenames': self.filenames, 'iframes': self.iframes}
AttributeError: 'MainW' object has no attribute 'iframes'

When installing with conda, the Batch Process button is available, and I am able to save the ROIs, but then pressing the Batch Process button first causes an OpenCV error and then either crashes the application or makes following attempts not work either. Here is the OpenCV error I am getting:

[ERROR:0] global C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-obtuy4a_\opencv\modules\videoio\src\cap.cpp (162) cv::VideoCapture::open VIDEOIO(CV_IMAGES): raised OpenCV exception:
OpenCV(4.5.2) C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-obtuy4a_\opencv\modules\videoio\src\cap_images.cpp:267: error: (-215:Assertion failed) number < max_number in function 'cv::icvExtractPattern'
QGraphicsScene::removeItem: item 0x1b7c074b0b0's scene (0x0) is different from this scene (0x1b7c0e3b1b0)
Traceback (most recent call last):
File "C:\Users\Micha\FaceMap\facemap\menus.py", line 17, in
openFolder.triggered.connect(lambda: io.open_folder(parent))
File "C:\Users\Micha\FaceMap\facemap\io.py", line 37, in open_folder
load_movies(parent)
File "C:\Users\Micha\FaceMap\facemap\io.py", line 251, in load_movies
parent.jump_to_frame()
File "C:\Users\Micha\FaceMap\facemap\gui.py", line 747, in jump_to_frame
self.imgs[i][:,:,:,1] = self.img[i].copy()
ValueError: could not broadcast input array from shape (1080,1920,3) into shape (0,0,3)

Do you know what I can do to get the Batch Processing working?

Thanks for the help and for making all these awesome packages!!
-- Michael

[Atika-Syeda]

Hello @Michaelhess17,

Please pull recent changes from the main branch (using the instructions below) to avoid "AttributeError: 'MainW' object has no attribute 'iframes'"

git clone https://github.com/MouseLand/facemap.git
Remove old conda environment for facemap and follow steps 1-5 for using the updated environment.yml file.

For batch processing, please follow the instructions after cloning the updated version. The opencv error seems to indicate some issue with loading video so make sure you are able to load the video and 'process ROIs' before proceeding with 'process batch'.

[Atika-Syeda]

Hello @dgbarson,

Are you trying batch processing of multiple ROIs from multiple videos taken simultaneously?
If your files are overwritten, I'd recommend storing file1.avi, file2.avi.. in separate folders and then use batch process for the different sets of ROIs independently.

[Michaelhess17]

Hey again!
I tried installing Facemap by creating a Conda environment with the GitHub environment.yml on a couple of machines (one with an outdated version of Conda, and one that's up-to-date) and I am still receiving the same error ("AttributeError: 'MainW' object has no attribute 'iframes'"), unfortunately!

Do you know anything else we might try?

Thanks again!

[Atika-Syeda]

Hi @Michaelhess17, the error tells me that your version of the code is different from the main branch. You can confirm that by searching for 'iframes' variable in your code and check against the relevant file and line number on the main branch. Hence, you will have to make sure your code is the same as main branch to avoid the error.

[Atika-Syeda]

Hello Michael, You don't want to install the pip version, but simply clone the main repo. Before cloning the main repo you'll have to uninstall and remove previous versions of facemap's conda env. After cloning the repo you don't need to run setup.py but just create the conda environment using the updated environment.yml file. Once you have activated the conda environment but still run into some package errors then you can run 'pip list' command to check whether the packages exist in your environment. Hope this helps.

…

________________________________ From: Michael Hess ***@***.***> Sent: Monday, August 23, 2021 10:53 PM To: MouseLand/facemap ***@***.***> Cc: Atika Syeda ***@***.***>; Mention ***@***.***> Subject: Re: [MouseLand/facemap] Batch processing not functioning properly (#30) [△EXTERNAL] Hi again @Atika-Syeda<https://github.com/Atika-Syeda>! I see the issue you are describing, as the code in my facemap package in my site-pacakges folder does contain this line, so it must be outdated relative to the code on github. What's puzzling though is that I have followed the instructions to do this install. When I do the install however, and try to complete step 5, I get an error that facemap is not installed, so I runpip install --upgrade facemap line to get that installed. However, the version of facemap that I get through this channel must be where the error is coming from. Do you know how I can install the version of the package found in the Github code here? I have also tried pip install git+https://github.com/mouseland/facemap but that seems to download the same version as just using pip install facemap, and I tried to clone the repo and run the setup.py script, but I get the following error: Could not find suitable distibution for Requirement.parse('PyQt5.sip') though I have PyQt5 installed... Any suggestions? Thanks so much for all your help!! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#30 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALJMAEA5WACIHFTXRPC3VATT6MQVXANCNFSM45DJEZNQ>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>.

[Atika-Syeda]

Also, here are some instructions for the common issues listed on facemap's readme page: If when running python -m facemap, you receive the error: No module named PyQt5.sip, then try uninstalling and reinstalling pyqt5 pip uninstall pyqt5 pyqt5-tools pip install pyqt5 pyqt5-tools pyqt5.sip

…

________________________________ From: Atika Syeda ***@***.***> Sent: Wednesday, August 25, 2021 5:58 AM To: MouseLand/facemap ***@***.***> Subject: Re: [MouseLand/facemap] Batch processing not functioning properly (#30) Hello Michael, You don't want to install the pip version, but simply clone the main repo. Before cloning the main repo you'll have to uninstall and remove previous versions of facemap's conda env. After cloning the repo you don't need to run setup.py but just create the conda environment using the updated environment.yml file. Once you have activated the conda environment but still run into some package errors then you can run 'pip list' command to check whether the packages exist in your environment. Hope this helps.

________________________________ From: Michael Hess ***@***.***> Sent: Monday, August 23, 2021 10:53 PM To: MouseLand/facemap ***@***.***> Cc: Atika Syeda ***@***.***>; Mention ***@***.***> Subject: Re: [MouseLand/facemap] Batch processing not functioning properly (#30) [△EXTERNAL] Hi again @Atika-Syeda<https://github.com/Atika-Syeda>! I see the issue you are describing, as the code in my facemap package in my site-pacakges folder does contain this line, so it must be outdated relative to the code on github. What's puzzling though is that I have followed the instructions to do this install. When I do the install however, and try to complete step 5, I get an error that facemap is not installed, so I runpip install --upgrade facemap line to get that installed. However, the version of facemap that I get through this channel must be where the error is coming from. Do you know how I can install the version of the package found in the Github code here? I have also tried pip install git+https://github.com/mouseland/facemap but that seems to download the same version as just using pip install facemap, and I tried to clone the repo and run the setup.py script, but I get the following error: Could not find suitable distibution for Requirement.parse('PyQt5.sip') though I have PyQt5 installed... Any suggestions? Thanks so much for all your help!! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#30 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALJMAEA5WACIHFTXRPC3VATT6MQVXANCNFSM45DJEZNQ>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>.

[Michaelhess17]

Thank you so much! As far as I can tell, this is working now!