Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash when batch importing files #49

Closed
aseques opened this issue Aug 27, 2020 · 10 comments
Closed

Crash when batch importing files #49

aseques opened this issue Aug 27, 2020 · 10 comments
Assignees
@jim-easterbrook
Labels
bug

Comments

@aseques
Copy link
Contributor

aseques commented Aug 27, 2020

I am importing a directory with a couple hundred of photos, photini works for a time (I set it to move the photos), but at the end it stops without any message.
All the photos of the directory are moved apparently but I can't be sure.
Photini version is current HEAD: c403b5f
Kubuntu 20.04 with neon packages
Python 3.8.2

Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized

Current thread 0x00007fd664b4f740 (most recent call first):
File "/usr/lib/python3/dist-packages/gi/overrides/GExiv2.py", line 44 in open_path
File "/usr/local/lib/python3.8/dist-packages/photini/exiv2.py", line 75 in init
File "/usr/local/lib/python3.8/dist-packages/photini/exiv2.py", line 566 in init
File "/usr/local/lib/python3.8/dist-packages/photini/exiv2.py", line 628 in open_old
File "/usr/local/lib/python3.8/dist-packages/photini/metadata.py", line 1095 in init
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 63 in init
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 730 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail
File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file
File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied
...
Aborted
@jim-easterbrook
Copy link
Owner

It appears you're running out of memory. Each photo is opened in Photini as you import / move it, and it's generally a good idea to limit the number of photos you open at the same time. Try moving them in batches of 100 or fewer, closing the opened images after each batch.

@jim-easterbrook
Copy link
Owner

Looking at this again it might possibly be a recursion problem, if copying files is quicker than Qt's thumbnail generation and display.

@jim-easterbrook
Copy link
Owner

jim-easterbrook commented Aug 28, 2020 •
edited

I think I've found the problem, but no solution yet.
The importer runs in its own thread for maximum speed. After each image has been copied the thread sends a signal to the main thread to update the image area thumbnail display. During this process Qt's processEvents is called to keep the display updated, but that can also trigger processing of another signal from the importer thread, even though the previous one hasn't been fully processed yet.

@jim-easterbrook jim-easterbrook self-assigned this Aug 28, 2020
jim-easterbrook added a commit that referenced this issue Aug 28, 2020
@jim-easterbrook
Remove processEvents from show_thumbnail
0e18604 
Processing events could cause problems when importer runs at high speed
(e.g. copying files on local disc) as discussed in bug #49.
@jim-easterbrook
Copy link
Owner

I think commit 0e18604 might be all that's needed to fix this.

@aseques
Copy link
Contributor Author

aseques commented Aug 31, 2020

@jim-easterbrook do you think that there are picutures that could have disapeared instead of moved? I could recover them from backup if needed, but I can't recall the original number of photos.

@jim-easterbrook
Copy link
Owner

I don't think so, but if the Python interpreter crashes in the middle of moving a file I don't know what could happen.

@aseques
Copy link
Contributor Author

aseques commented Sep 11, 2020 •
edited

I'm closing this, there was a crash again, but I didn't have the verbose flag so I can't provide more information at this time, when running photini a second time it did't detect missing files, so I assume that it crashed after doing the work.

@aseques aseques closed this as completed Sep 11, 2020
@jim-easterbrook
Copy link
Owner

I'm reopening it again. :)

I'm not totally sure my fix is robust and I really need to find a better way.

jim-easterbrook added a commit that referenced this issue Sep 15, 2020
@jim-easterbrook
Use a QMutex to limit file copying speed...
1bcd9f7 
...if image display can't keep up. This should be a more robust solution
to bug #49.
@jim-easterbrook
Copy link
Owner

I hope commit 1bcd9f7 is a better solution.

@jim-easterbrook
Copy link
Owner

My earlier mutex solution is not working for me since I updated my OS version. (It never was a nice solution.) In commit 0ce6148 I've implemented a much simpler solution which I hope will be more reliable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jim-easterbrook jim-easterbrook

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aseques @jim-easterbrook