random crash

[mortee]

Describe the bug
Tried to run the program on my image folder, but at a certain point, it consistently crashes, without any usable error log.

To Reproduce
Steps to reproduce the behavior:

1. Set up elastic, py-image-dedup using docker-compose
2. run on image folder
3. see it analyze files, and at a certain percentage, always crash

Expected behavior
The program should finish analyzing my files, and report back the duplicates.

Screenshots

INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 63% (2037/3233)
INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 64% (2070/3233)
INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 65% (2102/3233)
INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 66% (2134/3233)
INFO:py_image_dedup.library.progress_manager:Analyzing files in '/mnt/source': 67% (2167/3233)
Killed
<docker automatically restarts>
DEBUG:py_image_dedup.util:Starting daemon...
Starting daemon...
DEBUG:py_image_dedup.util:Starting prometheus reporter...
Starting prometheus reporter...
DEBUG:py_image_dedup.util:==> DRY RUN! No files or folders will actually be deleted! <==
DEBUG:py_image_dedup.util:Phase 1/6: Cleaning up database ...
==> DRY RUN! No files or folders will actually be deleted! <==
Phase 1/6: Cleaning up database ...

Desktop (please complete the following information):

• OS: Ubuntu 20.04, docker

[markusressel]

Does Killed mean the container was killed by docker or the host system?
Could this be related to running low on system memory at a certain point?

Otherwise it might be caused by a specific image file in your library. To further pinpoint the issue we would have to isolate this file and do further testing on it.

[mortee]

Does Killed mean the container was killed by docker or the host system?
Could this be related to running low on system memory at a certain point?

Honestly, I have no idea. Though the fact that it always crashes at the exact same percentage suggests that it's not a resource issue, but some unhandled exception (for which a stack trace is suppressed for some reason?)

Anyhow, how can I make it confess about which exact file it's processing while this happens?

[mortee]

Some suggestions here?

[markusressel]

Honestly, I have no idea. Though the fact that it always crashes at the exact same percentage suggests that it's not a resource issue, but some unhandled exception (for which a stack trace is suppressed for some reason?)

Well... the fact that docker just prints "Killed" suggests the opposite 😄
Not sure why a suppressed exception "for some reason" should be more likely.

Google: ubuntu Docker "Killed" message
https://stackoverflow.com/questions/44417159/docker-process-killed-with-cryptic-killed-message

No idea what ubuntu does compared to Mac OS though.

Anyhow, how can I make it confess about which exact file it's processing while this happens?

It should both log as well as print to the console and handle it gracefully:

py-image-dedup/py_image_dedup/library/deduplicator.py

Line 287 in 29c7b7f

logging.exception(e)

Stack Overflow

Docker process killed with cryptic `Killed` message

Running a python script in a docker container and everything seems to be running smoothly, seeing some STDOUT messages, about 5 minutes in I get a Killed message with no further explanation and the

[markusressel]

Closing this because of inactivity. Feel free to reopen if necessary.