Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scanning a 2TB of photos hangs after a day or two on Mac OS #1287

Open
jasontitus opened this issue May 22, 2024 · 2 comments
Open

Scanning a 2TB of photos hangs after a day or two on Mac OS #1287

jasontitus opened this issue May 22, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@jasontitus
Copy link
Sponsor

Bug Description
Scanning for near duplicates on a large volume (1.2M photos - ~3TB) of old photos seems to work fine for a day or so but that hangs at 65% of the way after hashing ~800k photos. Attaching STDERR which showed panics and such earlier in the process but nothing bad recently.

Steps to reproduce:
Scan large volumes of old photos, some of which are likely corrupt.

Terminal output (optional):

Processing took 805.932ms
Processing took 784.06525ms
Processing took 919.753417ms
Processing took 249.359686958s
Processing took 423.541633291s
Processing took 413.8683325s
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
thread '<unnamed>' panicked at czkawka_core/src/common.rs:271:50:
processing successful: Error { code: -2 }
Processing took 318.612774542s
Processing took 339.120791791s
Processing took 226.41121325s
Processing took 270.202117292s
Processing took 267.398843s
Processing took 295.143028708s
Processing took 319.169452333s



<details>
<summary>Debug log</summary>

# UNCOMMENT DETAILS AND PUT LOGS HERE

</details>

cargo run --release --bin czkawka_gui --features "heif,libraw"

warning: struct Hamming is never constructed
--> czkawka_core/src/similar_videos.rs:60:8
|
60 | struct Hamming;
| ^^^^^^^
|
= note: #[warn(dead_code)] on by default

warning: czkawka_core (lib) generated 1 warning
Compiling czkawka_gui v7.0.0 (/Users/jasontitus/experiments/czkawka/czkawka_gui)
Finished release profile [optimized] target(s) in 6.28s
Running target/release/czkawka_gui
16:06:36.895 [INFO] czkawka_core::common: App version: 7.0.0, release mode, rust 1.78.0 (2024-04-29), os Mac OS 14.4.1 [aarch64 64-bit], 20 cpu/threads
16:06:37.283 [INFO] czkawka_gui: Set thread number to 20

System

  • Czkawka/Krokiet version: czkawka_core::common: App version: 7.0.0, release mode, rust 1.78.0 (2024-04-29), os Mac OS 14.4.1
  • OS version: Mac OS 14.4.1
  • Installation method: build from pull a few days ago
@jasontitus jasontitus added the bug Something isn't working label May 22, 2024
@jasontitus
Copy link
Sponsor Author

Image 5-22-24 at 10 15 AM

@jasontitus
Copy link
Sponsor Author

With full backtraces -
thread '' panicked at czkawka_core/src/common.rs:568:51:
called Result::unwrap() on an Err value: SystemTimeError(40.598ms)
stack backtrace:
0: 0x102a4c9a4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he735dbc19bf16515
1: 0x102a6f3f4 - core::fmt::write::hef5cddb56c038e23
2: 0x102a384c8 - std::io::Write::write_fmt::h0dc631791748dd86
3: 0x102a4c7fc - std::sys_common::backtrace::print::h104bc4229e84cb49
4: 0x102a60ec8 - std::panicking::default_hook::{{closure}}::he5ba2bc230f0bb9e
5: 0x102a60c04 - std::panicking::default_hook::hf55ea1ed61f9a330
6: 0x102a61264 - std::panicking::rust_panic_with_hook::h29cd1affa2bec942
7: 0x102a4ce84 - std::panicking::begin_panic_handler::{{closure}}::h20e35afb75c4d3ea
8: 0x102a4cbc4 - std::sys_common::backtrace::__rust_end_short_backtrace::h3caf01002b4ebcc2
9: 0x102a60fac - _rust_begin_unwind
10: 0x102abcfe0 - core::panicking::panic_fmt::hd47b186e02bbaa3c
11: 0x102abd354 - core::result::unwrap_failed::hf18ffba820174349
12: 0x1024c6444 - std::sys_common::backtrace::__rust_begin_short_backtrace::h1a3f77febd6da073
13: 0x1025664b4 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hf7f08a4403702c18
14: 0x102a4708c - std::sys::pal::unix::thread::Thread::new::thread_start::he47ee5f2ef45b057
15: 0x18cccaf94 - __pthread_joiner_wake

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jasontitus