-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Excessive CPU and memory use + thread panics #23
Comments
Interesting. So, ddh shouldn't be following symlinks at all. That error comes from this bit of code
Where directory entries are sorted as files and dirs prior to hashing. The symlink term is used to map the rust file api. I can definitely make a patch to better handle that error, but would you be willing to test some changes on a branch? I'd like to dig into what the root cause is if possible. |
The |
Panics are not happening on the |
Ah, I was wrong: there was a panic, but I needed to wait longer:
|
I was able to reproduce the error locally and made a new commit on |
@pkolaczk any update? |
I was busy, I'll try to look at this on weekend. |
I've merged the |
Any update @pkolaczk? |
Ping @pkolaczk |
Closing. I believe the issue is resolved and the reporter has not followed up. |
Tried this on my home directory with ~1.5 million files.
There seems to be no way to turn off symlink following, so I guess it might have escaped from my home dir and tried to scan the whole filesystem. But anyways, the CPU and memory usage was extremely high. System time usage is quite unusal assuming this was I/O bound workload:
ddh
consumed 18 GBs, while other programs likefdupes, jdupes, rmfind, fclones need about 150-500 MB.
The text was updated successfully, but these errors were encountered: