Switch to globwalk

[beeb]

Not sure if you'll agree to this change, but I was not able to scan my full home directory with deduplicator, probably because it was trying to keep too many open file descriptors. globwalk seems to handle this much better (the default limits the number of open descriptors to a sane value). With this change I was able to scan my full home dir.

I have not tested it extensively but it seems to work for me, even with the --types argument.

[sreedevk]

Hey @beeb ! I've been trying to resolve this issue for a bit, I'm glad that you thought of the same thing too. I was trying to build a custom multi threaded directory walker (I couldn't get jwalk to work), but didn't really work with the progress indicator. I think this is definitely a step in the right path. I will review and merge this.

[beeb]

As a next step I think it could be good to switch to the Builder and set the BASE_PATH like in the example

[sreedevk]

@beeb I agree, we can also add options in deduplicator for --max-depth & --follow-links and directly map those options to the builder.