make it faster

[stevekm]

using regular du -shc * on a home dir with 17957 files (87GB) takes 0.8s but duh takes 20s

figure out how to make it faster

[stevekm]

refactor to use filepath.WalkDir (dca9bf4) only reduced execution time by about 3s on aforementioned home dir

need to do profiling and more investigation

[stevekm]

I think this must have something to do with the way that Golang does filesystem walking because in the profile, pretty much all the execution time is taken up by "syscall" entries, even after using filepath.WalkDir which supposedly reduces syscalls

[stevekm]

I think we don't need the nested DirSize call, we can probably make it faster like this;

https://go.dev/play/p/rReTZJ7o_-f

also more investigation shows that the act of retrieving file sizes in itself increases execution time by ~4x on MacBook Pro NVMe SSD so maybe even more on HDD

also consider parallel file size stat calls using something like os.ReadDir which does not recurse

[stevekm]

The new refactored SubDirSizes method in the latest master has brought execution time for duh much closer to du. In latest test, a home dir of ~150GB (849621 files) took du -shc ~ 32.765s, while duh took 46.614s

[stevekm]

I tested the latest release 1.2.1 on an instance of WSL2 on Windows 10 and it was still 1/10th the speed of du -sh so I am gonna try to look into this more