HDD performance is poor

[Davester47]

Performance of dust compared to single-threaded coreutils du on my HDD is about 2x worse. I was testing on my home directory, which according to dust has 139,000 files for a total of 711 gigabytes. The filesystem is ext4.

$ echo 1 | sudo tee /proc/sys/vm/drop_caches
$ time dust
...
711G ┌─┴ .
dust  1.00s user 3.06s system 4% cpu 1:26.04 total

And for du:

$ echo 1 | sudo tee /proc/sys/vm/drop_caches
$ time du -sh .
712G	.
du -sh .  0.29s user 1.54s system 4% cpu 45.589 total

I'm not sure what is causing the difference here, but I'm guessing it has to do with the directory traversal order used by the two programs. du uses depth-first search, whereas dust appears to be using breadth-first search through rayon. Perhaps depth first search is more effective on mechanical HDDs?

[bootandy]

du is a simpler tool.

It's also ancient, highly crafted C code.

Disks, folder structures are all highly varied. I'm not surprised that du is sometimes faster.

If you run dust like this: you can see how long it takes to run in single threaded mode
$ export RAYON_NUM_THREADS=1; time dust /

[Davester47]

I ran it like you said in single-threaded mode, and it was almost as fast as du! I guess something about the access pattern really disagrees with HDDs when it's running multi-threaded. I wouldn't suppose it'd be easy to change the order you go through the filesystem, would it?

$ echo 1 | sudo tee /proc/sys/vm/drop_caches
$ export RAYON_NUM_THREADS=1; time dust .
...
711G ┌─┴ .
dust .  0.53s user 1.89s system 5% cpu 48.260 total

[bootandy]

I wouldn't suppose it'd be easy to change the order you go through the filesystem, would it?

Not really.

This breath first approach works quite well with Rayon, Removing the recursion might be sensible but I can't really move to depth first easily.