Copy in serial?

[jamestalmage]

It seems really unlikely to me that copying a bunch of files in parallel would speed anything up. Indeed, I would not be surprised to find it slowed things down. Increasing the number of files read simultaneously seems likely to encourage lots of seek delays for HDD's, and SSD's should be able to saturate their bandwidth regardless of how many files are being copied simultaneously.

It's probably worth running some benchmarks to prove this, but I think you would have seen Operating Systems commands / GUI's doing parallel copies if there were any advantage to it.

I guess technically, you could be copying from multiple slow network mounted drives - in which case parallelization might make sense. But I think that's a very small (possibly non-existent) percentage of users.

[sindresorhus]

I think it really depends on what you copy. My theory is that lots of small files would be faster to copy serially, especially with sync fs methods, but with large files we could see a speed-up with parallelization. Only a wild guess though, so would need some benchmarks. At the very least, we should limit the concurrency.

[YurySolovyov]

At the very least, we should limit the concurrency.

We can start by replacing Promise.all with https://github.com/sindresorhus/p-all and choosing some finite number for concurrency

[sindresorhus]

@YurySolovyov Yup. We should do some benchmarking on this, but I think (os.cpus().length || 1) * 2 or (os.cpus().length || 1) * 3 is generally a good default.

[YurySolovyov]

@sindresorhus I'm just not sure where the bottleneck is, if it is CPU threads, yes this is reasonable, if it is number if file descriptors, then it should be something much higher, like 64 or 128

[sindresorhus]

Hence why we need benchmarks.

[YurySolovyov]

@sindresorhus do you have an experience of writing good ones? (I would like to learn if so)

[kevva]

@YurySolovyov, you could use https://github.com/logicalparadox/matcha for example.

[schnittstabil]

@YurySolovyov If you want to read some code: globby uses matcha (bench.js)

I don't fully trust results of matcha, but neither I can remember why nor do I know a good alternative – good benchmarking tools are extremely rare, for every language and platform.

Maybe @jamestalmage have had similar concerns when he handcrafted the AVA benchmarks, great work btw, however I would prefer some library, even though it's matcha in the end.

[kevva]

There is https://github.com/bestiejs/benchmark.js too.

[sindresorhus]

I think the easiest solution here is to just add p-all and expose a concurrency option, which defaults to (os.cpus().length || 1) * 2.