Add parallelism to checksum calculation for pulling multiple large files

[Ykid]

when checking out, we do it sequentially, stage after stage, that is why checksums calculations are sequential

It would be better if multiple stages (which might correspond to large files) can perform checksum calculation at the same time.

[Ykid]

@pared

[efiop]

@pared Do you remember the context? Aren't we trusting pulled checksums these days? Or is it about unpacked dir case?

[pared]

@efiop @Ykid Sorry for the delay,
Here is the start of our conversation:
https://discordapp.com/channels/485586884165107732/563406153334128681/682124179885260804

@Ykid has a repository where there are few big files (A, B, C), each having its own stage (A.dvc, B.dvc, C.dvc). @Ykid noticed that checksum calculations on dvc pull take a lot of time.

So, I see 2 potential problems:

1. Files were on NFS, so reading files and feeding it to hash function might be taking a lot of time.
2. We do parallelize checksum computations, but on out level. What I mean by that, is that when checking out here we do it stage by stage, so we will have:

• checkout A.dvc
• checkout B.dvc

and so on.
And as checksum calculation happens down the road of stage.checkout our checksum calculations will be sequential (because stage checkouts are sequential).

So I would vote for changing the name of the issue to Parallelize stage checkouts

[pared]

@Ykid
What version of dvc are you using?