How is the MD5 hash of entire directories in the .dvc files calculated?

[mblsf]

We are using the MD5 hashes from the .dvc files to check the consistency of the stored files. When storing entire directories, DVC calculates a MD5 hash with the suffix .dir. How is this MD5 hash calculated? We will need this MD5 hash to check the consistency of the entire directory...

Thank you very much.

[shcheklein]

Hey, I think this link should give an answer https://dvc.org/doc/user-guide/project-structure/internal-files#directories (specifically, the Directories subsection). Please take a look and let us know if that is clear and if it solves your need check the consistency of the entire directory. Btw, could you expand a bit on it?

[mblsf]

Thank you very much for your reply. I've already looked at the link and for the example there, I want to know how the hash 6fdb5336fce0dbfd669f83065f107551.dir is calculated from the 2 files with the hashes 402e97968614f583ece3b35555971f64 and de7371b0119f4f75f9de703c7c3bac16.
We are using the DVC API to download files. If the DVC process is killed during the download, an inconsistent file remains. So we check the hash of the file and re-download the file if it doesn't match. If the hash matches, we'll keep it

[pmrowla]

.dir files are JSON files that contain a listing of the files in the directory. The hash is the MD5 of the JSON content. This hash would mainly only be useful to know whether or not you have downloaded the correct JSON content for a given .dir file.

How are you using the API to download files?

Also, just to be clear, if you are using DVC 2.x, the MD5 hash computed by DVC is not always the MD5 of the original file content, and you cannot compare the DVC MD5 hash to the results of something like md5sum (or python's hashlib.md5)

[mblsf]

Thank you very much. I already tried this and it didn't worked, but now I found the mistake that I've made.

We are using DVCFileSystem with DVC 3 and aware of the changes regarding the calculated hashes compared to DVC 2.x.