I've recently moved to a new file system that supports hardlinks, and have set cache.type to hardlink,symlink. This greatly reduces my inode count, which makes the sys admins happy, and a number of tools that struggle to "see through" symlinks now work without having to dvc unprotect first.
When I dvc checkout --relink to replace my old symlinks by hardlinks, the process is delightfully fast, even for thousands of files summing to several TB in storage.
However, when I dvc add a folder with several TB of data I still have to wait hours for DVC to copy the data to the cache and then create links. When I was stuck with symlinks, I accepted this as the price to pay for the many benefits of DVC. But now that I have become accustomed to hardlinks, I wonder if all this copying is really necessary?
Surely DVC could just create a hardlink in the cache for the files in the workspace? The net result -- to file names pointing to the same inode -- will be the same, but the process should be orders of magnitude faster.
Obviously, this only makes sense on file systems that support hardlinks (and perhaps reflinks, altho I'm less certain there). And even then there might be ownership obstacles sometimes. But when possible this would be a great boost .. unless I'm missing something.
I've recently moved to a new file system that supports hardlinks, and have set
cache.typetohardlink,symlink. This greatly reduces my inode count, which makes the sys admins happy, and a number of tools that struggle to "see through" symlinks now work without having todvc unprotectfirst.When I
dvc checkout --relinkto replace my old symlinks by hardlinks, the process is delightfully fast, even for thousands of files summing to several TB in storage.However, when I
dvc adda folder with several TB of data I still have to wait hours for DVC to copy the data to the cache and then create links. When I was stuck with symlinks, I accepted this as the price to pay for the many benefits of DVC. But now that I have become accustomed to hardlinks, I wonder if all this copying is really necessary?Surely DVC could just create a hardlink in the cache for the files in the workspace? The net result -- to file names pointing to the same inode -- will be the same, but the process should be orders of magnitude faster.
Obviously, this only makes sense on file systems that support hardlinks (and perhaps reflinks, altho I'm less certain there). And even then there might be ownership obstacles sometimes. But when possible this would be a great boost .. unless I'm missing something.