Use git lfs for file tracking

[efiop]

Need to consider and investigate possibility of using git large-file-storage with dvc.
https://github.com/meltingice/git-lfs-s3
https://github.com/git-lfs/git-lfs

[colobas]

I was gonna suggest precisely this. It's the one thing keeping me from using DVC as of now.

[dmpetrov]

We considered git lfs from very beginning. Previously we found a few issues that stopped us from using git lfs as a "file back-end". Now we have much more information and experience and I'm happy to discuss this subject again.

What stopped me from using git lfs:

1. [Git lfs is not free] There are no free services for git LFS (even for small repositories). In contrast, the current dvc implementation works just fine with free github and bitbucket accounts.
2. [File sizes in data science projects] It looks like we have different points of view (compering to git-lfs devs) on a definition of large file. In git lfs documentation they use image files as examples (MB size range) and "Version large files—even those as large as a couple GB in size" (from https://git-lfs.github.com/). In data science projects we usually start with GB range and go to dozens GBs and sometimes up to hundreds GBs in a single machines. I usually keep in mind a scenario with a few 5-20Gb files (50Gb overall) and "optimize" dvc to this scenario. So, it looks like git lfs developers had a different scenario in mind, not related to data science.
3. [Data science process] Dozens and hundreds of file copies might be created because of iterative nature of data science and ML projects (model tuning). It is important to keep track of every single file but it does not make sense to push all the files to server. So, custom mechanism of data pushing to server is required while all data stay in a local machine.
4. [Cloud support] git lfs does not support clouds. Your repository and data files are tightly connected to git server. Your infrastructure is probably deployed in one of the clouds and an external data heavy service outside of your infrastructure might be not the best idea. One S3 analog exists -git-lfs-s3, but GCP and others - does not. And again, it doesn't look like an issue when you work with a small amount of data (see item 2 in the list). Also, cloud support helps dvc to easily overcome issue (1).
5. [Unnecessary copying] git lfs keeps file content (actual files) in .git/lfs directory and copies data to the working area with git checkout\pull commands. It is not an issue for small files (<2GB), but for big files it takes minutes, for 50Gb files git checkout takes half an hour. Current dvc implementation solves this issue by symlinks (so, it takes a second).

Now we need to think if some of these issues are actually important or how to overcome the limitations.

At the first glance only the first issue looks like a deal breaker. (4) does not looks like a big issue, (2) and (3) might be not important for many users and these issues can potentially be mitigated by "garbage collector" command on top of git lfs (like dvc gc).

The current dvc implementation solves the issues from the above but it has a few problems:

1. Symlinks in data/ directory do not look good.
2. We cannot track custom data file modifications vi data/file.txt because the cache dir has to be in .gitignore

An interesting technical question - can we support both?

Any ideas are welcome.

[efiop]

@colobas Hi! Could you elaborate on why is it stopping you from giving dvc a try? Maybe we could help.

[colobas]

Hey @efiop ! @dmpetrov 's reply to my first comment was very elucidative, so my current opinion is not quite the same I had when I first commented. At the moment of writing I was considering using dvc for a personal, academic project which requires me to apply several transformations to a dataset and try different stuff on each of the resulting transformed datasets, so I wanted something to help me keep track of all those versions. I liked the fact that I could use git-lfs and keep "everything together", but indeed the stuff pointed out by @dmpetrov changed my mind.

Thanks for the concern!

[dmpetrov]

git annex uses similar approach as dvc - "managing files with git, without checking the file contents into git" and it also based on symlinks.

It has no relation to data science, no reproducible features but still worth to look at.

It looks like people lost interest to git annex project after git-lfs release. It makes sense since git-lfs looks better for a regular large-file use cases. However, in data science context git annex model (and dvc model) looks more promising ("users can ... decide for every file whether to make it locally available.")

Update: "It aims to solve the problem of sharing and synchronizing collections of large files independent from a commercial service or even a central server."

[dmpetrov]

Question	Git LFS	Git Annex	DVC Git
Create repo	?	git annex init	dvc init
Local storage dir	.git	.git	.dvc
How to define "large file"?	Add file name pattern to .gitattributes	git annex add file.txt	Output of dvc import or dvc run
How checkout works?	Git hook. It copyies files to work space	Git hook creates symlinks without copying	Git hook OR dvc checkout create hardlinks without copying.
Storage for large files	A special Git server (no production ready servers so far except GitHub\GitLab)	Many cloud storages can be specified: S3, GCP, rsync, etc	A single cloud storage: S3 or GCP
Special commands	git lfs track '*.jpg', git lfs ls-files, git lfs pointers	git annex sync [--content], git annex get . (content), git annex drop file.txt, git annex unlock file.txt (modify), git annex fsck	dvc import https://example.com/file.txt file.txt (replace to source), dvc push file1 file2, dvc pull file1 file2, dvc gc file.

Special DVC scenarios (data science related):

1. *DVC run and repro we should not probably consider in this list since it is not DVC-Git related feature, DVC-Repro
2. DVC assumes that project has a few important branches and many not important experiment branches. It has an advanced feature to clean caches for not important branches dvc gc.
3. Syncing data only from current (important) branch by default - dvc pull, dvc push
4. A special Git Workflow visualization tool dvc show workflow (it is probably related to DVC-Repro).
   To be continued.

[dmpetrov]

Closing...

DVC is different to Git LFS because it tracks code and data separately: code through Git and data through local cache and clouds. So, Git LFS cannot replace DVC.

However, with the current design, Git LFS can be used for the data tracking. You just need to put dir .dvc/cache under control of Git LFS.