Dataset storage improvements

[dmpetrov]

There were many requests related to datasets storing which might require a redesign of DVC internals and the cli API. I'll list the requirements here in the issue description. It would be great to discuss possible solutions in comments.

1. A global place for all the datasets. People tend to use a single DVC repo for all their datasets. Otherwise, the number or git-repos explodes.
   1.1. Reusage. How to reuse these datasets from different projects and even repos?
   1.2. List all datasets.
2. Dataset versioning.
   2.1. Assign a version/tag/label like 1.3 to a specific dataset. Git tag won't work since we don't need a global tag for all files.
   2.2. See list of versions/tags/labels for a dataset.
   2.3. How to checkout a specific version of a dataset in a convenient way?
   2.4. Ability to get a dataset (with specified version) without Git. ML model deployment scenario when Git is not available in production servers.
3. Storage visibility for not technical folks like managers.
   3.1. Human readable cache would be great. Thus manager can see datasets and models through S3 web.
   3.2. If 3.1. is not possible - some UI is needed.
4. Diff's for dataset versions (see 2.1.). Which files were added\deleted\modified.
5. Datasets synchronization between machines. It looks like DVC solves this. Should we improve this experience?

Bonus question:

1. Access control. How can I give access to a dataset1 but not to dataset2 to a particular user?

The list can be extended.

UPDATE 1/15/19: Added 2.4.

A global place for all the datasets.

Why a shareable directory (NFS, S3 bucket, etc.) is not enough? (this would also cover access control)

Human readable cache would be great.

Indeed, maybe we can store our files with the following naming: .dvc/cache/version/path relative to root dvc + checksum
This way, we can know if a file was renamed and the modifications the file had in a version.

Datasets synchronization

Just to be on the same page, we are talking about dvc push and dvc pull right?

[dmpetrov]

@MrOutis "Why a shareable directory (NFS, S3 bucket, etc.) is not enough?" - great question! I'd separate the question into two:

1. Why the flat versions structure in a file system/S3/HDFS (like s3://my/dataset/{ver1, ver2, ver2_cleansed, ver3}) is not enough and why people need versioning tools?

• The dataset is evolving fast (new files every week or you modify files too frequently). You don't want to duplicate files by storing copies in each version directory.
• You build a dataset out of other datasets and don't want to duplicate files. Two raw datasets sources/ds1/{ver1, ver2, ver3 ...} and sources/ds2/{ver1, ver2, ver3 ...}. The third one is based on the first two - sources/myds/ver1 needs to contain sources/ds1/ver5 and sources/ds2/ver2.
• You want a single "pointer" to a dataset like latest or just a a link to dvc repro.

2. Why the current DVC is not enough?

• The number or git-repos explodes if you use a single repo for each of the datasets. A team don't want to manage 20+ git repositories.

".dvc/cache/version/path relative to root dvc + checksum" - we might have problems with duplication. The same file might be presented in a few different locations of a repo. Just a fun fact - the initial version of dvc (0.8) worked just like you described but we decided to use a single cache dir.

"Datasets synchronization" - yes, dvc pull\push.

@ternaus you are using a single repo for many datasets. Could you please share your motivation?

@villasv and @sotte, do you guys have any thought regarding this topic?

[villasv]

1. I'd keep the current cache directory flat structure and sync mechanism

Considering that DVC is a command line tool, I think S3/GCP/NFS/etc should still be the backing storage, which doesn't mean that their issues can't be addressed. I think Git is a role model in terms of "don't worry about what's inside .git", but with effort you can make it sane to inspect what matters.

1.1 Git has the concept of submodules. Admittedly, I find it a huge PITA and avoid it like the plague, but it also shows that it's feasible to reuse projects by composition/linkage. I've never seen how it works internally though, so I'm not sure if it's really something possible to replicate. In fact, git submodules sucks so much IMO much that I'd slap someone that suggest DVC to build on it. Assuming that importing data from another project's cache would be read-only, this might be simpler than it looks.

1.2 Listing all datasets requires us to disambiguate which files in the cache represent the same dataset, and probably be selective about versions as well. But once versioning is figured out, I think this is given.

2. Git handles tags the same way it handles branches, it simply saves an alias (properly called a ref) for a specific hash. Copying this mechanism allows DVC to keep the cache structure flat and makes checking out easy. The difference is that a git ref points to the state of the entire repo, while a dvc ref would point to the state of a single file, so the ref needs to know the file name as well.

3. Human readable cache sounds nice, I know. I too find myself inspecting cache files sometimes. But I only do that because I don't know how to checkout old versions and I can't diff them sanely. Is there really any other motivation for having human readable cache files? (except storage visibility for stakeholders)

3.2 Yes I would like to have UI for that, just after saying that I want the cache directory flat. Nothing fancy, though. Just list all refs, an aggregate count of versioned files without refs, total size. A stakeholder-friendly version of this might be a web version with nice-to-know things, like GitHub's useless graph of my most active days on the week.

4 Diff is tricky. I've scripted my own, but it's something very opinionated and dataset comparison is specially complex because sometimes the base object of comparison is not the row, but rows identified by some ID (assuming the dataset is ordered, otherwise it may be just implausible). Also, diffing might change by file format.

5 Security is a hairy topic. It's achievable with S3 access policy bending, but that's not stakeholder-friendly. Maybe combining data reuse from point 1 and a UI for versions... though I can't see how one could achieve meaningful access control without deeply integrating with the remote storage chosen.

My concerns:

1. What's the expected behavior of checking out a specific version of some data file A that has been generated by a script S, but that script also generated another file B and depends on input X? Are A, B, S and X versions aware of each other?

2. Does it really make sense to tag a single file? Git tags the entire repository because having a single file from another revision is an invalid state, as it should be with DVC. Isn't tagging all data files simultaneously enough, like one consistent repository?

3. Can I reproduce the output of the pipeline after checking out an old version of A? Or is the expected behavior simply to checkout A's correct version for the current code?

[dmpetrov]

Great feedback @villasv ! Thank you.

You are a DVC expert and it looks like you have an opinion on how to organize projects with DVC. Can I ask you a general question? How would you organize projects for a team which has 20 datasets and 30 projects (ml, analytics, data processing)? One project can use many datasets. One dataset can be reused in many projects. Options that I see:

1. A separate git-repo for each of the projects.

• do you need a "golden dataset" and how will you support it (keep datasets copies in S3\GCS)?

2. A mono-repo with all the datasets and projects.
3. Some combination of (1) and (2).

Which features in DVC are missing to support this scenario?

Details about your feedback

Totally agree with the cache directory structure and sync mechanism.

Yes, we should think about projects composition/linkage. The question is - should each dataset be presented as a single module/repo (so, 20+ repos/datasets for a single team) or we should come up with a structure where a single repo can naturally fit all the datasets (1 repo).

Re tags - this might require a separate tagging mechanism in addition to Git-tags. This is what I don't like but it might be the only solution.

"Human readable cache". Right, shareholders visibility is a major motivation. The minor one - it might lower the entrance bar to dvc (good for newbies). Yeah, some UI would be great if we keep cache structure.

"Diff" - agree. It might be enough just to have file counts (new, deleted, modified) and sizes.

Regarding your concerns:

1. In the context of this issue I was talking only about checking out datasets\sources, but not data derivatives.
2. It does make sense to version a dataset (with or with no DVC). The question - how to (should we) incorporate this feature into DVC? The problem happens when
3. If I understood the concern correctly - checking out an old dataset version should work for both of the scenarios. The primary motivation in the current issue - use a dataset version in a current (or new) code.

[villasv]

Ah, thanks for clarifying. I had the impression that versioning would also be targeting data derivatives. Indeed, that makes all my three concerns vanish, because input files are the root of the dependency trees so checking out a specific version of them can be done in such a way to keep everything consistent.

I didn't mean to use git tags though, what I mean is that DVC could somewhat copy how git tags are done, which is basically a refs directory with alias files. E.g. a file named version 1 contains a single line with the hash of the cache file it references. Like I said, the major difference is that extra attention is needed to also associate the ref with the original file to avoid conflicts.

Q: How would you organize projects for a team which has 20 datasets and 30 of projects (ml, analytics, data processing)?

If a group of projects code in scripts or data derivatives, I'd go for a mono-repo. I think that even if these shared only the dataset/sources I might go with a mono-repo, if those datasets/sources aren't very stable, because it's the safest way to ensure everyone is interpreting that data the same way.

Separate repos would be fine if those projects are mostly code and data, or they derive from a "golden dataset" that changes never or very slowly. At my work, we have two repos using DVC but they share nothing directly (one is for curation and the other is the result of that curated data being used in production, so one affects the other, but the files are totally independent because the second has a database dump as source).

In fact, the first of these projects - the one for curation - even uses the same repository for the NodeJS API that serves that content and other stuff like Elasticsearch mappings to index the derived data. Those are very coupled and I see no reason to separate them.

Q: What could be different if I had a way to version datasets and share data between repositories?

The first scenario (shared scripts and data derivatives) wouldn't change IMO, if I can't version and distribute derived data as well. I think the scenario of single-source-of-truth is the one that really benefits here.

Why dvc import is not enough? At first glance, the only missing feature is that you don't have explicit versions to understand. Maybe if dvc import was extended to import from another projects cache, it could handle dataset versions semantically.

[polvoazul]

@dmpetrov

How would you organize projects for a team which has 20 datasets and 30 projects (ml, analytics, data processing)?

I think having a monorepo or 30 repos depends on other factors apart from dvc. Dvc could offer full non-duplicated support if we store all dvc files somewhere else such as ~/.dvc (user) or /var (global).
There is no real advantage of having .dvc in the git directory if you assume there is a way to dvc pull your files.

Docker already does this by storing all images, and cache layers in /var, and i find that it works quite well

[polvoazul]

I will give my impressions on your questions:

There were many requests related to datasets storing which might require a redesign of DVC internals and the cli API. I'll list the requirements here in the issue description. It would be great to discuss possible solutions in comments.

1. A global place for all the datasets. People tend to use a single DVC repo for all their datasets. Otherwise, the number or git-repos explodes.

True, this is valuable. I suggest using a per-user or per-machine dvc directory (~/.dvc or /var/dvc). See comment above.

1.1. Reusage. How to reuse these datasets from different projects and even repos?
1.2. List all datasets.

Simple to implement if everything is in one place

2. Dataset versioning.
   2.1. Assign a version/tag/label like 1.3 to a specific dataset. Git tag won't work since we don't need a global tag for all files.

I dont like this very much. The version is the git commit hash + comment on the related *.dvc file. Use git for all your versioning/identification problems. Could be helpful to name the dvc cache files with something human readable (something like <MD5>_<commit message>)

2.2. See list of versions/tags/labels for a dataset.

Just git log related *.dvc file. If you want to store metadata such as size, or number of lines. do it in .dvc file. Its just a simple text file, all git tooling works beautifully.

2.3. How to checkout a specific version of a dataset in a convenient way?

git checkout branch/tag; dvc pull;
or
git show branch:path/to/file.dvc > tmp.dvc; dvc pull

3. Storage visibility for not technical folks like managers.
   3.1. Human readable cache would be great. Thus manager can see datasets and models through S3 web.

This would be very nice indeed. More metadata would be stored in .dvc files, but it is worth it! Maybe we could show a preview of the data such as the shape, columns and the first ten lines.

3.2. If 3.1. is not possible - some UI is needed.
4. Diff's for dataset versions (see 2.1.). Which files were added\deleted\modified.

use git? i dont understand this. I usually have one dataset per .dvc file so it is very simple.

5. Datasets synchronization between machines. It looks like DVC solves this. Should we improve this experience?

Yes. For big files S3 loses connection sometimes. Maybe there is a more robust s3 download/upload recoverable experience. Also i think most files are some form of tabular format. We could give special support to some formats (such as csv) and upload only deltas. This would be VERY NICE. Think rsync --partial

Bonus question:

1. Access control. How can I give access to a dataset1 but not to dataset2 to a particular user?

Just use S3/filesystem access control. Leave filesystem permissions to filesystems and we avoid a difficult subject.

The list can be extended.

If i think of anything new I will edit!

[dmpetrov]

@villasv thank you for the clarification!

Please correct me if I misunderstand your point. You are saying that a global dataset repo\place is not a perfect solution because it does not provide code and data lineage (snapshots) and the environment becomes fragile. In this case, a mono-repo does not have such kind of problems as you said.

If the above is correct, how about using "data repos" with versions or checksums like dvc data-repo https://github.com/iterative/mydataset1 6cdc2cb or tag ver1.3 instead of the checksum? There is an analogy with packaging systems.

dvc import - yes, the solution can be based on a "golden dataset" somewhere like S3 plus proper imports for versioning from repos (not matter how often it is changing). You mentioned explicit versioning issue - I'll think about it. Please let me know if you have some ideas.

[dmpetrov]

@polvoazul great feedback!

Awesome idea with per-user or per-machine dvc directory. And great analogy with Docker images.

I kind of agree with you about git commit hash + comment on the related *.dvc file but it looks like too heavy operation for a significant portions of users. Many people have asked about tags per datasets\dvc-file. I'm trying to understand if it is okay to keep such an advince scenario as we currently have and transfer all the complexity to documentation or it is better to simplify API.

Regarding the S3 connection loss - right, there is a corresponding issue #829.

[villasv]

I wasn't favoring a mono-repo each with its own cache against having a global cache like polvoazul suggested, only against many-repos each with its own cache. And caching is not the argument here, but sharing and lineage.

I didn't think about a central cache. At first glance, I have nothing against it... but it doesn't change much in the scenarios I've encountered so far. I'd still use mono-repos for projects that share data lineage because the transformations scripts are part of that lineage.

I'm associating the word "repository" here with actual git repositories, I don't see a "dvc repository" dissociated from it. I haven't looked at dvc as a data registry solution so far, only as an integrated LFS/Annex that properly handles reproducibility. Data registry is a nice thing though.

It seems that much of this thread is about making the cache into a proper registry. I think this can be done without actually changing anything about the cache inner structure.

[efiop]

@villasv I agree, we will probably add support for a few cache locations like in #1450 . So that you could have your local cache at .dvc/cache for your local files, as well as maybe /share/dvc/ for global datasets(e.g. some community repo with a special dataset). So the logic of cache directories might not actually change a lot compared to the current one and actually be a natural extension of the current approach.

[dmpetrov]

@villasv yeah, data registry is a great term. Let's start using it.
Where to keep cache is just an optimization question. The biggest question - how to simplify the dataset experience.

I've created a proposal based on the discussion. The proposal assumes that we keep dataset metainformation such as dataset name and tags into dvc-files. And it opens a set of command to manipulate with the dataset as a separate type of objects. But in fact, it is just a syntax sugar on top of data files.

The proposed dataset API looks like a pretty reasonable solution which can bring DVC to a next level and does not break distributed philosophy of DVC and Git like many other dataset versioning solutions do. The only problem that the proposal does not solve is storage visibility. The cache is not humanly readable which is probably not a surprise. And we should go with some additional UI (3.2.) probably as an addition to DVC project.

The proposal: https://gist.github.com/dmpetrov/136dd5df9bcf6de90980cec22355437a

Looking forward to your feedback, guys. Any comments are welcome.

PS: @drorata you had some issues with files versioning and updating versions. You might be interested in participating in this discussion.

[drorata]

It's a little hard to join such an in-depth discussion. I will try to share some of the thoughts that I had while reading the comments.

For me, one of the most important concerns which DVC addresses is reproducibility. When it comes to datasets, reproducibility means IMHO the combination of what raw data was used and what transformations were applied. If some (processed) dataset is used by several projects, it becomes in some sense a project of its own, and thus it would make sense to keep it separately. The depending projects, which use this set as an input, would then tap to the same source and enjoy a fixed reference.

I found git-submodulesuseful and it can fit into the setting I described rather nicely. A project can fetch the datasets it needs by adding the "data-projects" as sub-modules.

The problem of diffs of datasets is somewhat enigmatic. If the dataset is binary one is rather lost and metadata associated to the set might be the only remedy.

Personally, I like a lot the close coupling between DVC and git. If at some point, I realize that part of the project/data can be used by other projects, it means it has to be extracted into its own world. It is a bit like the flow when you code something in Jupyter, and then you realize it is used across different parts of the project, so you extract it as a module. Then you figure out it is actually useful in different projects, so you turn it into a package on its own. Along with this line, I don't understand how linking DVC to the user/machine is going to work; would not it make the link between the data and the project much less clear?

@dmpetrov Thanks for pinging me 👍