Mechanism to update a dataset w/o downloading it first

[shcheklein]

Story

When I have a dataset with 1M images
I want to update it- add one more image file
I have to download and checkout the previous version first now
It takes long time

Request

Come up with a set of commands/options and a flow to do this efficiently w/o downloading data first

[MetalBlueberry]

I'm facing a similar issue and I think this can be easily fixed in the case where the entire folder is tracked by dvc.

If someone wants to upload a single image to a folder tracked by dvc

• calculate md5 of the upload
• upload it to the remote
• get the md5.dir from the directory
• append the file md5 to the list.
• upload the new md5.dir to the remote.

just 2 file uploads and 1 download.

Maybe the solution is to allow the user to run dvc pull --dir-only to download only md5.dir references. That's all that dvc needs to upload a new file to a folder.

[puhoshville]

Fully agree with @MetalBlueberry: it will be nice to have an option to pull only directories from remote (dvc pull --dir-only).
Also, the high-level command for this case can simplify user experience:

dvc add --merge-into data.dvc data/ 

which will perform the following operations:

1. dvc pull --dir-only
2. add new files to relevant dirs:
   1. update dir's content
   2. remove previous md5.dir file (since the content of the file determines its name)
   3. create new md5.dir file

As additional feature we can implement merge of two different dvc files of same dir:

dvc add data/ --file data_update.dvc
dvc add --merge-into data.dvc data_update.dvc

This feature can be useful when teammates working in different branches and different data were collected.

@shcheklein @efiop, I can start the implementation of this feature, if you're okay with this view.

[efiop]

@MetalBlueberry @puhoshville Great ideas!

My 2c:

dvc pull --dir-only

Usually we just automatically pull .dir files when we need them without asking. E.g. https://github.com/iterative/dvc/blob/fcdb503b4eff1297e5c1c1ed18f5606fe780e481/dvc/output/base.py#L383 , we could probably do the same here, without adding a special --dir-only flag.

ii. remove previous md5.dir file (since the content of the file determines its name)

No need to remove the old .dir file. Dvc doesn't ever remove cache files except for gc, as cache might be used by multiple repo instances and you never know if that .dir file is being used by someone else. And, well, those are pretty tiny files anyway, so no need to be paranoid about cleaning them up during the merge.

@puhoshville I see that you use --merge-into data.dvc, which makes me think that you might be confusing the data with the metafile for it. The operation is adding something to a dir, so it should reflect that in the command. E.g. dvc add data --append , and the corresponding meta file could be just found automatically by dvc.

As additional feature we can implement merge of two different dvc files of same dir:

Hm, if I understand you correctly, you probably mean https://dvc.org/doc/user-guide/merge-conflicts#append-only-directories ?

[skshetry]

We could also create a separate command for it:

$ dvc append data.dvc file --path data/raw/file1
# append `file` as `data/raw/file1` into `data` directory (i.e. the `out` of `data.dvc`).

It will pull .dir files, updates it in the cache and modifies the data.dvc file.

[dmpetrov]

I like the idea of dvc pull --dir-only - it can bring the entire dir structure, help users to navigate in the dir and simplify API as a result.

--merge-into seems like a possible solution but it does not cover file removals. Also, it requires creating a separate dir structure while we can utilize the existing one (if I understand correctly data does not necessarily match the actual dir name dvc add --merge-into data.dvc data/).

Ideally, a user should not create a separate data dir and mess with dvc files (which might not exist if we decide to use run-cache for data files at some point). We should utilize the existing dir structure and regular CLI experience (cp, mv, ls) as much as we can.

It might look like:

$ dvc pull --dir-only data/
$ cp ~/Downloads/users-2020-10-05.tsv data/rawfiles/cust/
$ cp ~/Downloads/report-2020-10-01.tsv data/summaries/  # it can owervrite an existing file
$ dvc add --update data/ # or --append

# how to handle removes:
$ dvc add --remove data/users/update-2020-10.csv
$ dvc add --update data/

# apply all the changes
$ dvc push data/
$ git add data.dvc
...

Navigation

We might need a separate functionality for the navigation in these virtual/partaly-downloaded dirs - like list:

$ dvc pull --dir-only data/
$ dvc list --dir-only data/rawfiles/cust/ # without downloading files
users-2020-10-04.tsv
users-2020-10-02.tsv
users-2020-09-29.tsv
...
# Okay, now I know what to delete\replace
$ cp ~/Downloads/fixed-users.tsv data/rawfiles/cust/users-2020-10-02.tsv
$ dvc add --remove data/rawfiles/cust/users-2020-10-04.tsv
$ dvc add --update

Naming

Agree with @skshetry - a separate command might be helpful. But I'd not limit by append. We also need remote, update and navigate. It might look like dvc vdir add/remove/list.

PS: the navigation and file deletion is not part of this issue. But we should come up with the API that won't block us from these scenarios.

[RafayAK]

Hey fellows, @shcheklein asked me to chime in on this ticket with my use case. So, here it is.

I have created a DVC Data Registry with the intention of updating it automatically whenever one of my edge devices decides to push data to it. It looks something like this:

What I have tried:

Initially, I thought I could just create a sessions dir in the Data Registry and every time I would push to it from any of the edge devices it should just work. That didn't pan out. Whenever one of the edge devices does a dvc add sessions/ it creates a sessions.dvc file and when it's subsequently pushed to Github data registry (git push) and remote storage (dvc push), only the most recent sessions.dvc file is visible, pointing to the most recently pushed data. In, simple words DVC does not merge datasets put into the same directory.
One way to solve it would be that each edge device should first pull all the data, copy the local data to the pulled directory, and then finally push. One major problem that can arise is of race-conditions, eg. if Machine-A and Machine-B start pushing data at the same time and Machine-A finishes first either Machine-B's push may either get-lost, rejected, or be overwritten. Secondly, for every edge device to first pull data before pushing is really bandwidth and storage-intensive and just an overall naive solution to the problem at hand.

The other way I was guided on the discord channel was to add data with more granularity, instead of just adding a directory as a whole, so that each individual machine can push data to a directory independently. Something like this:

___________
Data Registry
___________
> Session
   > Data_A_timestamp.dvc
   > Data_A_timestamp
   ...
   ...
  > Data_C_timestamp.dvc
  > Data_C_timestamp
 ...
 ...

In this case, there is no monolithic sessions directory (no more sessions.dvc) tracked by DVC, instead, all the edge devices are doing something like this dvc add sessions/Data_A_timestamp which generates its respective .dvc file. This also has one problem, edge devices will still need to clone and keep the data registry from Github updated locally for a successful push.

I have not settled on a definitive answer to this problem. Hopefully, adding my two cents to this issue will help me in finding a definitive answer.

[MetalBlueberry]

Hello again!

We've developed a system to upload files to a dvc tracked folder without actually using dvc. It basically performs the steps described in the previous comment.

seeing this post, looks like there are more people interested in having cron jobs to upload data to dvc tracked repositories from edge devices. is it worth developing a small cli app for this purpose?

• no python/git dependencies
• focus on optimal file upload
• delete uploaded files so it can run on devices with small hdd.

If I find people interested in this, I will be really happy to implement it.

[RafayAK]

@MetalBlueberry that sounds great. I would love to have that feature in DVC or try out your system (if you've open-sourced it). If any help is required in implementing this feature, I'm completely up for it.

[shcheklein]

@MetalBlueberry you got me intrigued :) Could you share some details please? Especially, regarding the no python/git dependency. Primarily my understanding is that at some point to make DVC-tracked data accessible we have to save the new checksum somewhere (usually Git via .dvc) file. Otherwise, even if we, let's say, push the new, modified .dir, it's not clear how to we read it later? how downstream system "understand" that there is some change?

@RafayAK thanks for taking your time, btw sharing the use case. We are working right now on a POC for this problem. I can't promise that it will be no-git/no-python, but at least update won't require pulling the whole dataset. If multiple machine update it simultaneously- one of the will get a Git conflict and will need to run the operation again. Would it be a reasonable workflow for you?