how to update the data in a data registry within another project

[shelper]

i went through the doc for data registry at https://dvc.org/doc/use-cases/data-registry
but i am still not clear about how to update registry

if I understand correctly, for a project that imports data from a data registry (which is a git repo), and i changed the data in that project, when i run dvc add changed_data.dvc, what i changed is the data for my project, not the repository of the data regitry. and when i run git commit what i commit to is my project's git repo, not the registry's git repo

how could i push back the new data change to the original data registry?

[shelper]

i got answer on discord saying

Unfortunately, there is not currently a very seamless way to push back the changes. You would need to copy the data back to the data registry repo and push it from there.

So i think this could be a feature request then ...

Maybe a workaround is to use data-registry in a project in a way that is similar to git submodule, and use something like git sparse-checkout to checkout only the data for that specific project from the data-registry as a submodule?

[pared]

@shelper
Data registry is a separate project from your training and it should be treated as a "consumer" of the data rather than something that updates the registry repo. So the changes should be applied to data registry repo, and then, using dvc update the data should be updated on your training repo. As I understand you would like to update your data registry from your training repo?

[shelper]

@pared that's right, i wonder if it is feasible (and reasonable) to update data registry from training repo.

[pared]

I guess the right question to ask here is why not update the data registry? Is there a use case where one cannot update the data registry, and should be able to update it from another repo?

[shelper]

I guess the right question to ask here is why not update the data registry? Is there a use case where one cannot update the data registry, and should be able to update it from another repo?

I Agree with you but just a thought here is that if the registry gets bigger, and some one who only used one dataset from the registry will have to clone it to local and update it through GitHub

Just in case the person accidentally contaminates other datasets in the registry, if there is a way to isolate in-between the datasets in the registry?

[shcheklein]

Just a few points to clarify:

I Agree with you but just a thought here is that if the registry gets bigger, and some one who only used one dataset from the registry will have to clone it to local and update it through GitHub

A person can download only single dataset to update it, no need to checkout / pull all the datasets in the registry to update a single one.

Just in case the person accidentally contaminates other datasets in the registry, if there is a way to isolate in-between the datasets in the registry?

yes, you can always update only a single dataset and do git commit dataset1.dvc + git push. No matter if other datasets are updated or not or if you did dvc push for other datasets. This way nothing is contaminated. You are sharing exactly one new version of a specific dataset. Everything else stays safe.

Btw, in your specific case, where do you store your data - cloud, NAS, SSH, something else?

[shelper]

Humm..., please see my comments below based on my understanding..

A person can download only single dataset to update it, no need to checkout / pull all the datasets in the registry to update a single one.

this above step is done by dvc import registry-url dataset-folder, in the project folder

yes, you can always update only a single dataset and do git commit dataset1.dvc + git push. No matter if other datasets are updated or not or if you did dvc push for other datasets. This way nothing is contaminated. You are sharing exactly one new version of a specific dataset. Everything else stays safe.

Here above, the user needs to run the commands in the local registry repo, which means he/she has to clone the whole registry repo to local.

Btw, in your specific case, where do you store your data - cloud, NAS, SSH, something else?

I believe it does not matter, but for example, in cloud. if what i understand is correct as commented above, then here is what happens

• someone works in project folder and dvc import from remote registry repo, and run dvc pull to get data from cloud
• he/she changes the data in project,
• if he/she now wants to update the data change to registry, he/she first runs dvc push to copy any new data to the cloud storage.
• he/she then clones the registry repo to local and copy the update dvc file from the project folder to local registry repo folder and rum git commit and git push

if that is correct, my only concern is that it is little tedious 'cus the user needs to switch between project and registry folder, i wonder if there could be a safer way to update the registry without leaving the project folder

if that is the way it should work, then nevermind

[shcheklein]

Here above, the user needs to run the commands in the local registry repo, which means he/she has to clone the whole registry repo to local.

That was my point. They need to do git clone data-registry (that's should be fine, right?) , but no need to do the full dvc pull. A specific dataset can be pulled - dvc pull dataset1 and then committed dvc commit dataset1 .. It can be granular.

There is a way to avoid even the second copy on a machine where they would be updating a dataset that needs to be updated. data-registry repo can be setup to point to a project repo that where dvc import was used with dvc cache dir. It's a bit more involved, but this way there will be no additional downloads, data copy, etc, etc.

[shelper]

That was my point. They need to do git clone data-registry (that's should be fine, right?)

that is fine most cases, but it concerns me a little bit when some unexperienced github user cloned the registry repo and made some changes to it unexpectedly. expecially if he/she changes datasets other than the one he/she is using.

So i just wonder if there is a way to restrain the user only to make changes to the dataset he imports/gets from the data-registry. and by make changes i mean commit changes to the original data-registry. That is why i thoght it would be useful if we can directly commit the data changes (dvc file) to the data-registry within the project folder when the dataset is imported from the data-registry

I may be asking for something not have a strong reason for it, I will close this issue for now

thanks for answering my concerns though :)

[dberenbaum]

Hi @shelper! Please don't take the discussion above as dismissal of your points. It's just trying to understand and gather info.

I think it makes sense that you want to update the data registry from the consumer repo, and I have heard of others asking for something similar. As you said, it forces inexperienced users to understand a lot about how to make changes in Git. It also feels broken to me that you have to manually copy either the data or at least the .dvc file/checksum from the consumer repo to the data registry repo.

I don't have a good solution for you now, but let's keep this open for discussion!

[dberenbaum]

There's also a related proposal in #8066 for how to upload data back to a repo.