How to pull/retrieve directly to the external directory from remote instead of creating local copies?

[shahriar0651]

I am facing a challenge and would appreciate your help in finding the best solution. I'm currently working on a project where I want to push/version data from an external location to a remote storage using DVC, and subsequently pull/retrieve updates from the remote repository back to the external location. In this process, I intend to keep only the .dvc files in the local workspace.

To provide more context, here are the specific requirements:

• Tracking an External Dataset: I have an external dataset located at /external-to-workspace/dataset, and I want to track this dataset using DVC.
• Uploading Dataset Versions to DVC Remote: I need to upload different versions of the dataset to a DVC remote using ssh, which is located at ssh://example.com/path/to/storage.
• Updating the External Dataset from DVC Remote: Instead of creating a local copy of the dataset when pulling/retrieving from the remote, I want to update the external dataset itself. This means that the changes made to the dataset in the DVC remote should be reflected in the original external location.

I believe DVC provides the necessary functionalities to accomplish this, but I am unsure about the exact steps and configuration required. If anyone has experience with a similar setup or any suggestions on how to achieve this, I would greatly appreciate your guidance.

Thank you in advance for your help and expertise!

[dberenbaum]

* **Tracking an External Dataset**: I have an external dataset located at `/external-to-workspace/dataset`, and I want to track this dataset using DVC.

See https://dvc.org/doc/user-guide/data-management/importing-external-data for how you can do this.

* **Uploading Dataset Versions to DVC Remote**: I need to upload different versions of the dataset to a DVC remote using ssh, which is located at `ssh://example.com/path/to/storage`.

This is also covered by the guide linked above.

* **Updating the External Dataset from DVC Remote**: Instead of creating a local copy of the dataset when pulling/retrieving from the remote, I want to update the external dataset itself. This means that the changes made to the dataset in the DVC remote should be reflected in the original external location.

This is not possible with DVC now, although you could of course checkout locally and then upload it back as needed. We have explored this workflow in the past but it becomes clunky with the way DVC works otherwise. Could you explain more about your workflow and why you need to overwrite your source data directly in its original location?

[shahriar0651]

We are working on a collaborative project where every user is using their own copy of the GitHub repo in the same Ubuntu server. To save space, we have a dataset folder in common space (outside of the git workspaces), and everyone is using that directory to run their code from their individual git workspace. So, we wanted to have a way to version control that external dataset using DVC (both push\pull) to\from DVC remote and keep following the same working style.

However, I also found it challenging to this idea. Hence, we have adopted a more straightforward approach. The data creator has initiated a GitHub repo (along with DVC) in his/her local machine and only using that to push them remotely. The remaining users are also maintaining their own repo to pull the data to their workspace. As everything is happening in a single Unbunut server, there are some data duplications in each of the users' repo, but still, it is giving them to flexibility to decide on their data version, which is the beauty of DVC.

[shcheklein]

For the record, I think I answered the same question on SO recently https://stackoverflow.com/a/76643634/298182 . It has some other ideas on how to organize things.