Multiple users and cloud-storage - sharing cloud metadata without knowing

[snowpong]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

We are using targets with repository = "aws" and with endpoint pointing to an AWS compatible server we've got running locally (min.io). We use git for tracking code changes, and we also track _targets/meta/meta in git. We're multiple developers working on the same project.

It's been working fine, but we've sometimes noticed strange entries in the diff for _targets/meta/meta when checking in. New targets occurred with questionable origin, or existing targets changed although we didn't see an obvious reason for this.

It dawned on us today, while collaborating, that _targets/meta/meta (and friends) are constantly synced to the cloud, and there is no separate copy per user. We're essentially sharing the targets metadata. So if I am locally working on problem A in my checkout, and my co-worker is working on problem B in his checkout, whenever we're running tar_make() we're syncing to the same cloud metadata. Is this true?

The default in tar_option_set() for repository_meta is tar_option_get("repository") which in our case is "aws".

If the above is true (we're sharing metadata), is the proper solution in our case to change repository_meta to "local"? We want to share target objects between us, but the metadata we want to keep local per user until a new git commit is pushed.

[wlandau]

To ensure the metadata only lives locally, yes, repository_meta = "local" is the way to do that. Alternatively, to avoid conflicts with both metadata and target data, it may help to have each colleague use a different prefix in the bucket. Collaborating on a shared data store is tricky and prone to these kinds of issues unless you have a nice system to guarantee only one person runs the pipeline at a time.

[snowpong]

I tried, and it seems to work correctly now. We use min.io with versioning so I don't think there are any issues now even if running pipeline at the same time.