[general] New cloud hashing approach and collaborative workflows

[noamross]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

My team recently realized that this change in targets 1.4 may have significant consequences for our cloud-based workflows:

For a cloud target to be truly up to date, the hash in the metadata now needs to match the current object in the bucket, not the version recorded in the metadata (#1172). In other words, targets now tries to ensure that the up-to-date data objects in the cloud are in their newest versions. So if you roll back the metadata to an older version, you will still be able to access historical data versions with e.g. tar_read(), but the pipeline will no longer be up to date.

We use S3 versioning with a shared bucket for multiple users on a project. It has been excellent for letting us share compute-intense targets and avoiding downloading large targets that can be skipped, even as each user works on different branches of a project where some targets may diverge. As long a version of the target in the bucket exists that matches the local metafile, it could be skipped, and overwriting the latest version wouldn't interfere with others' state. This also means CI builds of targets can take advantage of already-built targets without interfering with development. Under targets 1.3, this requires setting `repository_meta = "local", which is fine. However, does this mean that in 1.4+, this approach to collaboration will no longer work? Is there a way to recover the old behavior? Being able to collaborate this way has been the primary benefit of cloud-based versioning for us.

@emmamendelsohn

[wlandau]

I did not realize that the versioning system in targets <= 1.3.2 would be used to implement a shared-datastore scenario like the one you described. Your team's original strategy is definitely a clever and compelling way to collaborate, but it is definitely "off-label" with respect to the intended scope of targets, and unfortunately it no longer seems feasible. Off the top of my head, I am not sure if there is a workaround that would accomplish the same thing as seamlessly with targets >= 1.4.0. And unfortunately it is not feasible to to support both the old and the new behavior at an infrastructure level.

For context, the primary reason I made the switch was to eliminate the egregious performance penalties from calling the AWS API in large pipelines. In targets <= 1.3.2, to check If the pipeline is up to date, tar_make() and tar_outdated() needed a separate HEAD request for each individual target. Because of the huge overhead (to say nothing of cost), S3 storage was unusable at the level of scale for which targets was intended. (Except if users manually set cue = tar_cue(file = FALSE), which would detract from the integrity of the pipeline if used on a regular basis.)

I solved this problem by caching the results of a paginated LIST request, which reduced AWS HTTPS calls by 1000x and time spent by >60x. But unfortunately, ListObjectsV2 is not version-aware. The only alternative that might work is ListObjectVersions, which would download every single version of every single object in the pipeline, which would be an unmanageably large payload.

So as a side effect, targets is forced to check the current version instead of the version in the metadata to see if a target is up to date. And for the majority of use cases, I actually think this is a good at a conceptual level: everywhere else in targets, the target is not up to date unless the current version of the data agrees with the metadata. This has always been true for local data storage an un-versioned cloud storage, and now versioned cloud storage is philosophically aligned with that.

[snowpong]

everywhere else in targets, the target is not up to date unless the current version of the data agrees with the metadata. This has always been true for local data storage an un-versioned cloud storage, and now versioned cloud storage is philosophically aligned with that.

Agreed, metadata is king. But now in the case for repository_meta = "local" this seems turned on it's head. Now tar_outdated() will list targets as outdated if the version in my local metadata do not match the latest version in the bucket, even though the version required by my metadata is in the bucket. So now bucket really seems more like king.

[noamross]

Hmm, this is definitely going to require us to consider different strategies. I wonder if it is possible for us to implement a version where targets are named by hash in the S3 store, rather than by target name, and drop the versioning feature in the S3 bucket entirely. (This would have the added benefit of allowing the store to move between S3 providers, which is not possible for S3 object version history).

[wlandau]

I wonder if it is possible for us to implement a version where targets are named by hash in the S3 store, rather than by target name

This is how Rich Fitzjohn's storr package works. storr was the storage backend of drake (and was involved in remake before that). For targets, I chose the current (simpler) system because it seems to be more efficient in terms of speed and storage space, and it would not be feasible to change direction at this point.

Even if it were feasible, it would create a lot of "garbage" objects: old files whose names are hashes and which no longer belong to any current target. This would have the nice effect of supporting the kind of seamless collaborative versioning you are looking for, but it would also cause storage usage to balloon in many cases, both in gigabytes used and the number of files. The large number of files would put a strain on local file quotas and slow down the AWS S3 paginated LIST request that currently helps targets efficiently check that each target is up to date. There would need to be a manual garbage collection mechanism for files, which is an inefficient and temporary remediation strategy, far from ideal. I have been down that road with drake.

I describe all this not to argue, but because I think someone else might ask the same question, and I want to be able to read this comment and remember the specific reasons why I made these design choices.

I wonder if Docker-like layering would help your team. Docker uses a union file system to implement its layering capabilities (e.g. Union FS or Aufs, I think the choice is pluggable.) Not sure if I'm off base.

[noamross]

I'm not going to argue for you to change directions, but I must say this change surprised me because of how well suited the S3 versioning feature was for this workflow. I really thought it was one of the primary design goals for it. I don't know how widespread the user base is of this feature but I would be curious what their use cases are and if this will be an issue for others.

[wlandau]

I'm not going to argue for you to change directions,

If it seems like I am belaboring the point, it's because I think someone else is going to ask this same question several months from now, and I want to be able to look back on this discussion and remember why I did what I did.

but I must say this change surprised me because of how well suited the S3 versioning feature was for this workflow. I really thought it was one of the primary design goals for it.

I was going for a "time machine" effect so you could revert the metadata and read the matching output data, but I was not thinking about the possibilities for multiple people collaborating on a single data store.

I don't know how widespread the user base is of this feature but I would be curious what their use cases are and if this will be an issue for others.

At my workplace, our sys admin created a bucket for all of us, and we are only allowed to access data at a prefix specific to each user. So we would not be able to exploit the old targets hashing strategy for collaborative versioning anyway. I am not sure if other regulated corporate environments work the same way.

[wlandau]

The thing that would make it possible to roll back to the previous behavior is some kind of inexpensive way to check the ETags of multiple S3 object versions. So if I have a list of 3000 AWS S3 objects and a version ID for each object (whether current or old), I would need a way to check the ETag of each of those specific 3000 object versions, while only using 3 API calls and without running through any of the objects or versions outside the list. So far I don't think AWS can do this, except possibly in an S3 batch operation, which looks like it is high-overhead and requires an esoteric set of permissions. And I would also want consistent behavior with Google Cloud so targets would have some semblance of consistency. It's always tough to request a new feature in such a large commercial product as AWS or Google Cloud, but until then it seems like my hands are tied.

[noamross]

One possibility is to allow for the option of using [ListObjectVersions](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html), using the [version-id-marker](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html#API_ListObjectVersions_RequestSyntax) to filter to objects newer than the oldest object in the metafile. There would certainly still be some performance penalty for this, would it be possible to implement something like this but allow it as a configuration option? If so, would you accept a PR?

Our other current idea is to stop targets S3 versioning and and create our own solution of having our own proxy object types that refer to saved files on S3. This would have the benefit of allowing some other caching options, as well, but it would only make sense for specific high-cost targets. In this case we would lose general versioning capabilities for the rest of the pipeline that would allow easy time-machine-like comparison across versions with https://github.com/ecohealthalliance/relic/

[wlandau]

As you alluded to, version-id-marker is still likely to list potentially thousands of superfluous versions of objects, which would get expensive. And unfortunately maintaining two different versioning strategies does not seem feasible. It's a lot of tech debt for a situation like this, and I hesitate to expose a choice that most users would struggle to understand and use correctly. (Simplifying behaviors and tightening up guardrails are major things I take away from developing drake.)

[snowpong]

Is the problem with the idea of using ListObjectVersions with version-id-marker that you have no proper way to limit which objects you get back (like you only want a subset not all objects), or is the problem that you get more versions than you want? I would assume the bucket is only used for targets, so all objects should be valid to get? And with regards to unwanted versions - is not the point of versioning them that you want to keep them around?

I can see that the API is not ideal though :) I found this blog-post https://tech.trivago.com/post/2018-09-03-s3-versioning where they were grappling with enormous buckets and this API call.

Did any of you look at the possibility of using the version-id-marker as a sort of caching key. So for example, if one downloads all versions for all objects using ListObjectVersions and paging (until isTruncated is false), and then later comes back after something has changed, and uses the version-id-marker from last time, do you only get the new updates since last?

[noamross]

I was about to ask if, in the long-term, you would be open to custom repository back-ends other than "aws" and "gcp" that might allow for custom behaviors. However, it occurs to me that this might be entirely possible to implement using custom formats, where read and write functions access a local or cloud key-value store and just save the keys to disk. I'll toy with this a bit, but I'm sure it is also an "off-label" use case so I'll check before trying anything serious with it.

[noamross]

[Continues thinking out loud] On the other hand, a format-only approach to this would mean that targets would be unaware of the storage/caching mechanism, so all the status and monitoring info would be wrong, and targets would error if the cache were cleared without clearing the key files. I can think of some solutions (e.g. a set-up/sync function in the targets script), though they may be pretty hacky. To be continued...🤔

[snowpong]

What did you end up doing to resolve it at your end, downgrade to before 1.4.0?

[noamross]

We're in-progress with this. Some projects are sticking with pre-1.4.0 versions (we run everything with fixed versions with renv, so this isn't too hard), others are moving to run locally where versioning isn't important, or just uploading objects outside of the targets mechanism to save objects at set tags. The key-value approach above is promising but has to wait until we have a the free time to implement it.

[snowpong]

We're also using S3 versioning with a shared bucket for multiple users on a project, so this is an interesting discussion to me. Our server is running locally (minio) and we're also doing repository_meta = "local". I'm trying to figure out what broke for you @noamross and what you fixed @wlandau - and if this change affects us as well.

As I can understand the change was done to speed up working with AWS. But the side-effect is that now targets determines if a pipeline is up-to-date differently? Before it was considering local metadata vs AWS store, and now it ignores local metadata?

[snowpong]

Reading the description again it says "For a cloud target to be truly up to date, the hash in the metadata now needs to match the current object in the bucket". Does that mean that user A could be running tar_outdated() and all is OK, go for lunch, then come back and run tar_outdated() and his pipeline is out of date, because user B has updated an object in the mean time? If so then this broke our workflow as well.

[noamross]

Yes, that's the case. repository_meta = "local" was necessary starting from 1.3, I think. From 1.4.0 you can't share the object store.

[snowpong]

Ouch😬. Thanks for verifying. I guess we'll have to downgrade.

[wlandau]

I wonder if it is possible for us to implement a version where targets are named by hash in the S3 store, rather than by target name

Looking back on this now. The approach you describe is apparently called "content-addressable storage" (CAS). Like I mentioned, storr always does this on purpose, and targets < 1.4.0 does it by accident with version-enabled S3 buckets.

It would be better to be able to opt into CAS regardless of the other storage settings (e.g. AWS vs local). I am still not sure this is feasible for targets, but today I had the idea that maybe there could be a separate _targets/cas/ folder for these objects that would exist alongside the name-based _targets/objects/, so the transition from one to the other could be smooth and non-breaking. I will take a closer look at what can be done natively.

[wlandau]

Maybe this could come in the form of tar_option_set(format = sync_pointer_files_and_define_custom_format()), where this new function only syncs pointer files once per R session and only runs inside a call to tar_make(). (Or maybe customize this with a function argument.)

Better yet, {targets} could implement something like a run_on_tar_make_only(r_expression) function, and the a third-party CAS could implement a wrapper that calls both this and tar_option_set(format = …) in _targets.R. Please let me know if that would be useful.

[noamross]

OK! This is still a ways off but I think it makes sense to implement to move ahead this way. There are some more specifics about things l'll follow up about for the proposal.

[wlandau]

Sounds good, happy to discuss details.

By the way, the initial sync step can already be done with something like this at the beginning of _targets.R:

if (grepl("^tar_make", targets::tar_runtime_object()$fun)) { 
  # only runs the on tar_make(), tar_make_clustermq(), or tar_make_future():
  synchronize_pointer_files()
}

list(targets::tar_target(...), ...)

You could easily encapsulate the if() statement in a wrapper in the third-party CAS system. I think the need for this is not common enough to write a simple/general wrapper directly in targets.

[snowpong]

We are very much interested in a CAS approach for targets with either a locally or remote/cloud solution. DVC is already built this way https://dvc.org/doc/user-guide/project-structure/internal-files#structure-of-the-cache-directory (works both GC / AWS and local shared directory as the remote). I am following this thread with interest 👍

[wlandau]

You could easily encapsulate the if() statement in a wrapper in the third-party CAS system.

Actually, a better way to do this is with tar_active(): https://docs.ropensci.org/targets/reference/tar_active.html

if (tar_active()) { 
  # only runs the on tar_make(), tar_make_clustermq(), or tar_make_future():
  synchronize_pointer_files()
}

list(targets::tar_target(...), ...)