Proposal: resources can declare a time-to-live if a certain version becomes "stale" but re-running in/get would refresh the cache

[cjcjameson]

We suggest a new field in resource declaration, cache_refresh_interval:

- name: release
  type: s3
  cache_refresh_interval: <time>
  source:
    bucket: releases
    private: true
    regexp: directory_on_s3/release-(.*).tgz
    access_key_id: ACCESS-KEY
    secret_access_key: SECRET

The current behavior of baggage claim is to keep the latest version indefinitely on a worker. This value would be set on baggage claim to provide a general, back-up cache invalidation window.

Our situation that prompts this:
The S3 Resource has a private: true configuration. This means that when in runs, it generates a signed URL. The signed-ness of the URL is only valid for a certain window (currently 86400 seconds / 1 day in s3-resource; AWS caps the duration at 1 week).

Let's say there's a daily build. But the build's unit-tests have been failing for a few days, so it's been a while since we last published to s3. Then someone triggers a downstream build, which fetches the latest s3 resource, not to use the file itself, but to use the signed url. At that point, the signed URL will have expired if the downstream job's tasks get assigned to the worker that originally ran the in on the first pass of the pipeline for this version of the resource. On the other hand, if the task is run on a worker whose baggage claim has not yet ined this version of the resource, then it will generate a new signed URL at that point, and have a fresh 24 hours on the clock to use it.

We have considered other options that are S3-specific -- should we think harder there?

• just use a longer TTL on S3 Signed URLs: if the artifact is a week old, maybe no one will want to use it anyway.
• Reconfigure how the S3 resource provides you with a signed URL: make that part of the put action, to ensure that it gets run every time
• Don't use the private: true feature; calculate a signed S3 URL ourselves in the body of our task
• Have our task have some retry logic if the URL has expired; punt to manual calculation
• Somehow contort the versioning semantic of S3 resources to include the signature / authentication (!!NO DEAR GOD!!)
• Write a separate resource that fetches signed S3 urls, and that's all it does. It would add weight to pipelines, but it's strange that the S3 resource in serves up some content (a special URL) that's not under the umbrella of the versioning semantics.

Other options for our particular case:

• Provide additional alerting in the body of our task's STDOUT to help people for the occasional times when this might come up.
• Make sure that our iterations get faster so that this doesn't come up!

Alternative implementation suggestion: add a property to baggageclaim itself, which provides a global cache-invalidation time-to-live. Pretty drastic, but maybe it's reasonable?

Downside of this feature: abuse by people who are scared of stale caches, when actually the versioning semantic of their resource is strong & complete such that this wouldn't be necessary.

Open question: how does this apply to other resource types? Would it be generally applicable?

Sometimes resources are perfectly repeatable and strongly versioned. For everything else, there's occasional cache invalidation. 😃

[vito]

Hmm. Having to set cache expiration on the resource feels like a leaky abstraction. Ideally there should be no configuration required for caching to work.

get calls are meant to be deterministic. Creating a signed URL involves a side-effect and is nondeterministic. So it shouldn't be done with get. I also don't generally like the idea of resources controlling their own caching. Resources should all fit the same mold, for more reasons than just caching.

How about something like:

- get: my-bucket-thing
- put: my-bucket-thing
  params: {sign: my-bucket-thing, expires_in: 24h}

The put would yield a version like {path: foo, expires_at: <timestamp>} (or version_id if using versioned_file). The get resulting from the put would generate the signed URL.

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #129750293 Proposal: resources can declare a time-to-live if a certain version becomes "stale" but re-running in/get would refresh the cache

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[cjcjameson]

Yep, that makes sense to a 80th percentile Concourse user and above, and is probably the best possible use of the current abstractions. I've had a lot of difficulty explaining to newcomers why "get-put-task" is a thing with the Pool resource, so I'm hesitant because it affects the UX.

No-action is OK; the most useful thing for us to work on might just be parameterizing the timeout value on the S3 resource so that you can configure how long the signed URL stays for.

[vito]

Closing this in favor of concourse/s3-resource#53