Resource Groups

[cppforlife]

Problem

BOSH core team runs a pipeline to produce a bosh release and bosh cli gem combination. It also tests them against a specific version of bats git repo.

BOSH cpi team runs a pipeline that expects to consume bosh release, cli gem and bats repo combination so that it can test it all against a specific version of cpi release. Ultimately this pipeline publishes a new cpi release referencing other asset versions.

We would like to capture group of resource versions instead of just individual resource versions, so that cpi team's pipeline can run against a specific set of resources that passed core team's pipeline.

Other examples:

• concourse release + garden release combination
• cf-release + diego release + cats src combination
• pieces of cf-release

Proposal

I think concourse should introduce a new primitive for specifying resource groups. Resource groups would be versioned similarly to any other resource. Builds would be able to produce a resource group version and consume specific resource group version.

In UI this could look like a box around multiple resources to indicate that they come as a whole.

  +--------------+
  | +----------+ |   +-------+
  | | bats-src |-|---|       |
  | +----------+ |   |       |
  | +----------+ |   |       |
  | | cli-ver  |-|---| tests |
  | +----------+ |   |       |
  | +----------+ |   |       |
  | | bosh-rel |-|---|       |
  | +----------+ |   +-------+
  +--------------+

Resources mentioned in a resource group are not any different from any other resources. They will be configured with all the other resources and have exactly same semantics.

Backing store for resource groups could work similarly to any resource's backing store. For example concourse release can include s3_resource_group that stores and retrieves files from a bucket just like s3_resource.

Contents of each resource group will only include versions for other resources so that storage/retrieval is delegated to each resource. For example contents of v1 of core-assets resource group will be:

candidate-core-assets-bucket/core-asset-1.json

(json in yaml)

bats-src:
  ref: 329ryfvbqeg83490tk549h39q40
cli-ver:
  version: 1.34.0
bosh-rel:
  version: 223

Possible configuration in the pipeline:

resources:
- name: bosh-init
  type: s3
  source:
    regexp: bosh-init-([0-9.]+)-linux-amd64
    bucket: {{init_bucket}}
- name: bats
  type: git
  source:
    uri: https://github.com/cloudfoundry/bosh-acceptance-tests.git
    branch: master
- name: bosh-release
  type: bosh-io-release
  source:
    repository: cloudfoundry/bosh

resource_groups:
- name: core-assets
  type: s3
  resources: [bosh-init, bats, bosh-release]
  source:
    regexp: core-assets-([0-9.]+)
    bucket: {{combos_buckets}}

I'll skip details of how concourse will be able to fetch specific versions of group's resources and its api format.

[cppforlife]

@vito @xoebus let's talk more about this some time today/tomorrow.

/cc @Amit-PivotalLabs @calebamiles @mariash

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #108822160 Resource groups

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[xoebus]

@dliebreich What do you think of this instead of Airfreight?

[dliebreich]

I'm not sure how the example yml and json describes the solution. Otherwise, conceptually, it seems like it would meet our needs.

One problem I can foresee is that you need to have the component resource configs match in the producer of the group and in the consumer of the group. If different teams maintained the producer and the consumer, that coordination might be somewhat expensive.

[amitkgupta]

For the purposes of integrating releases for Cloud Foundry (and this generalizes to any BOSH-deployed service composed of many releases), we require:

• some builds can produce an artifact recording the tuple of input releases that passed the build, and are therefore "blessed"
• some builds that can consume said artifact, and generate a BOSH deployment manifest consisting of all or some of the blessed releases
• end users (human operators) can also consume said artifact, and produce a manifest

We accomplish this like so:

https://github.com/cloudfoundry/cf-deployment/blob/master/blessed_versions.json

We maintain our own code for writing and reading this artifact because it needs to be used (at least read) outside the context of Concourse. We also specifically do not want to fetch the resources referenced in that blessed_versions.json file, because we use the data to construct a manifest where we can then delegate to the Director to fetch only the resources that it needs, and do it all Director-side to avoid unnecessary download-then-uploads on the client side.

[dliebreich]

+1 on not fetching all the resources in a group. We would group the ops manager images and stemcells for each of the 4 IaaSs with the .pivotal file, but we would only want the one stemcell and one ops manager for the IaaS we are testing in that pipeline.

[vito]

Here's a more fleshed out example:

resources:
- name: aws-stemcell
  type: bosh-io-stemcell
  source: {...}

- name: vsphere-stemcell
  type: bosh-io-stemcell
  source: {...}

- name: cf-release
  type: bosh-io-release
  source: {...}

- name: diego-release
  type: bosh-io-release
  source: {...}

resource_groups:
- name: diego-cf-compat
  type: s3
  source: {...}
  resources: [cf-release, diego-release]

- name: cf-stemcell-compat
  type: s3
  source: {...}
  resources: [cf-release, vsphere-stemcell, aws-stemcell]

jobs:
- name: something-with-cf-and-diego-on-aws
  plan:
  - aggregate:
    - get: cf-release
      groups: [diego-cf-compat, cf-stemcell-compat]
    - get: diego-release
      groups: [diego-cf-compat]
    - get: aws-stemcell
      groups: [cf-stemcell-compat]

So, you could pick and choose what parts of the tuple you actually want, and you're still dealing with regular old Concourse resources, rather than passing around giant "union tarballs" or something. If you wanted to just produce URLs for the director to upload that would be done on a resource-by-resource basis, i.e. a param telling the resource to just give you the URL and not download it (which a few already support).

Notes:

• resource_groups configure where to find the "tuples" and what resources they map to. (May need a map here instead of a list, not sure what the actual contents of the resource would look like.)
• You'll notice strong parallels between groups and passed on the get steps.
• The resources themselves are configured and used as normal, making this a drop-in feature so you can reuse your tasks and much of your pipeline config.

[Amit-PivotalLabs]

For our use case, not sure how much this buys us. We need to extract the
URLs or whatever metadata is contained in the S3 object backing the
resource_group, and use them to construct a manifest, outside the context
of concourse. And we would want to exercise this logic in our builds as
well, so we wouldn't want one way of extracting the data in Concourse and a
different way of extracting the data for end users.

On Thu, Nov 26, 2015 at 9:41 AM, Alex Suraci notifications@github.com
wrote:

Here's a more fleshed out example:

resources:- name: aws-stemcell
type: bosh-io-stemcell
source: {...}

• name: vsphere-stemcell
  type: bosh-io-stemcell
  source: {...}
• name: cf-release
  type: bosh-io-release
  source: {...}
• name: diego-release
  type: bosh-io-release
  source: {...}
  resource_groups:- name: diego-cf-compat
  type: s3
  source: {...}
  resources: [cf-release, diego-release]
• name: cf-stemcell-compat
  type: s3
  source: {...}
  resources: [cf-release, vsphere-stemcell, aws-stemcell]
  jobs:- name: something-with-cf-and-diego-on-aws
  plan:
  • aggregate:
    • get: cf-release
      groups: [diego-cf-compat, cf-stemcell-compat]
    • get: diego-release
      groups: [diego-cf-compat]
    • get: aws-stemcell
      groups: [cf-stemcell-compat]

So, you could pick and choose what parts of the tuple you actually want,
and you're still dealing with regular old Concourse resources, rather than
passing around giant "union tarballs" or something. If you wanted to just
produce URLs for the director to upload that would be done on a
resource-by-resource basis, i.e. a param telling the resource to just give
you the URL and not download it (which a few already support).

Notes:

• resource_groups configure where to find the "tuples" and what
  resources they map to. (May need a map here instead of a list, not sure
  what the actual contents of the resource would look like.)
• You'll notice strong parallels between groups and passed on the get
  steps.
• The resources themselves are configured and used as normal, making
  this a drop-in feature so you can reuse your tasks and much of your
  pipeline config.

—
Reply to this email directly or view it on GitHub
#210 (comment)
.

[xoebus]

Let's just store the information for this in Git. We're familiar with it now and it's much easier for a human to look at diffs and the GitHub web interface than S3. I'm liking the Alex's syntax proposal above.

@Amit-PivotalLabs I don't understand. Can you give an example of your workflow?

[xoebus]

@vito How would you put: to a resource group with your syntax?

[Amit-PivotalLabs]

@xoebus

• Build consumes cf-release, etcd-release and aws-stemcell resource
• Tests pass, then a "blessed-versions" file is checked into a git repo recording this group of resources passed together. The data included for each resource includes a version, URL, and SHA.
• Tool consumes "blessed-versions" data to produce a BOSH manifest stub saying which versions of releases, and which stemcell, to use. It heavily leverages the new feature in BOSH where you can directly specify URLs for releases and stemcells in the manifest, and delegate to the CLI and director to make sure the right releases and stemcells end up on the director (no explicit bosh upload release/stemcell).

The key point here is we don't need to get the resources, we sorta just need the data in the file backing the git repo backing the resource_group that you're proposing. The feature you're describing does the opposite, makes it possible to get a resource from a group (something we don't need), but not get the group metadata itself (all we need).

[vito]

@Amit-PivotalLabs

For the artifacts your team publishes at the end of the day, this won't be a silver bullet. For determining that set of outputs in the first place though, it could make your lives a bit easier. For example, if each team consistently reported the tuple of versions that worked together, mega's CI could just operate on the intersections for whatever deployment configurations you're concerned with. Disclaimer: no idea how you're determining these tuples today. If it's based on diego-cf-compatibility I'd expect that to be replaced with this resource groups feature instead. Not sure if other compatibility matrices currently exist.

That being said, it'd be pretty easy to write the function from the individual resources to your own format. Which seems like a useful thing generically. The point behind not having the tuple's format be the focal point is to have the pipeline usage just mirror the semantics of passed constraints. If we were passing around resource groups as their own objects you may end up with some tasks that deal with groups vs. some that deal with individual inputs; I'd rather have everything be the latter.

If all you need is metadata, it's up to the resources to support that, e.g. the S3 resource will give you /url and /version. I'd rather not optimize for avoiding downloading since caching should help out, and some resources support that via params anyway (or could have issues opened to support it, as in S3's case).

[jchesterpivotal]

We discussed this problem on an unsuccessful hackday effort, calling it "correlated resources".

If you think of a Concourse pipeline as a function, right now it doesn't allow you to control all the parameters at once. You can vary each individually, once, but backing up to try permutations is difficult (does every or version pinning do this? I'm not sure)

We're interested in this problem because we want to develop an "approvals" resource -- a resource representing human approval of versions of other resources. So if I have a git SHA 1abc234, I might also want an approval for that SHA. The approval is distinct from the SHA. In fact we'd like it to be distinct from what the other resource type is -- approvals should be assignable to any other resource version. SHAs, Tracker stories, S3 documents, BOSH images. Anything.

To achieve this, there needs to be a way to tell Concourse to treat correlated resources as a single unit.

This also feeds into a larger discussion, which is inverting the UI emphasis from job to resource. The UI shows the state of the world as the integral of the stream of all resources coming into the system. However, what is interesting is the differential -- what the pipeline looks for this particular set of resources. This data is available but requires sleuthing to reconstruct.

[vito]

/cc @kimeberz for the differential UI - I've thought the same, and want to redo the resources page to be a lot more useful for data ingestion (including potentially having a page for viewing a particular version)

[vito]

think we're headed towards #684 as a general solution that would allow for this kind of workflow. closing this to consolidate.