Concourse should have a resource marketplace. It should be called "duty free"

[robdimsdale]

A place to discover and publish concourse resources, perhaps located at resources.concourse.ci

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #107090998 Concourse should have a resource marketplace. It should be called "duty free"

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[zachgersh]

This is a great name.

[jchesterpivotal]

I see this was tagged "who-cares" in Tracker.

Let the record show:

Me. I care.

[timlombardo]

+1

[vito]

@jchesterpivotal probably from when i bulk labelled all of rob's stories as "who-cares". fixed. ;)

I'd like to do this without needing people to sign up for an account, similar to how Elm's package management works.

• Entrants to the marketplace are just done via PR to a canonical repo. If we go with this, be careful about the structure; we don't want merge conflicts if two people submit independent resources. Maybe each resource gets its own subdirectory?
• We should think about explicit versioning of resources, and how people are expected to consume them in a stable way. Maybe the metadata to store in the subdirectory is a docker URI string pointing at a fixed digest for each version?
• Related, we should think about how our core resources fit into this scheme, and whether we should change our ways to provide for a better model for external contributors.
• As much information as possible should be drawn from the upstream repo. This way we're not constantly managing PRs to our central repo. (Think descriptions/README info/etc.)
• Building on the last point, perhaps having a formal structure of the README would make it easier to inline docs for each resource. We've already started to establish a structure (see https://github.com/concourse/time-resource and https://github.com/concourse/git-resource).

Open questions:

• This is pretty coupled to GitHub, but I think that's a reasonable tradeoff.
• How to manage versions being published? What kind of auditing should be in place?
• How easy would it be for a mischievous contributor to sneak a resource into someone's CI that does bad things with their credentials? Related to auditing, but also just how autonomous is this once a resource is registered with the marketplace, and what kind of information is exposed to individual resources. Thankfully, not much (only the resource's individual config).

[xoebus]

@vito To make sure that the contents of the resource image match what is claimed we need to see the source that generated the resource image. I wonder if there's some way that we could accept PRs of task.ymls and Dockerfiles to allow us to build their resource image.

[thewoolleyman]

+1

[thewoolleyman]

What about reusable Jobs, in addition to resources? We have implemented support for automatically building arbitrary branches, which is implemented as both a reusable resource and a reusable job:

https://github.com/pivotaltracker/git-branches-resource
https://github.com/pivotaltracker/concourse-branch-manager

[xoebus]

@thewoolleyman We've thought about that before and it's something that we would like to have (along with having a full search to find these things). Instead of sharing at the job level we would allow the sharing of reusable pipelines. Task parameters and the rudimentary templating in Fly allow for simple generic pipelines to be shared. We have a few generic pipelines of our own that we would like to share.

@vito I had a look at how the Elm packaging works. I like how lightweight it is. I think they can get away with such a small amount of infrastructure because you consume the source code of Elm libraries directly. This let's them just download the source code from the releases section once a commit is tagged. Unfortunately we need a Docker image and even though resource images can be small they're still 2 orders of magnitude too big to commit into Git.

As far as I can see there are two things we could do in order to have a repository of Docker images.

1. We store the registry, name, and digest which lets us fetch the exact image that the resource maintainer intended.
2. We store the Docker image itself. We could make an organization on DockerHub that we would hand out credentials for to resource creators. I think we can use the built in Teams and Organizations for DockerHub to allow this. Each resource owner would only be allowed to push to their own image. If we didn't want to give out credentials then we would need to build the images ourself. The resource maintainer would give us a task.yml for building their resource and we would then watch their repository for a new Git tag. When a new tag is published we would build their image and push it under the name of the tag.

I think I prefer option 1 as it doesn't tie us to Docker. If another image format takes over then all we need to do is add type: docker to the package files and then allow different types. Option 2 is really tightly bound to Docker.

Something else that I'm very worried about is image integrity. A popular resource is a great attack vector into a lot of CI systems. I'm struggling to come up with a good way to mitigate this. I think having the changes to the resource repository require signing with a GPG key might be a good way to go. RubyGems and Git support this so if we went with the Git backed model or a service backed model then we have an example for both. This does add another barrier of entry to publishing a resource but I think it's really important that people can trust these images.

[xoebus]

Also, 👍 on having the documentation be grabbed from a standard format in the README.

[zachgersh]

Really glad the discussion has picked up on this issue!

I think the metadata / install script hosting model has worked well for a
lot of package managers, didn't know elm was doing this but homebrew has
worked this way for a long time. It probably is the most viable solution -
as long as a decent specification is provided for what the metadata parts
look like.

But...advocating for the docker solution - wouldn't a private registry that
is just for concourse resources get you 90% of the way there? Auth would be
one of the questions you would need to answer but at least you can use the
images as they are currently shipped. As much as we "don't want to tie
ourselves to docker" it doesn't seem like it will be going anywhere for
awhile and we can host most of the tooling ourselves.

On Monday, December 21, 2015, Chris Brown notifications@github.com wrote:

Also, [image: 👍] on having the documentation be grabbed from a
standard format in the README.

—
Reply to this email directly or view it on GitHub
#191 (comment)
.

[xoebus]

I don't want to have to maintain and keep up a Docker registry that more than just the Concourse team uses.

[vito]

@xoebus I'm leaning towards option 1, with the addition that we should consume them as GItHub releases. That seems like it could be a good way to delegate all the release authoring and publishing to an external source (plus changelogs, etc).

The artifacts in the releases could either be tarballs or metadata pointing to Docker images.

I'm not sure how this affects the image signing thoughts, aside from wondering whether GitHub releases fall under enough security scrutiny for something as critical as these images.