Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Determine full set of core resources that we should bundle with Concourse #3003

Closed
vito opened this issue Jan 7, 2019 · 12 comments · Fixed by #3183
Closed

Determine full set of core resources that we should bundle with Concourse #3003

vito opened this issue Jan 7, 2019 · 12 comments · Fixed by #3183

Comments

@vito
Copy link
Member

@vito vito commented Jan 7, 2019

Let's re-assess which resource types should come with Concourse in 5.0+ - there are a few that are very situational and can probably be omitted.

@vito vito added the enhancement label Jan 7, 2019
@vito vito added this to the v5.0.0 milestone Jan 7, 2019
@vito vito added this to Planned in One Big Repo via automation Jan 8, 2019
@xtremerui

This comment has been minimized.

Copy link
Contributor

@xtremerui xtremerui commented Jan 8, 2019

This is list of resource types that provided by Concourse on concourse-ci.org and my 2 cents for each of them. Discussion is welcome.

✓ git 
X  hg
✓ time
? s3 (how about gcs)
X archive
✓ semver
? github-release
✓ docker-image (will be replaced by registry-image)
X tracker
? pool (are there big number of users using it for non-pcf envs)
? cf (could be maintained by cloudfoundry team)
? bosh-io-release (maybe concourse pcf only)
X bosh-io-stemcell
@pn-santos

This comment has been minimized.

Copy link

@pn-santos pn-santos commented Jan 8, 2019

my 2c:

Generically speaking (dislaimer: I'm a minimalist), given how easy it is to add a resource that's not included, I think the resources included by default should be the ones that one would very likely need to do anything useful and all resource which are related to projects choices should be left as "add if needed".

For example I think semver should be left out. In cloud based CI/CD envs there's usually no need for semantic versioning, and semantic versioning itself is a choice that not every project goes for.

Same for s3, it's kinda situational, some projects will have it others won't.

pool I find very useful, especially in scenarios where each microservice has it's pipeline which includes end-to-end tests and you want to make sure no 2 pipelines run them at the same time (we use it in non-pcf envs). And it's the only way to have "resource locks" across pipelines.

Agree with all other . Everything else (all ? except pool) I think should be optional.

@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Jan 9, 2019

@pn-santos Agreed - I'd very much like to see Concourse ship with only very few "essential" resources and make resource_types: feel more "normal" and less like third-party/custom resources.

I think I agree with omitting semver, in part because we may end up reinventing the semver workflow once spaces lands, and it'll be easier to just introduce a new resource type.

I'm leaning towards omitting pool though since it actually is pretty situational in my experience. For example we don't make any use of it at all on our instance. It also feels a little awkward to use at times and may be in want of better Concourse primitives (i.e. some way to serialize pipeline segments), so I'm not sure about including it in core, even though it does have some perfectly fine use cases (e.g. for when you have separate Concourse clusters contending for a shared pool of resources).

So after all that we would end up with the following:

  • docker-image and/or registry-image (in the future?) - because without this there's no way to use resource_types:
  • time as timed triggers are pretty fundamental in other CI systems and it's cheap to include
  • git, honestly just for convenience because in all likelihood most installations will use git these days

Devil's advocate: what if we were to not include git (and maybe even time)? The obvious disadvantage would be that a ton of pipelines would now need to include it in resource_types:, but one major advantage is that they can pick and choose what version of the git resource they want to use, on a pipeline-to-pipeline basis, and roll ahead to new versions without relying on upgrading their cluster. This would make pipelines more self-contained in a multi-tenant environment. It would also very quickly introduce the concept of resource_types: to newcomers and emphasize the power of Concourse's "resource" abstractions.

That may be taking things a bit too far, but hey, it's an interesting thought experiment.

@pn-santos

This comment has been minimized.

Copy link

@pn-santos pn-santos commented Jan 9, 2019

I'm leaning towards omitting pool though since it actually is pretty situational in my experience.

Since it's so easy to add resource_types it's no big deal that it's not included (and I completely understand being situational).

what if we were to not include git (and maybe even time)?

I guess having a barebones install (with just docker-image/registry-image) is the more "consistent" option since, the truth of the matter is all other resources are "optional" in the sense there is always the possibility that they won't be needed.

And like you say @vito it does encourage the user to get to know/understand the resource_types and also allow to easily pin a particular version of it across concourse updates. Honestly I like this idea even though it would make anyone updating to v5 likely to have to add resource_types on all (or most) of their pipelines.

@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Jan 18, 2019

Just a FYI to all, at this point we're leaning towards only including registry-image and leaving the rest up to users to bring in via resource_types:.

@jchesterpivotal

This comment has been minimized.

Copy link
Contributor

@jchesterpivotal jchesterpivotal commented Jan 22, 2019

Just a FYI to all, at this point we're leaning towards only including registry-image and leaving the rest up to users to bring in via resource_types:.

I feel like that move should be held off until Duty Free is a thing. It's a disruptive change for anyone upgrading, so there'd need to be a carrot as well as a stick.

@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Jan 22, 2019

@jchesterpivotal There's already a carrot: your resource versions are no longer tied to Concourse versions, so you can get new features and bug fixes much more quickly. People ask all the time when a resource PR or bugfix will be available - now we can just ship it as soon as it's ready and stable.

I'd rather not boil the ocean and make this move dependent on Duty Free, but I think that we should make sure the common resource types are easy to discover and make sure all their READMEs are clear on how to install them just like all community resource types. We could think about starting a primitive form of Duty Free for this, but it might also just require an improvement to the docs.

@gdenn

This comment has been minimized.

Copy link

@gdenn gdenn commented Jan 24, 2019

i agree with @pn-santos keep it as minimalistic as possible.

I would suggest only git and docker-image since you require docker-image to include other resource types and git to get your scripts into the container.

Put a note in the breaking changes which resources are not contained anymore in comparison to the last major 4.x.x and i think everyone should be fine with that.

@vito vito removed this from Planned in One Big Repo Jan 26, 2019
@topherbullock

This comment has been minimized.

Copy link
Member

@topherbullock topherbullock commented Jan 27, 2019

We could think about starting a primitive form of Duty Free for this, but it might also just require an improvement to the docs.

I think there's a reasonable MVP somewhere in the description of what Duty Free is. But I agree, a primitive Duty Free (sounds fun... maybe some nice rocks in it?) doesn't need to happen before implementing this.

@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Jan 30, 2019

Hmm due to the user impact of this I think it might warrant an RFC, just to make sure we've done due diligence and made an effort to communicate this change as widely as possible. I'm a little worried about doing this with 5.0 and giving people yet another thing to be afraid of when upgrading and ending up with a long-term 4.x/5.x fragmentation.

I'm mainly worried that there may be people using Concourse and its core resources almost exclusively, and taking advantage of the fact that this setup works without reaching out to Docker Hub. Ripping them out in 5.0 without providing an easy and documented way to co-locate resources on a worker would be very disruptive.

Aside from that concern, one way to make the transition easier could be to have an operator-configured set of default resource types which are simply appended to each configured pipeline's resource types automatically. This way we could ease the transition by having Concourse still come with them as a default set of resource types. Users then wouldn't need to edit each and every pipeline. This is probably something to hash out in the RFC.

So all in all, I propose:

  • Keeping all the existing core resource types for now (for 5.0)
    • Even though we could probably remove some (like tracker), we might as well keep them all for now.
  • Starting on an RFC, likely proposing the removal of all but registry-image
    • This is where we can hash out things like operator-configured (team-configured?) default resource types
    • And perhaps be a catalyst for duty-free discussion
  • Doing the purge in 6.0 if the RFC goes well
    • We can also begin the transition to Ubuntu xenial images for everything (/cc @scottietremendous @osis) now that size of the Concourse distribution is less of a concern
@jchesterpivotal

This comment has been minimized.

Copy link
Contributor

@jchesterpivotal jchesterpivotal commented Jan 31, 2019

Aside from that concern, one way to make the transition easier could be to have an operator-configured set of default resource types which are simply appended to each configured pipeline's resource types automatically.

Another way would be a helper in fly that helps you to rewrite a pipeline. That could be introduced earlier than the change.

@christophermancini

This comment has been minimized.

Copy link
Contributor

@christophermancini christophermancini commented Feb 14, 2019

@vito I think that makes sense. IMHO a good approach would be to deprecate the feature in v5, making all of the embedded resources available on Docker Registry and upgraded / versioned in line with the Concourse release. Then in v6, remove them all together or offer two releases, one with and one without.

This would allow users to start upgrading their pipelines to the Docker registry variants, giving them time to upgrade all of them without breaking backwards compatibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.