RFC: set_pipeline step

[vito]

Rendered

Please comment on individual lines, not at the top-level.

[evanchaoli]

@vito Please consider to support pipeline self set. For example, for the set_pipeline step, if configure pipeline-name: self, where self is a reserved word, the it just do self set. With self set, pipeline name and team are known from the current pipeline.

I see you have create issue #4254 to track implementation of set_pipeline. Once this step is in place, it will simplify a lot of our use cases. I think set_pipeline step could be done on ATC, just call same function as current set-pipeline API, no need to start any container. If that's true and you are ok, I can contribute a PR, as we eager for this feature.

[git-committed]

We have multiple teams and 100s of pipelines coming from multiple git repos. We are addressing fly sp workflow for all changes to pipelines ymls with pipeline_resource and/or a custom built resource with fly binaries listening on these pipelines files/folders.
Even though we do not have strong preference for using either pipeline name or self as set_pipeline values( Good that we support both :) ), we are really excited for the set_pipeline plan. We also go through lot of compliance/audit related activities periodically and pipeline config changes being tracked in the same git repo as the pipeline definitions will help a lot.

[radoslav-tomov]

We have a relatively small set of pipelines, so both ways are ok for us. Honestly, I like the self build in reference it is clean, straightforward, and easy to deal with.

[osis]

Weird error...

unknown artifact source: '.' in file path './repo/ci/pipeline.yml'

changing the file path to repo/ci/pipeline.yml fixes it.

[TheDevMinerTV]

Works so far, although I would like to see the team name to be able to be changed. Right now I have four teams which all have their own auto-setting pipeline, which I would like to merge into one.

[DanielJonesEB]

Please comment on individual lines, not at the top-level.

Err, that guidance aside...

The old pipeline resource accepted globs in its vars_files elements. The new step doesn't. I'm not sure I can make a compelling case for accepting more generic input instead of specific input, but it is a bit more hassle if you've got a versioned file being used as a vars file. In our use case we have to use Minio rather than S3, so we have to have a version number in the file name of assets we track.

I thought I'd flag it in case this is an unintentional reduction in functionality.

[vito]

@DanielJonesEB Interesting, definitely not intended. Good call! That worked previously because the resource itself performed glob expansion against its local filesystem. For the set_pipeline step that will be a bit more challenging, as we don't have access to a local filesystem. Hmmm. 🤔 I'm going to record that as an open question.

@osis Ack, hmm, that's a silly bug, but funnily enough kind of related to the above problem too. 🤔 🤔 🤔 I wonder if we need some sort of pseudo-filesystem abstraction.

@TheDevMinerTV That's a tricky one. Configuring within the team is somewhat easy because we can assume someone set the configuring pipeline within that team and therefore they have authority to set other pipelines. But once you go cross-team, that same assumption is no longer valid - only if you're the main team, which feels like a pretty limited use case.

I think this touches on a larger goal of full-stack automation, which might involve things like a set_team step and perhaps even a set_project step (assuming #32) in the future. I think for now it's best that we leave that out of scope of this RFC, but it'd be a super interesting discussion to hold elsewhere. Until we hash that out I don't think we can/should have a set_pipeline step that can configure in other teams.

[TheDevMinerTV]

Sounds fair 😄

[xtremerui]

Coming from concourse/concourse-pipeline-resource#10, is set_pipeline step gonna support destroy or there will be a destroy_pipeline step in the roadmap?

[vito]

@xtremerui I don't think we'll need destroying with the advent of pipeline archiving, where you can just remove the step from you config.

[o-orand]

Hi !
I'd like to to know which part of 'concourse-pipeline-resource' dynamic configuration do you plan to implement ? It could be great to have all "keys" (var_files, team, config_file, vars, unpaused and exposed) except as you said pipelines.

[vito]

So far, vars and var_files are implemented, config_file is the same as file:, unpaused is just forced as true, team is experimentally supported (collecting feedback here: concourse/concourse#5731), and exposed has just been proposed (https://github.com/concourse/rfcs/discussions/75).

[o-orand]

Thanks for answering!
I should have miss something about dynamic configuration of set_pipeline. For me, dynamic config was like file for task (https://concourse-ci.org/config-basics.html#schema.file-path), or https://github.com/concourse/concourse-pipeline-resource#dynamic for a single pipeline. So, is it possible to generate set_pipeline config using previously executed task, for instance to select var_files to apply ?

[gid0]

Is it planned to implement a get_pipeline step in the future, to do what the concourse-pipeline-resource was doing as "in"?

[vito]

@gid0 There are no plans for this at the moment. If you've got a use case for this, feel free to open a new discussion!