Starting jobs with manual parameters

[weaversam8]

@taylorsilva suggested I post some ideas here we chatted briefly about regarding how Concourse fits together with manual workflows.

Background

I love Concourse, and I think it's fan-in fan-out functionality is really powerful, but resource types as a way to trigger jobs is primarily focused on use-cases that can be fully automated. Things like:

• Triggering a job on every commit
• Deploying code when a new Docker image is pushed
• Sending a message when a new object is uploaded to an S3 bucket

This fits well for CI/CD environments that have been fully automated, but interacting with these systems manually can be hard. In Concourse today, how would someone:

1. Deploy a specific version of an application on-demand?
2. Run only 1 test to validate a bug with an upstream API?
3. Gate a release until several contributors have approved it?

There are some existing approaches to solving these problems in Concourse, but they're not necessarily easy to use. Some examples:

• gate-resource allows you to gate a release but interacting with that gate requires committing files to a Git repository
• Jobs can be manually triggered from the UI, CLI, or API, but that is a single "start" signal. You can't, for example, say "deploy our application with arguments version:1.2.2 and environment:qa

Proposed solutions

So, what's the best way to solve this problem? Some ideas we discussed:

1. A manual "parameters" resource type

The idea here would be that the resource type would be integrated into the Concourse UI. In a pipeline view, a user could click on that resource, provide some information, and that information would immediately emit a new "version" of that resource type.

Coupling resource types so strongly to Concourse like this hasn't really been done, it's not clear how this would work architecturally.

2. Extending jobs themselves to accept a list of manual "parameters"

Jobs can already be "triggered" thru the UI/CLI/API (as described above) so we could extend that mechanism to support requiring additional parameters when triggering a job. This would probably be implemented by a new step type, maybe ask or prompt? This would also require all get steps within the job to have a trigger property to be set to false.

---

I'm open to other ideas, what thoughts do y'all have?

[Mo0rBy]

This please!
This is one of the things that is preventing me from moving to Concourse from Jenkins.

The ability to manually select certain parameters for the pipeline is a must for me and my team. The most basic form of options for us are something like a repository branch or an environment name (we have multiple development environments), allowing us to build and deploy feature branches whenever and wherever we want to.

[taylorsilva]

I'm not keen on a built-in resource-type thing. It would likely become a weird thing to maintain in the codebase over a longer period of time since it probably won't follow our current resource interface/workflow. You note a similar concern in the last line of that section.

The second proposal is interesting though, extending jobs.

First thought I had is that we could make it easier to trigger a job with a certain set of inputs. Right now if you want to trigger a job with certain versions for each resource, you need to go into each resource, click into its version history page and pin the version you want to use. Could we consolidate that workflow to make it easier to pin/select versions?

I'm imagining adding another button (or dropdown, don't fuss over UI right now) here on the job builds page:

Where it takes you to a page with a list of all resources that are inputs for that job. Each resource can have a field that defaults to the latest version (or what's currently pinned) and have a dropdown list of all prior versions.

---

Another thought I had as a middle ground is to maybe have a way to provide custom vars when triggering a job that would then be available to the job using the regular var's syntax (( )).
Could make it so the var names must be provided in the pipeline config first:

jobs:
- name: my-job
  vars:
    branch: main
    environment: dev01
  plan:
    - get: repo
    ...      

Then you can reference the branch and environment vars as usual: ((.:branch)) and ((.:environment)).
In the above example they have default values of main and dev01. When triggering in the web UI someone could then override those values if they want to.

---

Those are my initial thoughts on your idea of manual parameters. Let me know what you think.

[taylorsilva]

you create a resource that always returned a list of versions, and never automatically triggered a job? Then, when a user wanted to trigger a job from the UI, they could select from that list of versions?

yes, that's exactly what I was thinking this could lead to!

That still wouldn't solve the problem if the user wanted to enter something completely manually...

That's what I was thinking the job vars would address. I'm trying to pick apart the problem here because I think there are some slight differences between the use-cases each of these features would address.

[weaversam8]

Yeah - the only thing is that it feels like both might be needed by the same job - so at the very least, the interface to launch a job might need to have "select a resource version" and "enter a job var" side by side.

[weaversam8]

Re: job vars - is there an existing ATC API endpoint for users to override job vars when launching a job? Or is that only present for fly execute?

[taylorsilva]

Yeah - the only thing is that it feels like both might be needed by the same job - so at the very least, the interface to launch a job might need to have "select a resource version" and "enter a job var" side by side.

Correct. It would depend how the pipeline and job are designed. I would personally try to minimize/eliminate the need to have any manual inputs to jobs, though I know a need to pop up here and there. If I saw a job with something like 10 vars I'd think that pipeline was poorly designed. Even the example of choosing which environment to deploy an app to, I would try and design those as two separate instances of the same pipeline. That's me though and I know others may want it in one pipeline for their own reasons, though it feels like a footgun to me.

My ideas are also motivated by trying to build on top of what we have right now.

Re: job vars - is there an existing ATC API endpoint for users to override job vars when launching a job? Or is that only present for fly execute?

No there isn't. You're also conflating what fly execute is doing. fly execute creates what's called a one-off build and also only allows you to run a single task step. Completely different pathway from what we're talking about here, but I do see why you tried to connect the two.

[bcersows]

Just to pitch in another use case where manual parameters would be nice.We've got a deploy-to-staging pipeline, which in most cases would always just deploy the main branch. For some testing purposes, a way to deploy some other branch is required, tho. Right now we support that by creating a temporary pipeline, basically a copy, that sets a different branch.
So the above suggestion to define parameters on a per step base would be great.

Probably more difficult to stitch together, but even closer to perfect: if the selection could be backed by a resource (so a dropdown of Git branches).

[weaversam8]

@taylorsilva starting a new thread for some concourse internals questions for you. I'm trying to put together a proof of concept PR based on the ideas you mentioned above, starting w/specifying vars at the job level in a pipeline config, like this:

jobs:
  - name: job
    vars:
      name: world
    plan:
      - task: simple-task
        config:
          platform: linux
          image_resource:
            type: registry-image
            source: { repository: busybox }
          run:
            path: echo
            args: ["Hello ((name))!"]

I've added a "vars" option to the JSON config for a job:

diff --git a/atc/job_config.go b/atc/job_config.go
index dc2ba2da0..4c118a110 100644
--- a/atc/job_config.go
+++ b/atc/job_config.go
@@ -5,6 +5,9 @@ type JobConfig struct {
        OldName string `json:"old_name,omitempty"`
        Public  bool   `json:"public,omitempty"`
 
+       // Vars to use to parameterize the job config.
+       Vars Params `json:"vars,omitempty"`
+
        DisableManualTrigger bool     `json:"disable_manual_trigger,omitempty"`
        Serial               bool     `json:"serial,omitempty"`
        Interruptible        bool     `json:"interruptible,omitempty"`

And now I'm trying to decide on how to get those down to the actual task step so they can be templated. I had two main ideas:

1. Pass the vars to the root DoStep config, and then add them to the Validator/DoPlan when they're visited by StepValidator or the planVisitor (is Planner the plan visitor?).
2. Add the vars to the validator in validateJobs and the planner wherever it's created (not sure about this one yet)

I'm somewhat new to Go, so I hope these questions aren't too detached from reality. I'm hoping to better grasp Concourse internals to be able to build at least a proof of concept version of this feature myself, but I know I'm definitely still building my understanding here. I am sure I'm misunderstanding some of how this works already.

[taylorsilva]

There are a lot of intermediary things to think about here. Adding Vars to JobConfig seems okay at a quick glance.

This struct is not what Concourse eventually uses to run the job though. This struct represents the YAML-facing configuration that end-users deal with. It gets saved in the database but is then pulled out and represented by other structs, usually in the atc/db package.

Something to keep in mind is that Concourse is made up of a lot of small micro-service-like processes. They all run in the one concourse binary, but their behaviour and architecture is a lot like a bunch of micro-services. These services communicate through the database.

So this JobConfig will get saved in the database. You'll need to update the logic here:

concourse/atc/db/team.go

Lines 618 to 623 in 301f806

	func (t *team) SavePipeline(
	pipelineRef atc.PipelineRef,
	config atc.Config,
	from ConfigVersion,
	initiallyPaused bool,
	) (Pipeline, bool, error) {

Then what you'll probably want to update is the db.Build struct, specifically the Variables() it ends up returning so it includes the vars from the config:

concourse/atc/db/build.go

Lines 817 to 820 in 301f806

	// Variables creates variables for this build. If the build is a one-off build, it
	// just uses the global secrets manager. If it belongs to a pipeline, it combines
	// the global secrets manager with the pipeline's var_sources.
	func (b *build) Variables(logger lager.Logger, globalSecrets creds.Secrets, varSourcePool creds.VarSourcePool) (vars.Variables, error) {

And that's just looking at the simple path of someone triggering a job and using the default values for the vars. For manually triggering you'd need to update this endpoint potentially: https://github.com/concourse/concourse/blob/master/atc/api/jobserver/schedule_job.go
So it accepts user-defined values for the job's vars. If you follow the code path for job.RequestSchedule() you'll start to see what I mean about things being saved into the database by one thing and then some other service else where picks up that info.

If you're serious about wanting to give this a go, I think it would make sense to open an RFC about this and flesh out the user-facing interface and behaviour.