Overriding/default ((vars))

[allandegnan]

To save time, only really the ((vars)) matter in the below example, specifically the ((template_branch)), but I've added the rest for context.

So I'm creating a template repo for pipelines, which would look a bit like this:

pipeline-template/example/redacted.yml

resource_types:
- name: pull-request
  type: docker-image
  source:
    repository: teliaoss/github-pr-resource

resources:
- name: pull-request
  type: pull-request
  check_every: 1m
  source:
    repository: ((source_repo))

- name: pipeline-template
  type: git
  source:
    uri: https://github.com/my-pipeline-template
    branch: ((template_branch))

jobs:
- name: set-pipeline
  plan:
    - get: pull-request
      trigger: true
      version: every
    - get: pipeline-template
    - set_pipeline: ((pipeline_name))
      file: pipeline-template/example/redacted.yml
      var_files:
        - pull-request/ci/config.yml

Then I'm trying to consume this via another repo that has this file in it.

pull-request/ci/config.yml

---
pipeline_name: test
source_repo: allandegnan/allan-test
template_branch: ad/addingTerraformTemplate

The wider idea is I'm trying to make buildpacks but it's likely that the list of variables is going to grow pretty quickly, i.e. from having

• switchable tasks (i.e. I could have var to pick ECS, EKS, or Fargate for your docker deployment)
• the ability to turn steps on/off
• some vars for template development (I'm only including template_branch above so I can test patterns before I merge them into master, realistically users of the pattern shouldn't need that var).

Thus the question is, can I set a var that would be overwritten via var_files sections? Is there a hierarchy here? I feel like I've read this in the docs somewhere before but now I can't find it.

[vito]

Hmm, it looks like we don't explicitly document a precedence between vars and var_files, and that may be because we just haven't run into that question yet. Intuitively I would expect later-listed entries in var_files to take precedence over earlier-listed entries in var_files, but I'm not sure what I would expect between vars and var_files.

If we're to match the documented behavior of fly set-pipeline -l foo.yml -v foo=bar, we should have vars take precedence over var_files.

However, according to the current code...:

concourse/atc/exec/set_pipeline_step.go

Lines 179 to 196 in fab3de1

	staticVars := []vars.Variables{}
	if len(s.step.plan.Vars) > 0 {
	staticVars = append(staticVars, vars.StaticVariables(s.step.plan.Vars))
	}
	for _, lvf := range s.step.plan.VarFiles {
	bytes, err := s.fetchPipelineBits(lvf)
	if err != nil {
	return atc.Config{}, err
	}
	sv := vars.StaticVariables{}
	err = yaml.Unmarshal(bytes, &sv)
	if err != nil {
	return atc.Config{}, err
	}
	staticVars = append(staticVars, sv)
	}

It looks like we build up a list that looks like [vars, file1, file2] which I don't think can possibly satisfy both of these conditions at once, since we kind of expect some sort of order like "vars > file2 > file1".

Thankfully the step is still experimental, so I think we can change this behavior without too much trouble if we find that behavior to be desired. Would you agree that that precedence is what you're trying to achieve?

[allandegnan]

It didn't occur to me to just add another var_file, which basically meets my need regardless of order, so thanks for pointing that out.

I would argue that the order does seem slightly backwards:

        var_files:
          - pull-request/ci/config.yml
          - pipeline-template/template/terraform/defaults.yml 

I would have expected to define the defaults first, and would generally expect:

vars_file2 > vars_file1 > vars

So a var would be overwritten by my first defined var_file and my second defined var_file would overwrite both a defined var and the var_files before it, but I think you're right that being consistent is more important than the order.

For now, I'm personally unblocked though. :)

[vito]

Great! I've opened #5455 to track the change to make them consistent.