Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: var steps + local var sources #27

Merged
merged 2 commits into from Jun 24, 2020
Merged

Conversation

@vito
Copy link
Member

@vito vito commented May 21, 2019

Rendered

Related to, but not dependent on #24.

Signed-off-by: Alex Suraci <suraci.alex@gmail.com>
Signed-off-by: Alex Suraci <suraci.alex@gmail.com>
@vito vito changed the title RFC: trigger resources RFC: var steps + local var sources Jan 20, 2020
@vito vito marked this pull request as ready for review Jan 20, 2020
@evanchaoli evanchaoli mentioned this pull request Feb 3, 2020
@jgriff
Copy link

@jgriff jgriff commented May 12, 2020

Love This

This single feature has enabled us to streamline so many of our pipelines. 🙏
And in some cases overcome roadblocks for resources that didn't (easily) support file-based vars.

Improvement....Maybe?

I think one thing that could potentially make this better, at the risk of overcomplicating a simple file read, is support for jq-like extraction expressions (and others?) when loading the value.

For example, in our (GitLab) merge request pipelines we extract some meta info about the merge request (gitlab-merge-request-resource), which comes to us as a single merge-request.json buried in the .git folder of the merge request's branch that was fetched.

What I want to do is load the merge request number (.iid attribute) into a var.

Current Approach

To do that, I have to first jq extract the value to a file:

      - task: extract-merge-request-info
        config:
          platform: linux
          image_resource:
            type: registry-image
            source:
              # any image with 'jq' will do, our mr resource happens to have it and will have already been pulled to the job container so just use it (for job speed)
              repository: samcontesse/gitlab-merge-request-resource
          inputs:
            - name: merge-request
          outputs:
            - name: merge-request-info
          run:
            path: sh
            args:
              - -exc
              - |
                # extract the merge request number ---> merge-request-info/number
                jq <merge-request/.git/merge-request.json '.iid' >> merge-request-info/number
                
                # ...other attributes we may want (individually)

now I can use load_var to get what I want:

      - load_var: merge-request-number
        file:     merge-request-info/number

I could of course clean this up a little and write it as an external task, but the indirection step is still there and is still more cruft than I'd like.

Potential Improvement

It would be smoother if I could give load_var some instruction via a filter on what to extract from the file, instead of loading the entire content.

      - load_var: merge-request-number
        file:     merge-request/.git/merge-request.json
        filter:   {jq: '.iid'}

Other filters could be given/supported, perhaps a grep regex (but only one acceptable).

Increasing Complexity

This would of course make load_var more complex than it is currently. However I can see this as a pretty powerful feature for load_var to support scenarios where you are loading variables from non-simple files.

@vito vito mentioned this pull request May 20, 2020
@vito
Copy link
Member Author

@vito vito commented May 20, 2020

@jgriff Cheers! Glad it's paying off already. :)

Hmm, can't you already load the JSON content and access the .iid field off the resulting var?:

- load_var: merge-request-number
  file:     merge-request/.git/merge-request.json
- put: foo
  params: {id: ((.:merge-request-number.iid))}

@jgriff
Copy link

@jgriff jgriff commented May 20, 2020

@vito ah I see, yes! Did not realize the var was navigable like that, very nice 👍

@sahil87
Copy link

@sahil87 sahil87 commented Jun 4, 2020

This is really great.

This allowed me to remove a custom task whose job was just to create a json file that could be loaded as docker build_args (containing json like { "BUILD_VERSION": "b-7a2abea" }). Now, this is replaced by a BUILD_VERSION var. I need this because my docker images depend on the build versions (b-<git-ref>) to create CDN paths containing built js files.

Earlier:

plan:
  - { get: repo_teaxrapi, trigger: true, params: { depth: 1 } } #git repo
  - task: task_get_build_properties #With the sole purpose of generating a json file
    ...
    outputs: [{name: build-props}]
  - put: drepo_teaxrapi #Using type: docker-image
    ...
    params:
      build_args_file: build-props/docker_build_args.json

Now:

plan:
  - { get: repo_docker-creds, trigger: true, params: { depth: 1 } }
  - { get: repo_teaxrapi, trigger: true, params: { depth: 1, short_ref_format: "b-%s" } }
  - { load_var: BUILD_VERSION, file: repo_teaxrapi/.git/short_ref }
  - task: build #Using type registry-image to push, oci-build-task to build
    ...
    params:
      BUILD_ARG_BUILD_VERSION: ((.:BUILD_VERSION))
      DOCKER_CONFIG: repo_docker-creds

All this resulted in builds being 30% faster now. The only side effect of this change was having to inject docker credentials for multi-stage builds as they are now handled by oci-build-task.

@vito vito merged commit 9132cf8 into concourse:master Jun 24, 2020
2 checks passed
@vito vito deleted the trigger-resources branch Jun 24, 2020
@xeivieni
Copy link

@xeivieni xeivieni commented Oct 9, 2020

Hi guys,

Great job on this feature, it really simplified our pipelines.

A small improvement I can see would be to add a parameter to allow displaying the content of the file (thus, the value of the var) in the UI. It could be necessary in some cases.

👏

@Bluesboy
Copy link

@Bluesboy Bluesboy commented Oct 26, 2020

Please do not refuse this feature, extremely helpful.

@holgerstolzenberg
Copy link

@holgerstolzenberg holgerstolzenberg commented Nov 19, 2020

I also have to agree that this RFC is very useful and helped us to achieve some things that would have otherwise be overly complicated. Please keep it.

@tjhiggins
Copy link

@tjhiggins tjhiggins commented Nov 19, 2020

Seconded. Dramatically cleaned up our pipeline.

@lukasmrtvy
Copy link

@lukasmrtvy lukasmrtvy commented Jan 10, 2021

Guys? What about set_var ? Would not be better to interduce some K/V resource based on https://github.com/dgraph-io/badger ?

@vito
Copy link
Member Author

@vito vito commented Jan 11, 2021

@lukasmrtvy That makes sense to me! Just need someone to write up an RFC showing how it'd work. I didn't have any immediate use cases in mind so I didn't include it, but it seems intuitive.

Thanks everyone for the feedback! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants