Feature: Access meta-data directly in YAML

[gtirloni]

Passing data through meta-data requires a separate shell script (to run buildkite-agent meta-data get, assign to variable and do something with it). In some simple pipelines, that's an added script that needs to be maintained.

It would be useful to have the ability to do that in the YAML file directly through some kind of BuildKite-specific syntax that understood it should reach out to the meta-data values and substitute the value (or set it).

[keithpitt]

@gtirloni howdy! I've been thinking a lot about what the next version of the Meta Data gear should look like, so I'm interested to learn more here.

I'm not 100% sure what you're asking for, would you be able to give me an example? Maybe some pseudo code of what you'd like to be able to do in the YAML file?

[gtirloni]

We have a pipeline that builds Docker images, tests and pushes them to the registry and then triggers another pipeline for deployment. The deployment pipeline receives hashes and other information as environment variable, which is fine, but then we have a block step that requires someone to confirm what they are deploying to production (and allows them to change it or if they trigger that pipeline individually, they enter it for the first time). The problem happens on the next step after the block, we need to pass that information to a single command but instead of a single line of YAML where we could interpolate those 2-3 vars, we need a shell script that will run buildkite-agent meta-data get, and then run the actual command.

Build/test pipeline has this step:

  - name: "Release"
    trigger: "docker-release"
    branches: "master"
    async: true
    build:
      env:
        DOCKER_IMAGE_NAME: "${DOCKER_IMAGE_NAME}"
        DOCKER_SOURCE_TAG: "${BUILDKITE_COMMIT}"

And the deployment pipeline is like this:

steps:
  - block: "Release!"
    fields:
      - text: "Docker Image Name"
        key: "image-name"
        default: "${DOCKER_IMAGE_NAME}"
        required: true

      - text: "Image Source Tag"
        key: "source-tag"
        default: "${DOCKER_SOURCE_TAG}"
        required: true

      - text: "Image Target Tag"
        key: "target-tag"
        default: "latest"
        required: true

  - name: ":rocket:"
    command:
      - "release"
    concurrency: 1
    concurrency_group: "docker-release"

That release shell script is something we need to maintain separately, which is fine, this is a simple pipeline but we

[gerrywastaken]

@keithpitt Came here to ask the same thing. We have a block step with a select for controlling where code is deployed. However we really want the selection to also govern the concurrency group that is used. This way we only force people to wait if there is already a deploy happening on the environment they selected.

...also, can you make a video ;)

edit: oh wow: https://buildkite.com/screencasts awesome!

[jmendiara]

Will #428 solve your Use Case?

[felixfbecker]

Have the same problem - somewhere in our pipeline we release a package with an increasing version number, which uploads the version number to meta data. Later we trigger a build in a few dependent repos which need to know that version number.
I would need something like this:

- trigger: "enterprise"
  label: ":satellite_antenna:"
  build:
    meta_data:
      oss_published_version: "$(buildkite-agent meta-data get published_version)"

Does anyone know a workaround? Is it possible to upload the trigger step dynamically at a later point with buildkite-agent pipeline upload?

[tekumara]

Similar to the above, being able to trigger another pipeline and extract meta-data into an environment variable would be really useful to me.

Something like

- trigger: "api-perf-test"
  label: ":tornado: perf test"
  env:
    ebenv: "$(buildkite-agent meta-data get ebenv)"

[toolmantim]

@felixfbecker
Is it possible to upload the trigger step dynamically at a later point with buildkite-agent pipeline upload?

Yep, that’s possible — you can do a buildkite-agent pipeline upload from any step of the pipeline, and interpolate in whatever values you need before uploading. You should be able to pass in the data you need to the triggered build in either meta-data or env then, depending on what you need.

It’d be nice if you could inherit meta-data to triggered builds though! Something like:

- trigger: "enterprise"
  label: ":satellite_antenna:"
  build:
    meta_data:
      - oss_published_version

[0ndermijner]

Is this still being considered / added in a feature? Would be really nice :)