Filter build steps by branch

[sourcec0de]

Hey, everyone!
Great work on container builder. I'm loving the flexibility so far and the security model using KMS.
I'm currently working on a pipeline that will use a deis workflow container to deploy my app into GKE.

However, I don't want the deploy step to executed everytime a build is triggered. Only if I push to select branches. The plan is actually to use the $BRANCH var as the app name to deploy.

Branch filtering is a pretty common thing I've used in other CI systems like Codeship, Buildkite, etc.

I realize I could just create a trigger for each branch I want to deploy and use a different .yaml config but this seems a little redundant and wasteful.

It would be awesome if we could do something like this to only allow a step to execute when there is a branch match.

steps:
# build and push steps
- id: deploy
  name: 'gcr.io/$PROJECT_ID/deis-cli:latest'
  args: ['pull', 'gcr.io/$PROJECT_ID/app:$REVISION_ID', '-a', '$BRANCH']
  branches: ['*-staging', 'testing', 'master']
  env:
    - 'DEIS_PROFILE=/workspace/deis.json'

Is there anything like this on the horizon?

[bendory]

You can do this already. When you define your trigger, the "Branch" field is a regex. Just put in a pattern there that matches the branches you want to fire your trigger.

Note that Container Builder uses Go regular expressions. You can play with patterns in the Go Playground; a match on multiple branches is as simple as branch_match_1|branch_match_2 as seen in this sample code.

Also note that during trigger setup, the UI will helpfully tell you which branches your pattern currently matches.

[sourcec0de]

@bendory I did mention that I knew we could do this. I was more interested in finding out if we could submit feature requests here. I think it would be helpful to allow this in the cloudbuild.yaml instead of the triggers themselves.

Thank you for following up so quickly though. 👍

[skelterjohn]

This is an interesting idea - could have things like a step at the end that only ran on the master branch that published binaries to GCS (or pushed an image to GCR). Other branches, where people develop features or vet PRs, would skip this last step while still doing the rest of the build so that it's verified that artifacts are build properly and any unit tests pass.

I'll note that you can do this now by having your build step run a script that checks $BRANCH_NAME before continuing. I'd like to see how well that works in practice before we consider baking anything into the service config itself.

[bendory]

OIC, sorry @sourcec0de, I misread your initial query as thinking that you had to create a different trigger for every branch, so I responded with instructions to create a trigger that covers multiple branches.

But now I see that what you are really asking is how to use one configuration but have your build behave differently depending on what branch you are on. Thanks @skelterjohn for responding appropriately...

[sourcec0de]

@skelterjohn this is a very popular feature that Buildkite implemented based on requests on their slack channel a while back. I already went in the direction you mentioned above. It was the only way I could find to conditionally allow a build step. I think it would be nice to have a more declarative way to do this.

Again, thank you both for really fast response times. 👍
You're killing it!

[rimusz]

yes, this feature would make pipelines more powerful. and as long buildkite user really would love GCB to support it.

[sourcec0de]

@bendory it seems I'm not the only one interested in this.
Could we reopen the ticket and keep the conversation going?

@rimusz thanks for chiming in.

[christophersanson]

we're looking at implementing this in a more first class way (and will leave this issue open to update when supported). In the meantime though here's a workaround script example that will filter a build step by branch. https://bit.googleplex.com/#/jasmuth/5847651643817984

[sourcec0de]

@christophersanson that link is inaccessible to anyone outside of google.

[skelterjohn]

Contents of the link @christophersanson posted:

steps:
- name: 'gcr.io/cloud-builders/docker'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    echo "Here's a convenient pattern to use for embedding shell scripts in cloudbuild.yaml."
    echo "This step only pushes an image if this build was triggered by a push to master."
    [[ "$BRANCH_NAME" == "master" ]] && docker push gcr.io/$PROJECT_ID/image

[eezing]

For my case I wanted to deploy only master to my GKE cluster:

- name: 'gcr.io/cloud-builders/kubectl'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    [[ "$BRANCH_NAME" == "master" ]] && /builder/kubectl.bash set image deployment [deployment] [container]=gcr.io/$PROJECT_ID/[image]:[tag] || echo "skipping . . ."
  env:
  - 'CLOUDSDK_COMPUTE_ZONE=[zone]'
  - 'CLOUDSDK_CONTAINER_CLUSTER=[cluster]'

[andrew-waters]

The examples above are good workarounds. I'd like to see this as a native feature after this weeks GH integration.

There is a lot of value in adding these filters without having to resort to bash - not because it's complicated, but it's verbose and not so easy to follow just by sight.

My scenario is that I would like to build docker images with a latest tag on master and a SHA tag on branches to preserve commits to images at the expense of a denser registry. At the moment I actually build them both on trigger:

steps:
  - name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '-t', 'gcr.io/$PROJECT_ID/${_IMAGE_NAME}:latest',
      '-t', 'gcr.io/$PROJECT_ID/${_IMAGE_NAME}:$COMMIT_SHA',
      '.']

Whereas my ideal definition would be:

steps:
  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'gcr.io/$PROJECT_ID/${_IMAGE_NAME}:latest', '.']
    branches: ['master']
  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'gcr.io/$PROJECT_ID/${_IMAGE_NAME}:$COMMIT_SHA', '.']
    branches: ['*']

As mentioned, I know this can be achieved verbosely with bash but that doesn't seem like the right way this should be done. Also, because the trigger's can't be edited when using the new GH marketplace app I think that this is going to be a blocking issue for people onboarding.

[MichielDeMey]

Came here looking for the exact same solution to @andrew-waters issue.
With the new Google Cloud Build integration, there's currently no way to limit build steps to certain branches other than using bash scripts as mentioned above.

We're currently also looking at using https://github.com/keel-hq/keel for CD to our Kubernetes cluster.
As it currently stands, Keel requires semver versioning on your Docker images for deploying new k8s deployments/charts (which is the way to do it IMO).

However, I fear that when I use the $TAG_NAME variable substitution we might up automatically building a Docker image with a $TAG_NAME from a branch other than master and overwrite existing Docker images.

[ardagnir]

Configurable GitHub app triggers are coming. (demo here: https://www.youtube.com/watch?v=iyGHW4UQ_Ts)

Once this happens, you should be able to configure app triggers to run only on specific branches similarly to the non-app triggers.

[kubaj]

I think that override of an entrypoint isn't solution. Cloudbuilders have the entrypoint that setups workspace in some cases (for example, Go cloudbuilder creates symlink to the workspace in the gopath). If you override the entrypoint by providing arguments, you have to run the initializing script yourself. In other case, it will break the functionality of cloudbuilder.

I was trying Google Cloud Build integration on the Github and I didn't find the way to edit the triggers.

[sourcec0de]

I think we can finally close this 1 year later.
Cloud build can now filter triggers on tags, pr's and file globs.
There are other ways to think about triggering build steps. 😉

Thanks google cloud build team 👍

[fcollman]

can someone point to the documentation for this?

[prescottprue]

In my case I can not use the custom github triggers since they do not seem to support other build steps. My current setup needs other CloudBuild steps to happen such as decrypting files using kms decrypt, so it would be great to have the branches option shown by @andrew-waters.

Two workarounds I have used:

• Using entrypoint: 'bash' with custom bash logic (the response from @skelterjohn)
• Pointing the separate branch triggers to different cloudbuild.yaml files (both "branch regex" and "cloudbuild yaml file" are options in Build Triggers)

[NeoJRotary]

I make a docker image https://github.com/NeoJRotary/GCB-bridge with private Github App to temporarily solve this issue. Still hope GCB team can support those amazing features. It would be great to have integration with rich interface at GCP console.

[leo-baltus]

Please reopen this issue. Op seems happy but when using the GH app we are unable to do branch filtering, react differently to PR's, use include/exclude paths for mono-repos etc.

[gsf]

I could see a solution to this use case and others by allowing something like argo's conditionals.

[DiederikvandenB]

Would love to see this resolved soon.

[NinoFloris]

/cc @christophersanson

[bondarewicz]

Please reopen

Configurable GitHub app triggers are coming

does anyone knows any eta? this appears to be the only? option currently to notify GH about build status but we are missing trigger configuration

Cloud build can now filter triggers on tags, pr's and file globs"_

while we can configure trigger details, unfortunately they still do not notify github about build status...

[MPV]

Related to this, how do I specify "all other branches than master"?

[gsf]

@MPV You mean other than [ "$BRANCH_NAME" != "master" ] && ... in a build script?

[bondarewicz]

Related to this, how do I specify "all other branches than master"?

in bash [ "$BRANCH_NAME" != "master" ] as per @gsf would work just fine
in the trigger conf [^(?!.*master)].*

[humanchimp]

Character classes won't work. Your regular expression test fails for other inputs besides "master". For instance "msstteerraasttrrss" Lookaround is not supported by golang, so I don't know how to write a regexp for this. A checkbox to negate the entire pattern would work, however.

I cast my vote (if i'm entitled to one) in favor the "branches" feature of each step in the cloudbuild.yaml mentioned above. The seemingly common use-case which seems too onerous at the moment is wanting continuous delivery for only the master branch.

For now, I'm going to go the custom docker container route and use bash to perform the branch test.

[zargex]

@sourcec0de could you give us the link to those docs. Because here https://cloud.google.com/cloud-build/docs/running-builds/automate-builds, they said the build won't be lunch by Pull Request, they are launch by commits pushed to the origin.

We are using the Google Cloud Build app for Github and I only see we can change the repository. The example here https://cloud.google.com/cloud-build/docs/run-builds-on-github create a new branch and the pull request, but I think the trigger here is still the push to a new branch.

We are interested on running our Cloud Build pipeline only when a PR is created and having branch conditionals (for example, deploy to staging if branch is develop). For the branch conditional currently we are using the bash workaround as posted above.

Thanks.

[ilkkao]

I think #138 (comment) is still valid. It's error prone and not clean to override entry points for the pre-built cloud builder images.

[DiederikvandenB]

It is looking like this issue won't be reopened, while it is still not possible to filter branches when triggering Cloud Build through GitHub. Should we open a new issue to address this?

[bondarewicz]

@DiederikvandenB please do, we might get some answers

[DiederikvandenB]

@bondarewicz done:
#431

[tino]

@sourcec0de, @bendory or @skelterjohn, please reopen this issue. It's the most commented on and followed issue in this repo.

The feature requested by this issue – to filter build steps by branch – is still a very valuable feature that all comparable build tools have. Yes there are workarounds, but they pretty much destroy the simple and elegant solution that a branches filter on a step would be.

(#431 is about the complete filtering in triggers as I read it, not about individual steps.)

[kubaj]

In addition to entrypoints workarounds, cloud builder have to pull steps' images everytime even if step runs only the bash condition and exits. Because of this, builds take much more time if you use a lot of custom steps which are not cached.

[msm]

Completely agree with @tino and @kubaj brings up a good point as well. Please consider re-opening and enabling such a feature.

Personally, I would much rather have the filtering happen in the cloudbuild.yaml file in source control, rather than trying to work around it by configuring the github application (not in source control).

[imjasonh]

This bug has a lot of great insights, and a lot of useful feedback.

However, in an attempt to keep this repo on-topic (the topic being the builder images in the repo itself) and to improve repo communication (#443), we are asking people to file feedback about the GCB service itself as described here: https://cloud.google.com/cloud-build/docs/getting-support

Creating issues in the public issue tracker lets us dedupe them to focus responses and reduce noise, and lets us assign engineers and track progress.

[gsf]

I've created https://issuetracker.google.com/issues/124468298.

[jasondamour]

FYI, it seems like google has responded to the issue above. They have began working on it

[max-sixty]

Does anyone have an idea why this doesn't work:

  args: 
  - '-c'
  - |
    [[ "$BRANCH_NAME" == "master" ]] && bash update_deployer.sh test $COMMIT_SHA

but this does (i.e. with the extra line):

  args: 
  - '-c'
  - |
    [[ "$BRANCH_NAME" == "master" ]] && bash update_deployer.sh test $COMMIT_SHA
    echo "Pushed to test"

I also tried a variant with the command on a single line, in single quotes, while failed

Thanks!

[icereval]

@max-sixty,

The exit code ends up being 1 when the condition is not matched, it might work if you add a return 0.

e.g.

  args: 
  - '-c'
  - |
    [[ "$BRANCH_NAME" == "master" ]] && bash update_deployer.sh test $COMMIT_SHA || return 0
    echo "Pushed to test"

[max-sixty]

Thank you @icereval !

[liath]

Adding return 0 gave me a return: can only `return' from a function or sourced script
true at the end is slightly uglier but seems to work:

- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    [[ "$BRANCH_NAME" == "master" ]] && echo pass || echo fail && true

[Shigawire]

@liath Don't use return 0 but exit 0