[Feature Request] File or Path Filter | Sparse Git Checkout

[poida]

Hi,

I'd like to be able to only run a job in a pipeline if a particular file (or maybe matching path/glob) has changed.

I'd like to use this to only run our docker build and publish step if the Dockerfile has changed.

E.g.

steps:
  - label: ':hammer: build docker'
    command: ./scripts/build_publish_docker.sh
    branches: 'master'
    path: Dockerfile
    agents:
      queue: dev

 - wait

 - label: ':tornado: run tests'
 ...

Thanks!

[poida]

Maybe if it's easier what about support for sparse checkouts - only work in a specific subfolder?

http://stackoverflow.com/questions/600079/how-do-i-clone-a-subdirectory-only-of-a-git-repository

[njdnet101]

Is this two separate things?

1. Ability to trigger a build based on changes to a specific directory path. Concourse and GoCD both seem to support this.
2. Ability to checkout only part of a tree

[poida]

@njdnet101 I think for my use case I need 1

[lankaapura]

Hi @sj26, is there any update on this particular feature? Thanks.

[sj26]

Hi folks,

Dynamic pipelines put all of this within your control! Start your build with a dynamic pipeline upload which uses something like git fetch origin master and git diff origin/master...HEAD --name-only to detect which files have changed, and use that to upload only the pipeline steps you want. We do this in some of our own pipelines, like for pipelines including expensive Packer builds so they only rebuild machine images when their templates have changed.

You could wrap this up into a plugin so that this behaviour can be shared across your pipelines, and even shared with the wider community. This is how we prefer to see these things happen — features like this emerge from amazing folks doing awesome things.

The only downside is that if you upload an empty pipeline this will still be a "passed" build, even if it does nothing. We might eventually add a buildkite-agent skip or something which lets you say "this build should be skipped, even though it's in progress" to round things out and prevent those builds from skewing your stats.

Cheers,
Sam

[nikskiz]

It's a shame this feature is getting pushed back. Other CI tools seem to support this feature.

[keithpitt]

@nikskiz you're totally right, it's something we gotta add! What makes it tricky is we don't have access to the source code, but maybe there's another way to implement it...

If anyone is still subscribed to this issue, I'd love to hear more about how you want this to work! Even some examples of how you've seen it implemented before and what you like/don't like about them. Maybe either post here, or email me what you want to see added to keith@buildkite.com

[EvilPuppetMaster]

I agree, this seems to me like a "table stakes" kind of feature for a build tool.

All you need is a list of paths in .gitignore style syntax that should cause a step to trigger, if they have changed since the last successful build of the current branch. Incidentally Github Actions does it exactly like this https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#filtering-for-specific-branches-tags-and-paths

We have had to implement the workaround mentioned by @sj26 of git diff against master, and it is NOT trivial. There are a lot of gotchas to worry about, for example, you can't simply git diff against master, since there could have been multiple revisions on master since the last time the build completed successfully. And then you have to make that work on branches as well that might build into different (dev) environments, so they have to track the revision of the last successful build for the branch. And the custom code you have to put in your pipeline to keep track of the changed files and then compare against them to decide whether to run your step can also get pretty gnarly.

I can confirm It is possible to do it that way, but it really seems like a basic feature that a CI tool should make simple.

[txbm]

Seems like given the fact that BuildKite already reacts to Github webhook event payloads to create work for the agents, it could use Github PR metadata here to perform some basic filtering on paths affected? Even if it's just basic prefix filtering.

Maybe I'm missing some obvious blocker as to why this is hard for BuildKite to support though.

[EvilPuppetMaster]

In addition to my previous comment, it occurs to me that the new depends_on option should also be considered. If a step depends on an earlier one that was triggered because of a change, then that step should also be triggered.

[alexandresalome]

The "something like git fetch origin master and git diff origin/master...HEAD --name-only to detect which files have changed, and use that to upload only the pipeline steps you want is very complicated.

Imagine I have 4-5 sub-projects, with each one having specific tests, I want for every PR to test only what has changed, compared to master.

With Github Actions, I do something like:

- name: Build PHP docker image
  paths: docker/php
- name: Build Node docker image
  paths: docker/node

On a professionnal side, we are using Buildkite. And I had to implement a script that detects it for 20 different sub-trees, so we developped a python script to make it configurable. Here is the headline:

# Sub-directory configuration
# DIRECTORY1,DIRECTORY2,DIRECTORY3:PIPELINE1:PIPELINE2:PIPELINE3:...
#
# When a change is detected in DIRECTORY, associated PIPELINEs will be triggered
# In order to exclude a directory, prefix it with "!"
#
# If a pipeline slug starts with "?", it allows failure (async = true).
#

If buildkite had paths feature, I could remove those 175 lines of python and would not have spent days developping something functional.

[teriu]

We ended up implementing our own matching functionality in https://github.com/vital-software/monofo. monofo supports splitting of pipeline.yml files into multiple components too.

[michaelbayday]

i'd like to reiterate that this functionality seems like it should be a core functionality of a CI system. I have used it with Github Actions and Gitlab CI and having to implement this on your own seems undesirable and less reliable

[hvh-mt]

Chiming in that this feature would be immensely helpful. Has there been any progress on this front?

[jmyounker]

Damn. I could really use this right about now.

[sherryxcheng]

Follow up. Would be great to be a build-in feature as available in other equivalent market tools.

[pleegor]

Follow up. Path specific triggers would be super helpful.