Whitelist does not work if pipeline has an upstream pipeline as material

[rkrisztian]

Update: I had to correct my example, please see #50 (comment).

In a nutshell I have this config:

pipelines:
  p1:
    materials:
      my-repo:
        git: some-url
        auto_update: true
    stages:
      # (...)
  p2:
    materials:
      my-repo:
        git: some-url
        whitelist:
          - some-dir/**/*
        auto_update: true
      p1-as-upstream:
        pipeline: p1
        stage: some-stage

Now if I commit something outside of some-dir, I expect p2 not triggered, but it does, both pipelines are built. That sounds wrong. The whitelisting works fine if I remove p1-as-upstream from the p2 materials.

Do you know why it is not working? Is it a YAML config plugin bug or a GoCD bug?

[tomzo]

You'll have to be more specific about what are your expectations. Could you elaborate a bit?
But from what I see above, that is expected behavior, when you push to my-repo, it triggers p1 (because there is no whitelist there), and then p1 triggers p2 because it is a material of p2. This is justified by the logic that if project in p2 somehow depends on p1 then a new build in p1 could break something in p2, hence you should build p2 with new p1. Is that clear enough?

[rkrisztian]

You're right, my example is a bit off, so I'm going to provide a slightly different use case with more detail. I will need to test it in a short PoC though, to confirm that it's still a bug.

In my case my-repo is a repository containing 3 projects:

• pr1 as a library project (built by pipeline p1).
• pr2 and pr3 as end-user projects, both depending on pr1. (pr2 is built by p2, and pr3 is built by p3, configured similarly to p2. Both pipelines use the artifact generated by p1.)

Now, If only pr3 changed, then I don't want to build pr2, and if I only pr2 changed, I don't want to build pr3. For that I can add a blacklist that ignores pr3 in p2, and pr2 in p3.

So in my case, I am sure that changes in p3 will not break p2, and changes in p2 will not break p1.

=> As promised, I will return with a confirmed example.

[tomzo]

There is one minor bug that I am aware of. If you are introducing whitelist/blacklist change in current commit 123, then this will not apply to builds triggered by it. Meaning 123 will trigger pipelines using old rules. But every next build should respect filters. Be aware of this when you confirm this bug.

[rkrisztian]

Thanks. It still does not work the way I expect (tested multiple times). Here is my config:

pipelines:
  p1:
    materials:
      my-repo:
        git: some-url
        auto_update: true
  p2:
    materials:
      my-repo:
        git: some-url
        blacklist:
          - pr3/**/*
        auto_update: true
      p1-as-upstream:
        pipeline: p1
        stage: some-stage
  p3:
    materials:
      my-repo:
        git: some-url
        blacklist:
          - pr2/**/*
        auto_update: true
      p1-as-upstream:
        pipeline: p1
        stage: some-stage

Then, if make a change inside pr3, the p2 pipeline is still triggered. I also retested with pr2 changes, expecting no triggering of p3, but it did. => Not OK.

If I removed p1-as-upstream from the material config of p2, then changes in pr3 did not trigger p2, which is OK, but then I can't use artifacts from p1.

=> It's like whitelists/blacklists are ignored when upstream pipelines are used.

[tomzo]

OK, but does pr2 and pr3 get triggered only after p1 was built? If so then it is expected.

Above looks similar to first config you posted, but it has additional p3, which should not matter much. If you configure this with XML (via UI), I believe it would behave the same. The p2 or p3 are built because p1 was built and it is a dependency. You cannot "blacklist" triggers caused by one of dependency pipelines.

[rkrisztian]

OK, but does pr2 and pr3 get triggered only after p1 was built?

Yes.

The p2 or p3 are built because p1 was built and it is a dependency.

This is unfortunate, as here I don't need an OR-condition for triggering, but an AND-condition (if that makes sense). It looks like that the material configuration is not as sophisticated...

If the XML behaves the same, then I conclude that the right support channel is the GoCD authors, not here. Thanks for your help though.

[tomzo]

You should look into user forum - https://groups.google.com/forum/#!forum/go-cd

Above behavior is the CD-way because in your case, you have built a new library in p1, which is used by p2 and p3. The change in library could be breaking either of the end user projects, which is why both must be built.
It was disturbing to me at first too.
One way to stop all these builds from happening is to not have have an upstream pipeline and hard code released version of the library in the end projects. Each project is then built in independent pipeline. Builds only when you update the project or reference to library version.

[rkrisztian]

I understand your argument, but in my case:

• Practically no changes in pr2 or pr3 would build a different artifact in p1, as p1 does not use those directories. The directories are practically completely isolated in that sense.
• And putting all projects in the same repository was an important design decision, so that we don't have to manage hard-coded versions.

=> I'll contact the GoCD group.

[arvindsv]

If you don't want pr2 or pr3 changes to build p1, then shouldn't you be adding those to the blacklist for p1?

[rkrisztian]

Thanks for the tip, @arvindsv, in fact you are right, but I tried it, and it's even worse: failed for the first 2 test cases below:

Test Case 1

• Change both pr1 and pr3 in the same commit.
• => p2 should not be built, but the rest.

Problem: all 3 builds were executed.

Test Case 2

• Do a change in pr3 only.
• => p1 & p2 should not be built, only p3.

Problem: no builds were executed. (Incorrect => serious issue.)

Test Case 3

• Do a change in pr1 only.
• => All pipelines should be rebuilt.

Worked.

[rkrisztian]

I contacted the developers directly, I think we should continue the conversation there (see the referenced issue).