New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Combined/aggregated triggers #2324

Closed
romerojunior opened this Issue Jun 27, 2018 · 7 comments

Comments

Projects
None yet
5 participants
@romerojunior
Copy link

romerojunior commented Jun 27, 2018

Feature Request

What challenge are you facing?

We currently need a way of aggregating a trigger, for example:

  • If you have a integration test job that depends on two (or more) resources outputted by previous jobs, you would want this test to ONLY be triggered after all previous jobs have been successfully completed.

A Modest Proposal

Currently, when using multiple get steps with triggers, the expected behaviour is similar to an OR operation, for example:

 - name: INTEGRATION_TEST
    plan:
    - aggregate:
      - get: package_A
        passed: [BUILD_PACKAGE_A]
        trigger: true
      - get: package_B
        passed: [BUILD_PACKAGE_B]
        trigger: true

In the example above, the INTEGRATION_TEST will be triggered whenever package_A OR package_B is successfully outputted by their respective build jobs.

My proposal is to come up with a way of (optionally) changing the default OR behaviour to an AND, where both BUILD_PACKAGE_A and BUILD_PACKAGE_B jobs have to successfully output the resources fetched by the INTEGRATION_TEST in order for the trigger to work. In other words: INTEGRATION_TEST would wait for all get steps to be triggered.

 - name: INTEGRATION_TEST
    plan:
    - trigger:
      - get: package_A
        passed: [BUILD_PACKAGE_A]
      - get: package_B
        passed: [BUILD_PACKAGE_B]
@cloudsquirrel

This comment has been minimized.

Copy link

cloudsquirrel commented Jul 11, 2018

precisely what we've been looking for! 👍

@vito

This comment has been minimized.

Copy link
Member

vito commented Jul 24, 2018

I'm not sure I follow. How is this different from a normal fan-in, where you list multiple jobs in passed? Can you provide a full pipeline example for this?

@vito vito added this to Icebox in Core via automation Jul 24, 2018

@romerojunior

This comment has been minimized.

Copy link
Author

romerojunior commented Jul 24, 2018

Hallo!

@vito currently we're limited to: if a single condition is met, the job will be triggered.

As I see it, the limitation is not related to the passed functionality, but the trigger. Imagine you wish to start (trigger) a job only if a series of conditions have been met.

One example for this would be an application that requires multiple modules/libraries/whatever to be compiled/tested separately, but still, at some point in the pipeline, if they all pass a series of jobs, they can be put together, tested and packaged.

With this "if a single condition is met, the job will be triggered" logic in place, this "final packaging job" (as described above) will be triggered for each time a single module/library/whatever hits the trigger, generating a bad package.

@vito

This comment has been minimized.

Copy link
Member

vito commented Jul 24, 2018

@romerojunior But don't you need something correlating the packages so that you know they're "for" each other in the first place? Or do you want it to only run when there are new versions of all 3 resources? (...But what if you only need to make a change to one?)

I'd recommend taking a look at the "Fanning In" example here, if you haven't already: https://concourse-ci.org/get-step.html - it shows how passed implements fan-in and also correlates jobs across get steps. Not sure understanding this is core to your issue but it should help anyway.

As a real-world example, the Concourse project puts all of our components in one repo as submodules. This way each commit of the toplevel repo is the "source of truth" for the versions that are expected to be compatible with each other. We push a bump commit, which then triggers individual unit tests, each with the same resource as their input. We then "fan in" on that same resource (concourse in our case) having passed all of the individual unit test jobs, and pass it along to the next job.

You can see the rendered pipeline here: https://ci.concourse-ci.org/teams/main/pipelines/main - and the config here: https://github.com/concourse/pipelines/blob/ae3d57a596adb063b2066d1c3b096b172d32a86c/ci.concourse-ci.org/concourse.yml#L57-L184 (I've highlighted a subset that's relevant, but there are other examples of this workflow in the same config.)

Without having a single resource that you can correlate the jobs and resources to, you would need something like #604 if you want to only trigger on them all changing. But then I'm still not sure how you'd handle the (I'd expect common) case of only changing one of the packages.

@JohannesRudolph

This comment has been minimized.

Copy link
Contributor

JohannesRudolph commented Nov 28, 2018

Just stumbled across this, we solved our needs for complex fan-in logic using https://github.com/Meshcloud/gate-resource. It doesn't exactly make concourse itself understand these fan-ins on a resource-level but we use the metadata published in gates to do fan-in in task code (i.e. download specific artifact revisions).

This is certainly not ideal and you should rely as much as possible on concourse's preferred way to do things. However, for our use case of building & testing a mono-repo of many services efficiently and collecting the builds for deployment concourse's available resource-flows didn't cut it, but gate-resource has been working great.

@vito vito removed this from Icebox in Core Jan 29, 2019

@loganmzz

This comment has been minimized.

Copy link

loganmzz commented Feb 7, 2019

To handle such cases I passed original resource (source code ?) across entire pipeline to form a single stream. It doesn't do the job ?

---
resources:
- name: source-code
- name: package-common
- name: package-a
- name: package-b

jobs:
- name: build-common
  plan:
  - get: source-code
  - task: build-common
  - put: package-common

- name: build-a
  plan:
  - aggregate:
    - get: source-code
      passed: [build-common]
    - get: package-common
      passed: [build-common]
  - task: build-a
  - put: package-a

- name: build-b
  plan:
  - aggregate:
    - get: source-code
      passed: [build-common]
    - get: package-common
      passed: [build-common]
  - task: build-b
  - put: package-b

- name: check-integration
  plan:
  - aggregate:
    - get: source-code
      passed: [build-a, build-b]
    - get: package-a
      passed: [build-a]
    - get: package-b
      passed: [build-b]
  - task: run-integration-test
@no-response

This comment has been minimized.

Copy link

no-response bot commented Feb 12, 2019

This issue has been automatically closed because there has been no response to our request for more information from the original author.
It will be re-opened when the original author replies. Otherwise, a new issue should be created to continue the conversation. Thank you!

@no-response no-response bot closed this Feb 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment