diff --git a/docs/docs/using-semaphore/img/fan-out-fan-in.jpg b/docs/docs/using-semaphore/img/fan-out-fan-in.jpg new file mode 100644 index 000000000..950d08081 Binary files /dev/null and b/docs/docs/using-semaphore/img/fan-out-fan-in.jpg differ diff --git a/docs/docs/using-semaphore/img/fan-out-stage1.jpg b/docs/docs/using-semaphore/img/fan-out-stage1.jpg new file mode 100644 index 000000000..2ac55451b Binary files /dev/null and b/docs/docs/using-semaphore/img/fan-out-stage1.jpg differ diff --git a/docs/docs/using-semaphore/img/fan-out-stage2.jpg b/docs/docs/using-semaphore/img/fan-out-stage2.jpg new file mode 100644 index 000000000..f171c75b7 Binary files /dev/null and b/docs/docs/using-semaphore/img/fan-out-stage2.jpg differ diff --git a/docs/docs/using-semaphore/img/fan-out-stage3.jpg b/docs/docs/using-semaphore/img/fan-out-stage3.jpg new file mode 100644 index 000000000..26b77a869 Binary files /dev/null and b/docs/docs/using-semaphore/img/fan-out-stage3.jpg differ diff --git a/docs/docs/using-semaphore/jobs.md b/docs/docs/using-semaphore/jobs.md index a95f095fa..e01b41dd0 100644 --- a/docs/docs/using-semaphore/jobs.md +++ b/docs/docs/using-semaphore/jobs.md @@ -1089,6 +1089,8 @@ It's not possible to use job parallelism at the same time as [job matrices](#mat ## Job matrix {#matrix} + + A job matrix is a more advanced form of [job parallelism](#job-parallelism) where you can define multiple variables with different values and run all the possible permutations. For example, let's say we want to test our application using three Node.js versions using npm and yarn diff --git a/docs/docs/using-semaphore/workflows.md b/docs/docs/using-semaphore/workflows.md index b154e40ff..18814dd60 100644 --- a/docs/docs/using-semaphore/workflows.md +++ b/docs/docs/using-semaphore/workflows.md @@ -27,7 +27,7 @@ Before you can run a workflow in Semaphore you need: - A Semaphore project linked to the repository - One or more pipelines -The [project page](./projects#view-projects) shows all the recent workflows for the project. +The [project page](./projects#view-projects) shows all the recent workflows for the project. ![Types of workflows](./img/workflow-types.jpg) @@ -35,7 +35,7 @@ The [project page](./projects#view-projects) shows all the recent workflows for -You can define most aspects of your workflows using the visual editor. +You can define most aspects of your workflows using the visual editor. To access the editor, open one of your projects on and press **Edit Workflow**. All your changes are stored as YAML pipeline definitions on your Git repository. You can make changes using the visual editor, or edit the YAML directly by clicking on the pipeline YAML file. @@ -43,6 +43,135 @@ To access the editor, open one of your projects on and press **Edit Workflow**. See the [jobs page](./jobs) to learn how to define jobs and blocks. +## Modeling Complex Workflows {#modeling-complex-workflows} + +This section provides guides to model complex, non-linear CI/CD processes. + +### Fan-out Fan-In {#fan-out-fan-in} + + + +The Fan-Out Fan-In workflow provides consistency and maximum parallelization. It can be split into 3 stages: + +1. **Build stage**: you build your project once +2. **Fan-Out stage**: all your tests fan out from the build stage and run in parallel +3. **Fan-In stage**: once tested, the workflow fans in to a release or deploy stage + +![Fan-Out Fan-In](./img/fan-out-fan-in.jpg) + + + + + + +1. Create your build [job](./jobs). Depending on the nature of your project, you can save the built artifact to the [artifact store](./artifacts) or push it to a [Docker registry](./containers/docker) + + ![Build stage](./img/fan-out-stage1.jpg) + +2. Add your test jobs. Set dependencies so all your tests depend on the build job created on Step 1 + + ![Fan-out stage](./img/fan-out-stage2.jpg) + +3. Add the release/deploy job. Use dependencies so the new job depends on all your tests. This will make the release/job run only if all tests have passed + + ![Fan-in stage](./img/fan-out-stage3.jpg) + + + + + + +You can create a Fan-Out Fan-In workflow by setting the dependencies in your blocks. The Fan-Out stage is achieved by defining `dependencies`. For example: + +```yaml +blocks: + - name: "Build" + dependencies: [] + ... + + - name: "Unit tests" + dependencies: ["Build"] + ... + + - name: "Integration tests" + dependencies: ["Build"] + ... + + - name: "E2E tests" + dependencies: ["Build"] + ... + + - name: "Release candidate" + dependencies: ["Integration tests", "Unit tests", "E2E tests"] + ... +``` + +Find below a complex example of a Fan-Out, Fan-In for a Node-based workflow: + +```yaml +version: v1.0 +name: Continuous Integration Pipeline +agent: + machine: + type: f1-standard-2 + os_image: ubuntu2204 +blocks: + - name: Build Stage + dependencies: [] + task: + jobs: + - name: Build Project + commands: + - 'checkout' + - 'npm run build' + - 'artifact push workflow build/' + - name: End to end tests + dependencies: + - Build Stage + task: + jobs: + - name: Run E2E tests + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run test:e2e' + - name: Integration Tests + dependencies: + - Build Stage + task: + jobs: + - name: Run integration tests + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run test:integration' + - name: Unit tests + dependencies: + - Build Stage + task: + jobs: + - name: Run unit tests + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run test:unit' + - name: Release candidate + dependencies: + - End to end tests + - Integration Tests + - Unit tests + task: + jobs: + - name: Generate RC + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run package' +``` + + + + ## Workflow triggers The following events or actions trigger workflows by default: @@ -50,7 +179,7 @@ The following events or actions trigger workflows by default: - Pushing commits into any branch - Pushing Git tags - Changing any pipelines -- Manually re-running workflows +- Manually re-running workflows - Running pipelines using [Tasks](./tasks) Additionally, you can configure workflows to be triggered by: @@ -117,12 +246,11 @@ Approving forked pull requests is limited to new comments only and does not work ## How to skip commits {#skip} -If you don't want to start a workflow, type one of the following strings in the commit message. +If you don't want to start a workflow, type one of the following strings in the commit message. - `[ci skip]` - `[skip ci]` - For example, this push does not trigger a Semaphore pipeline execution, it is completely ignored: ```shell title="Skipping a commit" diff --git a/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-fan-in.jpg b/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-fan-in.jpg new file mode 100644 index 000000000..950d08081 Binary files /dev/null and b/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-fan-in.jpg differ diff --git a/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-stage1.jpg b/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-stage1.jpg new file mode 100644 index 000000000..2ac55451b Binary files /dev/null and b/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-stage1.jpg differ diff --git a/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-stage2.jpg b/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-stage2.jpg new file mode 100644 index 000000000..f171c75b7 Binary files /dev/null and b/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-stage2.jpg differ diff --git a/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-stage3.jpg b/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-stage3.jpg new file mode 100644 index 000000000..26b77a869 Binary files /dev/null and b/docs/versioned_docs/version-CE/using-semaphore/img/fan-out-stage3.jpg differ diff --git a/docs/versioned_docs/version-CE/using-semaphore/jobs.md b/docs/versioned_docs/version-CE/using-semaphore/jobs.md index f22be6a64..cf34de6b3 100644 --- a/docs/versioned_docs/version-CE/using-semaphore/jobs.md +++ b/docs/versioned_docs/version-CE/using-semaphore/jobs.md @@ -4,12 +4,6 @@ description: Jobs and blocks are the basic unit of work # Jobs - - - - - - Jobs get stuff done. This page explains to create and configure jobs. ## Job lifecycle {#job-lifecycle} @@ -1025,6 +1019,8 @@ It's not possible to use job parallelism at the same time as [job matrices](#mat ## Job matrix {#matrix} + + A job matrix is a more advanced form of [job parallelism](#job-parallelism) where you can define multiple variables with different values and run all the possible permutations. For example, let's say we want to test our application using three Node.js versions using npm and yarn diff --git a/docs/versioned_docs/version-CE/using-semaphore/workflows.md b/docs/versioned_docs/version-CE/using-semaphore/workflows.md index 3fc4649ff..632867416 100644 --- a/docs/versioned_docs/version-CE/using-semaphore/workflows.md +++ b/docs/versioned_docs/version-CE/using-semaphore/workflows.md @@ -43,6 +43,135 @@ To access the editor, open one of your projects on and press **Edit Workflow**. See the [jobs page](./jobs) to learn how to define jobs and blocks. +## Modeling Complex Workflows {#modeling-complex-workflows} + +This section provides guides to model complex, non-linear CI/CD processes. + +### Fan-out Fan-In {#fan-out-fan-in} + + + +The Fan-Out Fan-In workflow provides consistency and maximum parallelization. It can be split into 3 stages: + +1. **Build stage**: you build your project once +2. **Fan-Out stage**: all your tests fan out from the build stage and run in parallel +3. **Fan-In stage**: once tested, the workflow fans in to a release or deploy stage + +![Fan-Out Fan-In](./img/fan-out-fan-in.jpg) + + + + + + +1. Create your build [job](./jobs). Depending on the nature of your project, you can save the built artifact to the [artifact store](./artifacts) or push it to a [Docker registry](./containers/docker) + + ![Build stage](./img/fan-out-stage1.jpg) + +2. Add your test jobs. Set dependencies so all your tests depend on the build job created on Step 1 + + ![Fan-out stage](./img/fan-out-stage2.jpg) + +3. Add the release/deploy job. Use dependencies so the new job depends on all your tests. This will make the release/job run only if all tests have passed + + ![Fan-in stage](./img/fan-out-stage3.jpg) + + + + + + +You can create a Fan-Out Fan-In workflow by setting the dependencies in your blocks. The Fan-Out stage is achieved by defining `dependencies`. For example: + +```yaml +blocks: + - name: "Build" + dependencies: [] + ... + + - name: "Unit tests" + dependencies: ["Build"] + ... + + - name: "Integration tests" + dependencies: ["Build"] + ... + + - name: "E2E tests" + dependencies: ["Build"] + ... + + - name: "Release candidate" + dependencies: ["Integration tests", "Unit tests", "E2E tests"] + ... +``` + +Find below a complex example of a Fan-Out, Fan-In for a Node-based workflow: + +```yaml +version: v1.0 +name: Continuous Integration Pipeline +agent: + machine: + type: f1-standard-2 + os_image: ubuntu2204 +blocks: + - name: Build Stage + dependencies: [] + task: + jobs: + - name: Build Project + commands: + - 'checkout' + - 'npm run build' + - 'artifact push workflow build/' + - name: End to end tests + dependencies: + - Build Stage + task: + jobs: + - name: Run E2E tests + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run test:e2e' + - name: Integration Tests + dependencies: + - Build Stage + task: + jobs: + - name: Run integration tests + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run test:integration' + - name: Unit tests + dependencies: + - Build Stage + task: + jobs: + - name: Run unit tests + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run test:unit' + - name: Release candidate + dependencies: + - End to end tests + - Integration Tests + - Unit tests + task: + jobs: + - name: Generate RC + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run package' +``` + + + + ## Workflow triggers The following events or actions trigger workflows by default: diff --git a/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-fan-in.jpg b/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-fan-in.jpg new file mode 100644 index 000000000..950d08081 Binary files /dev/null and b/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-fan-in.jpg differ diff --git a/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-stage1.jpg b/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-stage1.jpg new file mode 100644 index 000000000..2ac55451b Binary files /dev/null and b/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-stage1.jpg differ diff --git a/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-stage2.jpg b/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-stage2.jpg new file mode 100644 index 000000000..f171c75b7 Binary files /dev/null and b/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-stage2.jpg differ diff --git a/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-stage3.jpg b/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-stage3.jpg new file mode 100644 index 000000000..26b77a869 Binary files /dev/null and b/docs/versioned_docs/version-EE/using-semaphore/img/fan-out-stage3.jpg differ diff --git a/docs/versioned_docs/version-EE/using-semaphore/jobs.md b/docs/versioned_docs/version-EE/using-semaphore/jobs.md index f22be6a64..cf34de6b3 100644 --- a/docs/versioned_docs/version-EE/using-semaphore/jobs.md +++ b/docs/versioned_docs/version-EE/using-semaphore/jobs.md @@ -4,12 +4,6 @@ description: Jobs and blocks are the basic unit of work # Jobs - - - - - - Jobs get stuff done. This page explains to create and configure jobs. ## Job lifecycle {#job-lifecycle} @@ -1025,6 +1019,8 @@ It's not possible to use job parallelism at the same time as [job matrices](#mat ## Job matrix {#matrix} + + A job matrix is a more advanced form of [job parallelism](#job-parallelism) where you can define multiple variables with different values and run all the possible permutations. For example, let's say we want to test our application using three Node.js versions using npm and yarn diff --git a/docs/versioned_docs/version-EE/using-semaphore/workflows.md b/docs/versioned_docs/version-EE/using-semaphore/workflows.md index 3fc4649ff..341466fac 100644 --- a/docs/versioned_docs/version-EE/using-semaphore/workflows.md +++ b/docs/versioned_docs/version-EE/using-semaphore/workflows.md @@ -43,6 +43,136 @@ To access the editor, open one of your projects on and press **Edit Workflow**. See the [jobs page](./jobs) to learn how to define jobs and blocks. +## Modeling Complex Workflows {#modeling-complex-workflows} + +This section provides guides to model complex, non-linear CI/CD processes. + +### Fan-out Fan-In {#fan-out-fan-in} + + + +The Fan-Out Fan-In workflow provides consistency and maximum parallelization. It can be split into 3 stages: + +1. **Build stage**: you build your project once +2. **Fan-Out stage**: all your tests fan out from the build stage and run in parallel +3. **Fan-In stage**: once tested, the workflow fans in to a release or deploy stage + +![Fan-Out Fan-In](./img/fan-out-fan-in.jpg) + + + + + + +1. Create your build [job](./jobs). Depending on the nature of your project, you can save the built artifact to the [artifact store](./artifacts) or push it to a [Docker registry](./containers/docker) + + ![Build stage](./img/fan-out-stage1.jpg) + +2. Add your test jobs. Set dependencies so all your tests depend on the build job created on Step 1 + + ![Fan-out stage](./img/fan-out-stage2.jpg) + +3. Add the release/deploy job. Use dependencies so the new job depends on all your tests. This will make the release/job run only if all tests have passed + + ![Fan-in stage](./img/fan-out-stage3.jpg) + + + + + + +You can create a Fan-Out Fan-In workflow by setting the dependencies in your blocks. The Fan-Out stage is achieved by defining `dependencies`. For example: + +```yaml +blocks: + - name: "Build" + dependencies: [] + ... + + - name: "Unit tests" + dependencies: ["Build"] + ... + + - name: "Integration tests" + dependencies: ["Build"] + ... + + - name: "E2E tests" + dependencies: ["Build"] + ... + + - name: "Release candidate" + dependencies: ["Integration tests", "Unit tests", "E2E tests"] + ... +``` + +Find below a complex example of a Fan-Out, Fan-In for a Node-based workflow: + +```yaml +version: v1.0 +name: Continuous Integration Pipeline +agent: + machine: + type: f1-standard-2 + os_image: ubuntu2204 +blocks: + - name: Build Stage + dependencies: [] + task: + jobs: + - name: Build Project + commands: + - 'checkout' + - 'npm run build' + - 'artifact push workflow build/' + - name: End to end tests + dependencies: + - Build Stage + task: + jobs: + - name: Run E2E tests + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run test:e2e' + - name: Integration Tests + dependencies: + - Build Stage + task: + jobs: + - name: Run integration tests + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run test:integration' + - name: Unit tests + dependencies: + - Build Stage + task: + jobs: + - name: Run unit tests + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run test:unit' + - name: Release candidate + dependencies: + - End to end tests + - Integration Tests + - Unit tests + task: + jobs: + - name: Generate RC + commands: + - 'checkout' + - 'artifact pull workflow build/' + - 'npm run package' +``` + + + + + ## Workflow triggers The following events or actions trigger workflows by default: