Skip to content

Commit

Permalink
Defining test jobs for proposal 2
Browse files Browse the repository at this point in the history 
* Rename to test job, workflow, and instructions
* Split use cases and add sections for auth, versioning
  • Loading branch information
@nikimanoledaki
nikimanoledaki committed Jun 28, 2024
1 parent 599bf88 commit e783d3b
Show file tree
Hide file tree
Showing 2 changed files with 62 additions and 53 deletions.
0 ...posals/diagrams/calling-benchmark-job.png → docs/proposals/diagrams/calling-test-job.png
View file
File renamed without changes
115 changes: 62 additions & 53 deletions docs/proposals/proposal-002-run.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,12 @@ Draft
- [Risks and Mitigations](#risks-and-mitigations)
- [Design Details](#design-details)
- [Definitions](#definitions)
- [How the project workflow calls the benchmark job](#how-the-project-workflow-calls-the-benchmark-job)
- [Running \& Defining Benchmark Workflow](#running--defining-benchmark-workflow)
- [Calling the test workflow](#calling-the-test-workflow)
- [Use Case 1: A GitHub Action job using a workflow defined in the _same_ repository (preferred)](#use-case-1-a-github-action-job-using-a-workflow-defined-in-the-same-repository-preferred)
- [Use Case 2: A GitHub Action job using a workflow defined in a _different_ repository](#use-case-2-a-github-action-job-using-a-workflow-defined-in-a-different-repository)
- [Test instructions](#test-instructions)
- [Authentication](#authentication)
- [Versioning](#versioning)
- [Drawbacks (Optional)](#drawbacks-optional)
- [Alternatives](#alternatives)
- [Infrastructure Needed (Optional)](#infrastructure-needed-optional)
Expand Down Expand Up @@ -78,7 +82,7 @@ List the specific goals of the proposal. What is it trying to achieve? How will
know that this has succeeded?
-->

- Describe the actions to take immediately after the trigger from [Proposal 1](https://github.com/cncf-tags/green-reviews-tooling/issues/84)
- Describe the actions to take immediately after the trigger and deployment of the CNCF project defined in [Proposal 1](./proposal-001-trigger-and-deploy.md)
- Describe how the pipeline should _fetch_ the benchmark tests either from this repository (`cncf-tags/green-reviews-tooling`) or from an upstream repository (e.g. Falco's [`falcosecurity/cncf-green-review-testing`](https://github.com/falcosecurity/cncf-green-review-testing)).
- Describe how the pipeline should _run_ the benchmark tests through GitHub Actions for a specific project e.g. Falco
- Communicate the changes needed to be made by the Falco team to change the benchmark test to a GitHub Action file.
Expand Down Expand Up @@ -179,98 +183,106 @@ change are understandable. This may include manifests or workflow examples
about HOW your proposal will be implemented, this is the place to discuss them.
-->

The Green Reviews automated pipeline relies on putting together different reusable GitHub Action workflows to modularise the different moving parts. A workflow runs one or more jobs, and each job runs one or more actions. It may be helpful to familiarise oneself with the documentation on [GitHub Action workflows](https://docs.github.com/en/actions/using-workflows/about-workflows) and especially [Reusing workflows](https://docs.github.com/en/actions/using-workflows/about-workflows) first to better understand the rest of the proposal as it explains some of these concepts well. The section on [Calling reusable workflows](https://docs.github.com/en/actions/using-workflows/about-workflows) describes an important concept that will be referenced later in this proposal.
The Green Reviews automated pipeline relies on putting together different reusable GitHub Action workflows to modularise the different moving parts. A workflow runs one or more jobs, and each job runs one or more actions. It may be helpful to familiarise oneself with the documentation on [GitHub Action workflows](https://docs.github.com/en/actions/using-workflows/about-workflows) and especially [Reusing workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows) first to better understand the rest of the proposal as it explains some of these concepts well. The section on [Calling reusable workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows#calling-a-reusable-workflow) describes an important concept that will be referenced later in this proposal.

### Definitions

There are different components defined here and shown in the following diagram.

![Green Reviews pipeline components](diagrams/green-reviews-pipeline-components.png "Green Reviews pipeline components")

Let's recap some of the components defined in [Proposal 1](proposal-001-trigger-and-deploy.md):
1. **Green Reviews pipeline**: the Continuous Integration pipeline which deploys a CNCF project to a test cluster, runs a set of benchmarks while measuring carbon emissions and stores the results. It is implemented by the workflows listed below.
2. **Cron workflow**: This refers to the initial GitHub Action workflow (described in proposal 1) and which dispatches a project workflow (see next definition).
3. **Project workflow**: The project workflow is dispatched by the Cron workflow. A project workflow can be, for example, a Falco workflow. A project workflow deploys the project, calls the benchmark workflow (see below), and then cleans up the deployment. A project workflow can be dispatched more than once if there are multiple subcomponents. In addition, a Project workflow, which is also just another GitHub Action workflow, contains a list of GitHub Action Jobs. These are a list of Benchmark Jobs - more info below.
4. **[new] Benchmark Job**: This is an instance of a GitHub Action Job within the Project workflow. The benchmark job runs the test benchmarks of a CNCF project. Which benchmark test to run is defined by inputs in the calling workflow: a CNCF project and a subcomponent. It is the main concern of this proposal. This is essentially the technical implementation of the benchmark test definition. It is what contains the test instructions. It is described in Defining Benchmark Jobs.
5. **[new] Benchmark workflow**: This is a separate manifest containing the Benchmark Instructions.
6. **[new] Benchmark Instructions**: This refers to the actual benchmark test steps that should run on the cluster. These are usually related to the tool's Functional Unit as defined by the SCI. It is described in Defining Benchmark Jobs.
3. **Benchmark workflow**: The project workflow is dispatched by the Cron workflow. A project workflow can be, for example, a Falco workflow. A project workflow deploys the project, calls the test workflow (see below), and then cleans up the deployment. A project workflow can be dispatched more than once if there are multiple subcomponents. In addition, a Project workflow, which is also just another GitHub Action workflow, contains a list of GitHub Action Jobs. These are a list of test jobs - more info below.

### How the project workflow calls the benchmark job
This proposal adds the following components:
1. **[new] Test job**: A test job is an instance of a GitHub Action Job. It is called right after deploying the CNCF project from the test workflow. The test job runs the test workflow and instructions for a CNCF project. Which benchmark test to run is defined by inputs in the calling workflow: a CNCF project and a subcomponent.
2. **[new] Test workflow**: A test workflow is a separate manifest containing the test instructions.
3. **[new] Test instructions**: Test instructions define what should run tests on the cluster. These are usually related to the tool's Functional Unit as defined by the SCI. It is described further in the sections below.

When the project workflow starts, it deploys the project on the test environment and then runs the benchmark job. For modularity and/or clarity, the benchmark test instructions could be defined in two different ways:
### Calling the test workflow

When the project workflow starts, it deploys the project on the test environment and then runs the test job. For modularity and/or clarity, the test instructions could be defined in two different ways:

As a Job that calls another GitHub Action workflow (yes, yet another workflow 🙂) that contains the instructions. The workflow can be either:
1. In the Green Reviews WG repository (**preferred**)
2. In a separate repository, such as an upstream CNCF project repository

The two use cases for defining a benchmark test are illustrated below.

![Calling the benchmark job](diagrams/calling-benchmark-job.png "Calling the Benchmark job")

### Running & Defining Benchmark Workflow
The two use cases for defining a test workflow are illustrated below.

This section defines _benchmark workflow_ and _benchmark test instructions_. It describes how to run them from the _project workflow_. It dives deeper into the following:
![Calling the test job](diagrams/calling-test-job.png "Calling the test job")

* How a benchmark workflow should be called from the project workflow
* What a benchmark workflow must contain in order to run on the cluster
* How a benchmark workflow is related to benchmark instructions
This section defines _test workflow_ and _test instructions_. It describes how to run them from the _project workflow_. It dives deeper into the following:

At a bare minimum, the benchmark workflow must contain test instructions of what should run in the Kubernetes cluster. For example, the Falco project maintainers have identified that one way to test the Falco project is through a test that runs `stress-ng` for a given period of time. The test instructions are contained in a Deployment manifest which can be directly applied to the benchmark test cluster using `kubectl`
* How a test workflow should be called from the project workflow
* What a test workflow must contain in order to run on the cluster
* How a test workflow is related to test instructions

Below are two use cases: the benchmark workflow may be defined in the Green Reviews repository or in a separate repository.
At a bare minimum, the test workflow must contain test instructions of what should run in the Kubernetes cluster. For example, the Falco project maintainers have identified that one way to test the Falco project is through a test that runs `stress-ng` for a given period of time. The test instructions are contained in a Deployment manifest which can be directly applied to the community cluster using `kubectl`

**Use Case 1: A GitHub Action job using a workflow defined in the _same_ repository**

In this use case, the project workflow’s benchmark job calls a separate workflow which is located in the Green Reviews repository.
The test workflows will be stored in the same JSON file as the other parameters for CNCF projects as defined in [Proposal 1](./proposal-001-trigger-and-deploy.md). It can be added as an additional input.

```yaml
# falco-ebpf-project-workflow.yaml
# .github/workflows/benchmark-pipeline.yaml
jobs:
# first, must authenticate to the Kubernetes cluster
# this is a Job
call-workflow-in-current-repo:
# job calls on a separate workflow
uses: octo-org/current-repo/.github/workflows/workflow.yml@v1 # refers to Job contained in the current repository
test-job:
# test job calls on test workflow
uses: ${{ inputs.test_path }} # refers to test workflow path
```

First, we assume that the workflow already contains a secret with a kubeconfig to authenticate with the test cluster (see for example here) and Falco has already been deployed to it. It is required that the pipeline authenticates with the Kubernetes cluster before running the job with the test.
This will fetch the workflow using the `jobs.<job_id>.uses` syntax defined [here](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_iduses).

Below are two use cases: the test workflow may be defined in the Green Reviews repository or in a separate repository.

The job workflow which contains the workflow with the benchmark test instructions is defined separately. In this case, it is in the Green Reviews repository.
#### Use Case 1: A GitHub Action job using a workflow defined in the _same_ repository (preferred)

If the test workflow is located in the Green Reviews repository, `test_path` would refer to, for example, `cncf-tags/green-reviews-tooling/.github/workflows/falco-test-workflow.yml@v1`.

In terms of the directory structure, in the `green-reviews-tooling` repository, we could create a subfolder such as `./github/workflows/tests` to contain the test workflows.

#### Use Case 2: A GitHub Action job using a workflow defined in a _different_ repository

We want to accommodate different methods of setting up the tests depending on the CNCF project. Given this, the test workflow containing the test instructions could be defined in a different repository. In this case, the `test_path` would be, for example, `falcosecurity/cncf-green-review-testing/.github/workflows/workflow.yml@v1`.

![Pipeline run](diagrams/pipeline-run.png "An example pipeline run")

### Test instructions

The test workflow which contains the test instructions may look like the following:

```yaml
# stress-ng-falco-test.yaml
# .github/workflows/tests/falco-test-workflow.yaml
jobs:
example_falco_test:
stress-ng-test:
runs-on: ubuntu-latest
steps:
- run: |
# the action to take here depends on the Functional Unit of the CNCF project. wait for amount of time, for resources
kubectl apply -f https://raw.githubusercontent.com/falcosecurity/cncf-green-review-testing/main/kustomize/falco-driver/ebpf/stress-ng.yaml
wait 15m
kubectl delete -f https://raw.githubusercontent.com/falcosecurity/cncf-green-review-testing/main/kustomize/falco-driver/ebpf/stress-ng.yaml
# other Falco tests:
# - redis-test e.g. https://github.com/falcosecurity/cncf-green-review-testing/blob/main/kustomize/falco-driver/ebpf/redis.yaml
# - event-generator-test e.g. https://github.com/falcosecurity/cncf-green-review-testing/blob/main/kustomize/falco-driver/ebpf/falco-event-generator.yaml
# TODO: should each test be a workflow or a job in a single workflow? as in, one test workflow per cncf project or multiple workflows per cncf project? TBD
```

The job applies the upstream Kubernetes manifest. The Kubernetes manifest contains the test instructions for Falco. It contains a while loop that runs stress-ng. The functional unit test is time-bound in this case and scoped to 15 minutes. Therefore, we deploy this test, wait for 15 minutes, then delete the manifest to end the loop. The action to take in the Job depends on the Functional Unit of the CNCF project.
The job has test instructions to apply the upstream Kubernetes manifest which contains a `while` loop that runs `stress-ng`. The manifest already defines where the test should run in the cluster i.e. in which namespace. The functional unit test is time-bound in this case and scoped to 15 minutes. Therefore, we deploy this test, wait for 15 minutes, then delete the manifest to end the loop. The test instructions depend on the functional unit of each CNCF project.

In the example above, the Kubernetes manifest is located in a different repository. This may be due to preferences for CNCF project maintainers who may have their own repository with configuration related to the Green Reviews pipeline, as is the case with the Falco project. To work around this, the Job points to the location of the `falcosecurity/cncf-green-review-testing repository`. The pipeline should accommodate different CNCF projects. If CNCF project maintainers are able to contribute to green-reviews-tooling, then they are welcome to store test artefacts in the green-reviews-tooling repository. There is also nothing stopping us from having a Kubernetes manifest inline.
In the example above, the Kubernetes manifest that is applied to the cluster is located in a different repository, but this could be located anywhere. This is due to preferences for CNCF project maintainers who may have a own repository with configuration related to the Green Reviews pipeline, as is the case with the Falco project.

For other CNCF projects, the test instructions could look different. For example, for Flux and ArgoCD, the benchmark tests defined here test these CNCF projects by using their CLI and performing CRUD operations on a demo workload. The functional unit of Flux and ArgoCD are different to Flux. For these GitOps tools, the functional unit depends on whether there are changes detected that they need to reconcile. In other words, the GitHub Action Job could do different things depending on how the CNCF project should be tested.
Each workflow should ensure that any artefacts that were deployed as part of the test instructions should be deleted at the end of the test run.

Lastly, it is important that the test should run in a way that is highly configurable. For example, the job can configure parameters to indicate where the test should run e.g. define which namespace.
### Authentication

**Use Case 2: A GitHub Action job using a workflow defined in a _different_ repository**
Before the test workflow is called on, we assume that the workflow already contains a secret with a kubeconfig to authenticate with the test cluster and Falco has already been deployed to it. It is required that the pipeline authenticates with the Kubernetes cluster before running the job with the test.

We want to accommodate different methods of setting up the tests depending on the CNCF project. Given this, the Job containing the test instructions could be defined in a different repository. In this case, the pipeline could call the job like this:
### Versioning

```yaml
# falco-ebpf-project-workflow.yaml
jobs:
call-workflow-in-another-repo:
uses: octo-org/another-repo/.github/workflows/workflow.yml@v1 # refer to other repo
```

A Job can be fetched from other GitHub organizations and repositories using the `jobs.<job_id>.uses` syntax defined [here](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_iduses). This syntax can be configured to use `@main` or `@another-branch` which would be nice for versioning and testing specific releases.

![Pipeline run](diagrams/pipeline-run.png "An example pipeline run")
For versioning, this syntax can be configured to use `@main` or `@another-branch` which would be nice for versioning and testing specific releases.

## Drawbacks (Optional)

Expand All @@ -297,8 +309,5 @@ new subproject, repos requested, or GitHub details. Listing these here allows a
SIG to get the process for these resources started right away.
-->

TODO:
* From the trigger pipeline, go to the Project Pipeline to run the tests of a specific CNCF project. → I suppose this is covered between Proposal 1 and this. Only thing we didn’t dig into is the subcomponents.
* Directory structure for CNCF projects, their pipeline/Jobs, etc. Collecting the Jobs & categorising them in "libraries"? → We should probably specify a standard way to store the tests in directories, where do they go and what is the directory structure, something in line with the subcomponents in Falco.
* Cleaning up test artefacts → Oops, not yet there either 😛
* Creating a template for how to "register" a benchmark test with everything that is needed and clear instructions on how to add a new one. Audience is primarily CNCF project maintainers.
<!-- TODO:
* Creating a template for how to "register" a benchmark test with everything that is needed and clear instructions on how to add a new one. Audience is primarily CNCF project maintainers. -->

0 comments on commit e783d3b

Please sign in to comment.