Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP:docs(native-tekton): add initial docs explaining conversion procedure #3705

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion content/en/v3/about/changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ updatebot fatal: could not read Username for 'https://github.com': No such devic
* its now much easier to [write system tests against Preview Environments](https://github.com/jenkins-x/jx-preview#system-tests-in-previews) so it's easier to test images and charts function as you expect inside a Pull Request before you are happy to merge the work for faster feedback
* check out the new [DevOps, GitOps and Cloud Native](https://jenkins-x.io/v3/devops/) documentation we're putting together based on the learnings of continuously deliverying Jenkins X with Jenkins X.
* new clusters created using Terraform that use Vault will be using the `jx-vault` namespace to setup Vault (so that its managed by Terraform)
* you can now use [jx pipeline convert](/v3/develop/reference/jx/pipeline/convert) to [convert any old pipelines](/v3/develop/pipelines/upgrading/#converting-older-pipelines) across to the latest [concise syntax](/v3/develop/pipelines/catalog/)
* you can now use [jx pipeline convert uses](/v3/develop/reference/jx/pipeline/convert) to [convert any old pipelines](/v3/develop/pipelines/upgrading/#converting-older-pipelines) across to the latest [concise syntax](/v3/develop/pipelines/catalog/)
* we have an awesome [new syntax to help share pipelines across git repositories](/v3/develop/pipelines/catalog/) that makes it easier to simplify the pipelines in each repository while keeping things vanilla Tekton YAML and letting you override and customise anything anywhere
* the boot job now upgrades the `docs` folder to show what charts and versions are deployed in each namespace. You can view the `docs` folder in your own git repositories once you've [upgraded your cluster](/v3/admin/setup/upgrades/cluster/)
* you can see the default reports for [kubernetes](https://github.com/jx3-gitops-repositories/jx3-kubernetes/tree/master/docs ), [aws](https://github.com/jx3-gitops-repositories/jx3-eks-vault/tree/master/docs), [azure](https://github.com/jx3-gitops-repositories/jx3-azure-akv) and [gke](https://github.com/jx3-gitops-repositories/jx3-gke-gsm/tree/master/docs )
Expand Down
4 changes: 2 additions & 2 deletions content/en/v3/admin/guides/migrate/v2.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ You can then test the projects work fine in the new v3 cluster. If they do you c

## Converting pipelines

You can follow the [guide on migrating pipelines individually](/v3/develop/pipelines/upgrading/#converting-older-pipelines) via [jx pipeline convert](/v3/develop/reference/jx/pipeline/convert) command to convert pipelines in a repository:
You can follow the [guide on migrating pipelines individually](/v3/develop/pipelines/upgrading/#converting-older-pipelines) via [jx pipeline convert uses](/v3/develop/reference/jx/pipeline/convert) command to convert pipelines in a repository:

```bash
jx pipeline convert
jx pipeline convert uses
```

## Batch importing
Expand Down
4 changes: 2 additions & 2 deletions content/en/v3/admin/guides/migrate/v3-alpha.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ You can use the `--strategy` to change this default if you wish. See `jx updateb

## Converting pipelines

You can follow the [guide on migrating pipelines individually](/v3/develop/pipelines/upgrading/#converting-older-pipelines) via [jx pipeline convert](/v3/develop/reference/jx/pipeline/convert) command to convert pipelines in a repository:
You can follow the [guide on migrating pipelines individually](/v3/develop/pipelines/upgrading/#converting-older-pipelines) via [jx pipeline convert uses](/v3/develop/reference/jx/pipeline/convert) command to convert pipelines in a repository:

```bash
jx pipeline convert
jx pipeline convert uses
```

### If you lose some pipeline changes
Expand Down
14 changes: 14 additions & 0 deletions content/en/v3/admin/guides/native-tekton/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
title: Native Tekton
linktitle: Native Tekton
type: docs
description: How to integrate existing clusters with native Tekton
weight: 100
aliases:
- /v3/guides/native-tekton
---

As part of the efforts to upgrade Jenkins X's compatibility with more recent Kubernetes & Tekton versions we've made the
decision to fully migrate to native Tekton pipelines. This brings us inline with the latest version of Tekton;
and allows us to take advantage of features such as the Tekton Dashboard and whilst increasing the transparency of the
pipeline mechanisms.
100 changes: 100 additions & 0 deletions content/en/v3/admin/guides/native-tekton/changes-to-pipelines.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
---
title: Changes to Pipelines
linktitle: Changes to Pipelines
type: docs
description: Describes how the pipelines have changed with the move to native Tekton
weight: 100
aliases:
- /v3/guides/native-tekton
---

Migrating to native Tekton brings a number of changes to the way that Jenkins X pipelines are configured and run.
The main changes are:
- The lighthouse PipelineRun resolver (responsible for handling the `uses:sourceURI` syntax) has been replaced with
[Tekton's git resolver](https://tekton.dev/docs/pipelines/git-resolver/).
- Pipelines are no longer confined to a single pod running multiple containers (1 task, many steps). Instead, each step
now runs as its own pod (many tasks, many steps). This is to allow for the use of taskRefs to inherit tasks. Tekton
doesn't currently allow for step inheritance.
- As each pipeline now contains many tasks, [Tekton's workspaces](https://tekton.dev/docs/pipelines/workspaces/) are
used to share files between tasks.

For more information regarding how the new pipelines are configured, please see the [Tekton Docs](https://tekton.dev/docs/).

## Functionality Changes
We've tried to keep the functionality as close to the original as possible, however there are a few changes
that are worth noting.

### Version Stream
The version stream functionality continues to exist as before, so any pipelines using the
[pipeline catalog](https://github.com/jenkins-x/jx3-pipeline-catalog) will continue to have their version resolved by
lighthouse and the `versionStream` defined in your cluster repository.

Example git resolver version stream usage:
```yaml
params:
- name: url
value: https://github.com/spring-financial-group/jx3-pipeline-catalog.git
- name: revision
value: versionStream
- name: pathInRepo
value: tasks/git-clone/git-clone.yaml
```

### Lighthouse Default Params
Lighthouse will continue to add and populate the [default params and environment variables](https://jenkins-x.io/v3/develop/reference/variables/#environment-variables)
on PipelineRuns as before. However, due to taskRef resolution now being handled by Tekton, any standalone task will need
to have these explicitly defined. Propagated params and workspaces are part of the [Tekton roadmap](https://github.com/tektoncd/community/blob/main/roadmap.md#roadmap)
so will hopefully be available in the future.

### Overwriting Inherited Tasks
Previously it was possible to overwrite certain fields of inherited steps/tasks by redefining them in the child PipelineRun.
Unfortunately, Tekton does not currently support this functionality. As a result, to overwrite an inherited task, you
will need to define an embedded task on the child PipelineRun and copy in the relevant fields.

Inherited Task:
```yaml
- name: build-make-linux
runAfter:
- jx-variables
taskRef:
params:
- name: url
value: https://github.com/jenkins-x/jx3-pipeline-catalog.git
- name: revision
value: versionStream
- name: pathInRepo
value: tasks/go/pullrequest/build-make-linux.yaml
resolver: git
workspaces:
- name: output
workspace: pipeline-ws
```

Embedded Task:
```yaml
- name: build-make-linux
runAfter:
- jx-variables
taskSpec:
stepTemplate:
workingDir: /workspace/source
steps:
- image: golang:1.19.3@sha256:7ffa70183b7596e6bc1b78c132dbba9a6e05a26cd30eaa9832fecad64b83f029
name: build-make-linux
resources: {}
script: |
#!/bin/sh
make linux
workspaces:
- description: The workspace used to store the cloned git repository and the generated
files
mountPath: /workspace
name: output
workspaces:
- name: output
workspace: pipeline-ws
```

### Pipeline Catalog
The structure of the pipeline catalog has changed slightly. The `/tasks` directory will now only contain the tasks
themselves, with the PipelineRun definitions now only residing in the `/packs` directory.
55 changes: 55 additions & 0 deletions content/en/v3/admin/guides/native-tekton/pipeline-migration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
title: Pipeline Migration
linktitle: Pipeline Migration
type: docs
description: How to migrate existing pipelines to native Tekton
weight: 100
aliases:
- /v3/guides/native-tekton
---

## Prerequisites
You're cluster git repository will need to be at `versionStream>=[insert version here]` before proceeding with the
pipeline upgrade. This can be done by running the following command on your cluster git repository.
```bash
jx gitops upgrade
```
The upgrade will bring in the relevant pipeline changes for your cluster repository so the `verify` & `bootjob` pipelines
will continue to work during your migration.

## Upgrading pipelines in your repositories
You can upgrade the pipelines in all the repositories you've imported or created via quickstarts by running:
```bash
jx updatebot pipeline
```
This uses the default [kpt](https://kpt.dev/) strategy of `resource-merge` to merge any local changes with the those in the
[pipeline-catalog](https://github.com/jenkins-x/jx3-pipeline-catalog). The strategy used can be changed using the `--strategy` flag.
For more information see`jx updatebot pipeline --help`.

**Warning:** This is substantial change to the pipeline configuration and will likely result in merge conflicts.
If so, you will need to resolve these manually or by using the pipeline conversion tool seen in [converting pipelines](#converting-pipelines).

## Converting pipelines
Both in-repository pipelines and personal catalogs can be converted by using the following command in the root of the
repository:
```bash
jx pipeline convert remotetasks
```

The command identifies whether the PipelineRun is child or parent (in-repository or catalog) by checking for the presence
of a `uses:sourceURI` image in the stepTemplate.

Child PipelineRuns have they're steps converted to either taskRefs or embeddedTasks depending on whether the original step was inherited.

Parent PipelineRuns have their steps converted into standalone tasks as native Tekton does not support PipelineRuns
inheriting tasks from other PipelineRuns.

Standalone Tasks have the Lighthouse's default params and environment variables added to them. These can be found
[here](https://jenkins-x.io/v3/develop/reference/variables/#environment-variables).

By default, the workspace size of the converted pipeline is calculated from the size of the repository plus 300Mi - to account
for any additional files that may be added during the pipeline run. This can be overridden using the `--workspace-size` flag.
For catalogs the calculation can be disabled by setting `--calculate-workspace-volume=false` in which case the workspace size
will default to 1Gi.

For more information on this command, see `jx pipeline convert remotetasks --help`.
6 changes: 3 additions & 3 deletions content/en/v3/develop/pipelines/upgrading.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ weight: 500

## Converting older pipelines

If you have older pipelines in your git repository created with an older version of Jenkins X you can convert them to the new concise `image: uses:sourceURI` syntax via the [jx pipeline convert](/v3/develop/reference/jx/pipeline/convert) command:
If you have older pipelines in your git repository created with an older version of Jenkins X you can convert them to the new concise `image: uses:sourceURI` syntax via the [jx pipeline convert uses](/v3/develop/reference/jx/pipeline/convert) command:

```bash
jx pipeline convert
jx pipeline convert uses
```

The pipelines should be modified and if you have a `Kptfile` it will be removed.
Expand Down Expand Up @@ -76,4 +76,4 @@ You can now perform a batch of Pull Requests if you need to upgrade your pipelin

See: [generate pull requests to upgrade pipelines](/v3/admin/guides/migrate/v3-alpha/#upgrading-pipelines-in-your-repositories)

Here is an example of [an automated Pull Request](https://github.com/jenkins-x/jx-gitops/pull/551) that was used to migrate one of our pipelines on [jx-gitops](https://github.com/jenkins-x/jx-gitops) - as you can see the resulting file is much simpler and easier to maintain.
Here is an example of [an automated Pull Request](https://github.com/jenkins-x/jx-gitops/pull/551) that was used to migrate one of our pipelines on [jx-gitops](https://github.com/jenkins-x/jx-gitops) - as you can see the resulting file is much simpler and easier to maintain.
8 changes: 4 additions & 4 deletions content/en/v3/develop/reference/jx/pipeline/convert/_index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: jx pipeline convert
title: jx pipeline convert uses
linktitle: convert
type: docs
description: "Converts the pipelines to use the 'image: uses:sourceURI' include mechanism"
Expand All @@ -10,7 +10,7 @@ aliases:
### Usage

```
jx pipeline convert
jx pipeline convert uses
```

### Synopsis
Expand All @@ -24,10 +24,10 @@ So that pipelines are smaller, simpler and easier to upgrade pipelines with the
```bash
# Convert a repository created using the alpha/beta of v3
# to use the nice new uses: syntax
jx pipeline convert
jx pipeline convert uses

# Convert a pipeline catalog to the uses syntax and layout
jx pipeline convert --catalog
jx pipeline convert uses --catalog

```
### Options
Expand Down