Skip to content

Commit

Permalink
馃帀 Create a Helm Chart For Airbyte (#5891)
Browse files Browse the repository at this point in the history 
See number #1868. This creates an initial helm chart for installing Airbyte in Kubernetes to make it easier for users who are more familiar with helm. It also includes GitHub actions to help continually test that the chart works in the most basic case.
All of the templates are based off of the kustomize folder, but minio and postgres have been removed in favor of adding the bitnami helm charts as dependencies since they have an active community and allow easily tweaking their install.
  • Loading branch information
@jonstacks
jonstacks committed Sep 17, 2021
1 parent 108d9e0 commit fe460ce
Show file tree
Hide file tree
Showing 27 changed files with 1,898 additions and 0 deletions.
74 changes: 74 additions & 0 deletions .github/workflows/helm.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
name: Helm
on:
push:
paths:
- '.github/workflows/helm.yaml'
- 'charts/**'
pull_request:
paths:
- '.github/workflows/helm.yaml'
- 'charts/**'
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v2
- name: Setup Kubectl
uses: azure/setup-kubectl@v1
- name: Setup Helm
uses: azure/setup-helm@v1
with:
version: '3.6.3'
- name: Lint Chart
working-directory: ./charts/airbyte
run: ./ci.sh lint

generate-docs:
name: Generate Docs Parameters
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v2
- name: Checkout bitnami-labs/readme-generator-for-helm
uses: actions/checkout@v2
with:
repository: 'bitnami-labs/readme-generator-for-helm'
ref: '55cab5dd2191c4ffa7245cfefa428d4d9bb12730'
path: readme-generator-for-helm
- name: Install readme-generator-for-helm dependencies
working-directory: readme-generator-for-helm
run: npm install -g
- name: Test can update README with generated parameters
working-directory: charts/airbyte
run: ./ci.sh check-docs-updated

install:
name: Install
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v2
- name: Setup Kubectl
uses: azure/setup-kubectl@v1
- name: Setup Helm
uses: azure/setup-helm@v1
with:
version: '3.6.3'
- name: Setup Kind Cluster
uses: helm/kind-action@v1.2.0
with:
version: "v0.11.1"
image: "kindest/node:v1.21.1"
- name: Install airbyte chart
working-directory: ./charts/airbyte
run: ./ci.sh install
- if: always()
name: Print diagnostics
working-directory: ./charts/airbyte
run: ./ci.sh diagnostics
- if: success()
name: Test airbyte chart
working-directory: ./charts/airbyte
run: ./ci.sh test
2 changes: 2 additions & 0 deletions charts/airbyte/.gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Charts are downloaded at install time with `helm dep build`.
charts
25 changes: 25 additions & 0 deletions charts/airbyte/.helmignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/

ci.sh
12 changes: 12 additions & 0 deletions charts/airbyte/Chart.lock
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
dependencies:
- name: common
repository: https://charts.bitnami.com/bitnami
version: 1.8.0
- name: postgresql
repository: https://charts.bitnami.com/bitnami
version: 10.9.4
- name: minio
repository: https://charts.bitnami.com/bitnami
version: 7.2.0
digest: sha256:6b5428c4bffa53a16e1015635d712f28b1c1991eea4d048928e4fbb88974cf4f
generated: "2021-08-31T22:58:47.835303672-05:00"
39 changes: 39 additions & 0 deletions charts/airbyte/Chart.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
apiVersion: v2
name: airbyte
description: Helm chart to deploy airbyte

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "0.29.13-alpha"

dependencies:
- name: common
repository: https://charts.bitnami.com/bitnami
tags:
- bitnami-common
version: 1.x.x
- condition: postgresql.enabled
name: postgresql
version: 10.x.x
repository: https://charts.bitnami.com/bitnami
- condition: minio.enabled
name: minio
version: 7.x.x
repository: https://charts.bitnami.com/bitnami
149 changes: 149 additions & 0 deletions charts/airbyte/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# airbyte

## Parameters

### Global Parameters

| Name | Description | Value |
| --------------------- | -------------------------------------------- | ----- |
| `global.storageClass` | Global StorageClass for Persistent Volume(s) | `""` |


### Common Parameters

| Name | Description | Value |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------- | --------------- |
| `nameOverride` | String to partially override airbyte.fullname template with a string (will prepend the release name) | `""` |
| `fullnameOverride` | String to fully override airbyte.fullname template with a string | `""` |
| `serviceAccount.annotations` | Annotations for service account. Evaluated as a template. Only used if `create` is `true`. | `{}` |
| `serviceAccount.create` | Specifies whether a ServiceAccount should be created | `true` |
| `serviceAccount.name` | Name of the service account to use. If not set and create is true, a name is generated using the fullname template. | `airbyte-admin` |


### Webapp Parameters

| Name | Description | Value |
| ---------------------------- | ---------------------------------------------------------------- | ---------------- |
| `webapp.replicaCount` | Number of webapp replicas | `1` |
| `webapp.image.repository` | The repository to use for the airbyte webapp image. | `airbyte/webapp` |
| `webapp.image.pullPolicy` | the pull policy to use for the airbyte webapp image | `IfNotPresent` |
| `webapp.image.tag` | The airbyte webapp image tag. Defaults to the chart's AppVersion | `0.29.13-alpha` |
| `webapp.podAnnotations` | Add extra annotations to the scheduler pod | `{}` |
| `webapp.service.type` | The service type to use for the webapp service | `ClusterIP` |
| `webapp.service.port` | The service port to expose the webapp on | `80` |
| `webapp.resources.limits` | The resources limits for the Web container | `{}` |
| `webapp.resources.requests` | The requested resources for the Web container | `{}` |
| `webapp.nodeSelector` | Node labels for pod assignment | `{}` |
| `webapp.tolerations` | Tolerations for webapp pod assignment. | `[]` |
| `webapp.ingress.enabled` | Set to true to enable ingress record generation | `false` |
| `webapp.ingress.className` | Specifies ingressClassName for clusters >= 1.18+ | `""` |
| `webapp.ingress.hosts` | Ingress Hosts configuration | `[]` |
| `webapp.ingress.annotations` | Ingress annotations done as key:value pairs | `{}` |
| `webapp.ingress.hosts` | The list of hostnames to be covered with this ingress record. | `[]` |
| `webapp.ingress.tls` | Custom ingress TLS configuration | `[]` |


### Scheduler Parameters

| Name | Description | Value |
| ------------------------------ | ------------------------------------------------------------------- | ------------------- |
| `scheduler.replicaCount` | Number of scheduler replicas | `1` |
| `scheduler.image.repository` | The repository to use for the airbyte scheduler image. | `airbyte/scheduler` |
| `scheduler.image.pullPolicy` | the pull policy to use for the airbyte scheduler image | `IfNotPresent` |
| `scheduler.image.tag` | The airbyte scheduler image tag. Defaults to the chart's AppVersion | `0.29.13-alpha` |
| `scheduler.podAnnotations` | Add extra annotations to the scheduler pod | `{}` |
| `scheduler.resources.limits` | The resources limits for the scheduler container | `{}` |
| `scheduler.resources.requests` | The requested resources for the scheduler container | `{}` |
| `scheduler.nodeSelector` | Node labels for pod assignment | `{}` |
| `scheduler.tolerations` | Tolerations for scheduler pod assignment. | `[]` |


### Pod Sweeper parameters

| Name | Description | Value |
| ------------------------------- | ---------------------------------------------------- | ----------------- |
| `podSweeper.image.repository` | The image repository to use for the pod sweeper | `bitnami/kubectl` |
| `podSweeper.image.pullPolicy` | The pull policy for the pod sweeper image | `IfNotPresent` |
| `podSweeper.image.tag` | The pod sweeper image tag to use | `latest` |
| `podSweeper.podAnnotations` | Add extra annotations to the podSweeper pod | `{}` |
| `podSweeper.resources.limits` | The resources limits for the podSweeper container | `{}` |
| `podSweeper.resources.requests` | The requested resources for the podSweeper container | `{}` |
| `podSweeper.nodeSelector` | Node labels for pod assignment | `{}` |
| `podSweeper.tolerations` | Tolerations for podSweeper pod assignment. | `[]` |


### Server parameters

| Name | Description | Value |
| ------------------------------------------- | ---------------------------------------------------------------- | ---------------- |
| `server.replicaCount` | Number of server replicas | `1` |
| `server.image.repository` | The repository to use for the airbyte server image. | `airbyte/server` |
| `server.image.pullPolicy` | the pull policy to use for the airbyte server image | `IfNotPresent` |
| `server.image.tag` | The airbyte server image tag. Defaults to the chart's AppVersion | `0.29.13-alpha` |
| `server.podAnnotations` | Add extra annotations to the server pod | `{}` |
| `server.livenessProbe.enabled` | Enable livenessProbe on the server | `true` |
| `server.livenessProbe.initialDelaySeconds` | Initial delay seconds for livenessProbe | `30` |
| `server.livenessProbe.periodSeconds` | Period seconds for livenessProbe | `10` |
| `server.livenessProbe.timeoutSeconds` | Timeout seconds for livenessProbe | `1` |
| `server.livenessProbe.failureThreshold` | Failure threshold for livenessProbe | `3` |
| `server.livenessProbe.successThreshold` | Success threshold for livenessProbe | `1` |
| `server.readinessProbe.enabled` | Enable readinessProbe on the server | `true` |
| `server.readinessProbe.initialDelaySeconds` | Initial delay seconds for readinessProbe | `10` |
| `server.readinessProbe.periodSeconds` | Period seconds for readinessProbe | `10` |
| `server.readinessProbe.timeoutSeconds` | Timeout seconds for readinessProbe | `1` |
| `server.readinessProbe.failureThreshold` | Failure threshold for readinessProbe | `3` |
| `server.readinessProbe.successThreshold` | Success threshold for readinessProbe | `1` |
| `server.resources.limits` | The resources limits for the server container | `{}` |
| `server.resources.requests` | The requested resources for the server container | `{}` |
| `server.service.type` | The service type to use for the API server | `ClusterIP` |
| `server.service.port` | The service port to expose the API server on | `8001` |
| `server.persistence.accessMode` | The access mode for the airbyte server pvc | `ReadWriteOnce` |
| `server.persistence.size` | The size of the pvc to use for the airbyte server pvc | `1Gi` |
| `server.persistence.storageClass` | The storage class to use for the airbyte server pvc | `""` |
| `server.nodeSelector` | Node labels for pod assignment | `{}` |
| `server.tolerations` | Tolerations for server pod assignment. | `[]` |


### Temporal parameters

| Name | Description | Value |
| --------------------------- | --------------------------------------------- | ----------------------- |
| `temporal.replicaCount` | The number of temporal replicas to deploy | `1` |
| `temporal.image.repository` | The temporal image repository to use | `temporalio/auto-setup` |
| `temporal.image.pullPolicy` | The pull policy for the temporal image | `IfNotPresent` |
| `temporal.image.tag` | The temporal image tag to use | `1.7.0` |
| `temporal.service.type` | The Kubernetes Service Type | `ClusterIP` |
| `temporal.service.port` | The temporal port and exposed kubernetes port | `7233` |
| `temporal.nodeSelector` | Node labels for pod assignment | `{}` |
| `temporal.tolerations` | Tolerations for pod assignment. | `[]` |


### Airbyte Database parameters

| Name | Description | Value |
| -------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------ |
| `postgresql.enabled` | Switch to enable or disable the PostgreSQL helm chart | `true` |
| `postgresql.postgresqlUsername` | Airbyte Postgresql username | `airbyte` |
| `postgresql.postgresqlPassword` | Airbyte Postgresql password | `airbyte` |
| `postgresql.postgresqlDatabase` | Airbyte Postgresql database | `db-airbyte` |
| `postgresql.existingSecret` | Name of an existing secret containing the PostgreSQL password ('postgresql-password' key) | `""` |
| `externalDatabase.host` | Database host | `localhost` |
| `externalDatabase.user` | non-root Username for Airbyte Database | `airbyte` |
| `externalDatabase.password` | Database password | `""` |
| `externalDatabase.existingSecret` | Name of an existing secret resource containing the DB password | `""` |
| `externalDatabase.existingSecretPasswordKey` | Name of an existing secret key containing the DB password | `""` |
| `externalDatabase.database` | Database name | `db-airbyte` |
| `externalDatabase.port` | Database port number | `5432` |


### Minio parameters

| Name | Description | Value |
| -------------------------- | ------------------------------------------------ | ----------- |
| `minio.enabled` | Switch to enable or disable the Minio helm chart | `true` |
| `minio.accessKey.password` | Minio Access Key | `minio` |
| `minio.secretKey.password` | Minio Secret Key | `minio123` |
| `externalMinio.host` | Minio Host | `localhost` |
| `externalMinio.port` | Minio Port | `9000` |


0 comments on commit fe460ce

Please sign in to comment.