The kURL manifest is made up of top level add-ons, each with it's own properties for configuration. kURL.sh hosts an API where users can POST kURL manifests and a deterministic hash is stored and returned. This hashed can then be used to retrieve a kURL installation script or air-gap bundle.
apiVersion: cluster.kurl.sh/v1beta1
kind: Installer
metadata:
name: 5e61e80
spec:
kubernetes:
version: 1.23.5
containerd:
version: 1.4.13
weave:
version: 2.6.5
longhorn:
version: 1.2.2
ekco:
version: 0.19.0
minio:
version: 2020-01-25T02-50-51Z
contour:
version: 1.20.1
registry:
version: 2.7.1
kotsadm:
version: 1.69.1A URL can be constructed including the hash to retrieve a kURL script (curl https://k8s.kurl.sh/5e61e80) or air-gap bundle (curl -LO https://k8s.kurl.sh/bundle/5e61e80.tar.gz). Additionally, a version of kURL can be pinned in that URL (https://k8s.kurl.sh/version/v2022.04.19-0/5e61e80).
The kURL.sh API can be used to create installers and retrieve install scripts and air-gap bundles.
The kURL API is made up of 5 services.
https://github.com/replicatedhq/kurl-api
The browser UI for creating and viewing kURL installers.
https://github.com/replicatedhq/kURL-api/tree/main/src
The API for creating kURL URLs and rendering installers. The API accepts a kURL Installer spec and returns a URL including a deterministic hash of the spec for installing a cluster based on that spec, for example https://k8s.kurl.sh/5e61e80. This reproducible hash is the first 7 characters of the SHA-256 checksum of the sorted spec. The API stores these hashes in its relational database for retrieval, as a hash is not reversible. Additionally, the API is responsible for resolving the add-on version ("latest" and ".x") and rendering the spec when a script is requested, for example https://k8s.kurl.sh/5e61e80/install.sh or shorthand https://k8s.kurl.sh/5e61e80. The API reads the current kURL version from the object storage bucket (https://kurl-sh.s3.amazonaws.com/dist/VERSION) and uses that version to lookup add-on version information (e.g. https://kurl-sh.s3.amazonaws.com/dist/v2022.04.19-0/supported-versions-gen.json) for spec resolution.
https://github.com/replicatedhq/kURL-api/tree/main/cmd/server
This API proxies most requests directly to the Typescript API. Additionally, this API is responsible for assembling and streaming air-gap bundles back to the end-user. Air-gap packages are assembled from individual add-on package archives (e.g. https://kurl-sh.s3.amazonaws.com/dist/v2022.04.19-0/rook-1.0.4.tar.gz) and streamed as a single archive back to the end user when requested (e.g. https://k8s.kurl.sh/bundle/5e61e80.tar.gz).
Stores installer hashes.
The database schema is defined here as a SchemaHero Table.
NOTE: The schema is currently in a private repository.
Stores built kURL assets including add-on package archives.
| Object | Description |
|---|---|
| s3://[bucket]/[(dist|staging)]/VERSION | The current kURL version, used by the Typescript API to determine what version subdirectory to serve packages from |
| s3://[bucket]/[(dist|staging)]/[kURL version]/supported-versions-gen.json | Add-on and version information for a given release of kURL, used by the Typescript API for resolving "latest" and "dot x" versions as well as validating installers |
| s3://[bucket]/[(dist|staging)]/[kURL version]/[entrypoint].tmpl | Script templates rendered by the Typescript API, install.sh, join.sh, upgrade.sh and tasks.sh |
| s3://[bucket]/[(dist|staging)]/[kURL version]/[addon-version].tar.gz | Add-on package archives |
| Script | Description |
|---|---|
| install.sh | Initialize and upgrade a kURL cluster. |
| join.sh | Join additional nodes to the cluster. |
| upgrade.sh | Utility script for upgrading additional nodes. |
| tasks.sh | For various tasks including re-generating the join script and loading images from an air-gap bundle. |
Exists on the server at /var/lib/kurl.
| Directory | Description |
|---|---|
| addons/ | Stores add-ons, including images and assets. |
| bin/ | Stores utility binaries. |
| krew/ | Has some plugins we use including support-bundle and preflight. |
| kustomize/ | Scripts output |
| packages/ | Host packages |
| shared/ | Everything else |
Add-ons are components that make up a kURL cluster.
- Kubernetes distribution - Kubeadm
- CRI - Docker or Containerd
- CNI - Flannel, Weave or Antrea
- CSI - Longhorn, Rook, OpenEBS
- Ingress - Contour
- Misc. - Prometheus, KOTS, Velero...
| Directory | Description |
|---|---|
| addons/[addon]/[version]/Manifest | Manifest of assets, host packages and container images |
| addons/[addon]/[version]/install.sh | Entrypoint to the add-on installation script |
| addons/[addon]/[version]/host-preflight.yaml | Troubleshoot.sh preflight spec |
| addons/[addon]/[version]/assets/ | Runtime assets built during CI |
| addons/[addon]/[version]/images/ | Runtime images built during CI |
| addons/[addon]/[version]/[distro-version]/ | Runtime host packages for each supported Linux OS built during CI |
Add-ons can implement a set of lifecycle hooks that are invoked when creating, joining or upgrading the cluster. See the flow charts for more details.
For more details about each add-on lifecycle hook, see the add-on README.md
kURL maintains a list of externally built and hosted add-ons (current only "kotsadm").
kURL automation, more specifically the import-external-addons GitHub action, polls this list for newly available versions.
New versions are published to the external add-on registry and packages are copied from the source and stored in the kURL S3 bucket.
The kURL API merges the external add-on registry with its internal list of add-on versions, making them available to the end-user.
Upon releasing kURL, scripts and add-on package archives are built and uploaded to to the kURL object storage bucket along with metadata including the Git sha from which they were generated. Once complete, the VERSION file is updated with the current version of kURL. The Typescript API makes use of this VERSION file to resolve the scripts and add-on packages.
Production release are triggered by running the command make tag-and-release or pushing a tag in the format "v*..".
Production releases are uploaded to the object storage bucket at prefix dist, for example https://kurl-sh.s3.amazonaws.com/dist/v2022.04.19-0/.
Before building add-on packages, the workflow will first check if there were any changes made to the add-on since the previous production release based on metadata included with the add-on package.
If no changes were made, the package will be copied from the previous production release to optimize for build times.
Next, the workflow will check if there were any changes made since the previous staging release and copy from staging if no changes were made.
The is to account for a scenario where a commit were being tagged other than what is at the HEAD of main.
If changes were made to both production and staging, the package will be built from source and uploaded.
Finally, VERSION file is updated at https://kurl-sh.s3.amazonaws.com/dist/VERSION to point to the new production version.
Historical production releases are never removed from the object storage bucket.
Staging releases are triggered on merge to main.
Staging release versions use the most current release version tag and append the short commit sha -abcdef0, for example v2022.04.19-0-5af497c.
Due to this versioning scheme, staging releases will overwrite the previous staging release if no production release occurred prior.
This is intentional to optimize for storage costs.
Staging releases are uploaded to the object storage bucket at prefix staging, for example https://kurl-sh.s3.amazonaws.com/staging/v2022.04.19-0-5af497c/.
Before building add-on packages, the workflow will first check if there were any changes made to the add-on since the previous staging release based on metadata included with the add-on package.
If no changes were made, the package will be copied from the previous staging release to optimize for build times.
If changes were made, the package will be built from source and uploaded.
Finally, VERSION file is updated at https://kurl-sh.s3.amazonaws.com/staging/VERSION to point to the new staging version.
Historical staging releases are never removed from the object storage bucket.