Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add helper image for generating static deploy yamls for the operator. #254

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ export NAMESPACE ?= devworkspace-controller
export IMG ?= quay.io/devfile/devworkspace-controller:next
export ROUTING_SUFFIX ?= 192.168.99.100.nip.io
export PULL_POLICY ?= Always
export WEBHOOK_ENABLED ?= true
export DEFAULT_ROUTING ?= basic
export KUBECONFIG ?= ${HOME}/.kube/config
REGISTRY_ENABLED ?= true
Expand Down Expand Up @@ -87,7 +86,6 @@ _print_vars:
@echo " IMG=$(IMG)"
@echo " PULL_POLICY=$(PULL_POLICY)"
@echo " ROUTING_SUFFIX=$(ROUTING_SUFFIX)"
@echo " WEBHOOK_ENABLED=$(WEBHOOK_ENABLED)"
@echo " DEFAULT_ROUTING=$(DEFAULT_ROUTING)"
@echo " REGISTRY_ENABLED=$(REGISTRY_ENABLED)"
@echo " DEVWORKSPACE_API_VERSION=$(DEVWORKSPACE_API_VERSION)"
Expand Down Expand Up @@ -369,6 +367,5 @@ help: Makefile
@echo ' KUBECONFIG - Kubeconfig which should be used for accessing to the cluster. Currently is: $(KUBECONFIG)'
@echo ' ROUTING_SUFFIX - Cluster routing suffix (e.g. $$(minikube ip).nip.io, apps-crc.testing)'
@echo ' PULL_POLICY - Image pull policy for controller'
@echo ' WEBHOOK_ENABLED - Whether webhooks should be enabled in the deployment'
@echo ' REGISTRY_ENABLED - Whether the plugin registry should be deployed'
@echo ' DEVWORKSPACE_API_VERSION - Branch or tag of the github.com/devfile/api to depend on. Defaults to master'
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,6 @@ The repository contains a Makefile; building and deploying can be configured via
| `NAMESPACE` | Namespace to use for deploying controller | `devworkspace-controller` |
| `ROUTING_SUFFIX` | Cluster routing suffix (e.g. `$(minikube ip).nip.io`, `apps-crc.testing`). Required for Kubernetes | `192.168.99.100.nip.io` |
| `PULL_POLICY` | Image pull policy for controller | `Always` |
| `WEBHOOK_ENABLED` | Whether webhooks should be enabled in the deployment | `false` |
| `REGISTRY_ENABLED` | Whether the plugin registry should be deployed | `true` |
| `DEVWORKSPACE_API_VERSION` | Branch or tag of the github.com/devfile/api to depend on | `v1alpha1` |

Expand Down
33 changes: 33 additions & 0 deletions build/yaml-templates/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
## Preparing static yaml files for deployment on OpenShift and Kubernetes

The Dockerfile in this directory can be used to automatically build static yaml files that can be applied to a cluster to deploy the DevWorkspace Operator without the need for kustomize.

### Building the image
```bash
docker build -t <template-image> -f yaml-builder.Dockerfile .
```
The docker build is self-contained; it clones the repo and builds directly from that codebase. Supported args are the same as those used in the Makefile, i.e.

| Arg | Purpose | Default |
| --- | ------- | ------- |
| DEVWORKSPACE_BRANCH | Branch of this repo to clone & check out | master
| NAMESPACE | Namespace used for deployment | devworkspace-controller
| IMG | DevWorkspace Operator image to be deployed | quay.io/devfile/devworkspace-controller:next
| PULL_POLICY | Sidecar PullPolicy used for workspaces | Always
| DEFAULT_ROUTING | Default routingClass to use for DevWorkspaces that do not define one | basic
| DEVWORKSPACE_API_VERSION | Commit hash of devfile/api dependency for DevWorkspace and DevWorkspaceTemplate CRDs | aeda60d4361911da85103f224644bfa792498499

### Using the image
The container image above creates a tarball of the relevant deployment files, and can be extracted from the container:
```bash
docker create --name builder <template-image>
docker cp builder:/devworkspace_operator_templates.tar.gz ./devworkspace_operator_templates.tar.gz
docker rm builder
tar -xzf devworkspace_operator_templates.tar.gz
```

This will extract the files to the `deploy` directory, with subdirectories for the OpenShift and Kubernetes deployments of the operator (on OpenShift, the service-ca operator is used to provide certificates where necessary; on Kubernetes the deployment depends on the cert-manager operator and includes a Certificate object).

Within each platform-dependent directory, the (large) file `combined.yaml` is a single file that can be applied to deploy the operator, and `objects/` contains each object in `combined.yaml` named according to `<resource-name>.<k8s-kind>.yaml`

As the yaml generation happens statically, the configmap leaves the `devworkspace.routing.cluster_host_suffix` property unset; on Kubernetes a value must be provided here to correctly generate ingresses.
59 changes: 59 additions & 0 deletions build/yaml-templates/prepare_templates.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#!/bin/bash

set -e

if [ -z "$DEVWORKSPACE_BRANCH" ]; then
echo "Environment variable DEVWORKSPACE_BRANCH must be set"
exit 1
fi

BASE_DIR=${1:-/build}
OUTPUT_DIR="${BASE_DIR}/deploy"
TARBALL_PATH="${BASE_DIR}/devworkspace_operator_templates.tar.gz"

# Clone repo
git clone --depth=1 --branch "${DEVWORKSPACE_BRANCH}" https://github.com/devfile/devworkspace-operator.git
cd devworkspace-operator

# Grab devfile/api CRDs
./update_devworkspace_crds.sh --init --api-version "$DEVWORKSPACE_API_VERSION"

# Fill env in template files
mv config/cert-manager/kustomization.yaml config/cert-manager/kustomization.yaml.bak
Copy link
Member

@sleshchenko sleshchenko Feb 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems to duplicate stuff we have in Makefile, which means we have more to maintain and test.

According to discussion with @tolusha, it would simplify things for chectl if it can just grab static files from devworkspace-operator repo (even if then it will modify some pieces, like namespace, possible operator image, ...).
After go vendoring we're not afraid of redundant ~16k lines )

So, it make me thinking what if:

  1. We have a script that generates static deploy files, according to the current env vars.
  2. We generate static deploy files with default values and keep them in the repo, PR checks that they are up to date.
  3. Makefile invokes the script from 1, but result is stored in another not commited place.

@amisevsk WDYT?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense; my main concern is the levels of indirection (pkg/api -> CRDs -> deployable yamls) but that can be addressed with PR checks I suppose. I'll close this one and port the changes over to a new one that includes the full yamls.

mv config/service-ca/kustomization.yaml config/service-ca/kustomization.yaml.bak
mv config/base/config.properties config/base/config.properties.bak
mv config/base/manager_image_patch.yaml config/base/manager_image_patch.yaml.bak
envsubst < config/cert-manager/kustomization.yaml.bak > config/cert-manager/kustomization.yaml
envsubst < config/service-ca/kustomization.yaml.bak > config/service-ca/kustomization.yaml
envsubst < config/base/config.properties.bak > config/base/config.properties
envsubst < config/base/manager_image_patch.yaml.bak > config/base/manager_image_patch.yaml

# Generate yaml files
mkdir -p "${OUTPUT_DIR}"/{kubernetes,openshift}
echo "===== Building yaml templates for Kubernetes ====="
kustomize build config/cert-manager > "${OUTPUT_DIR}/kubernetes/combined.yaml"
echo "===== Generated yaml templates for Kubernetes ====="
echo "===== Building yaml templates for OpenShift ====="
kustomize build config/service-ca > "${OUTPUT_DIR}/openshift/combined.yaml"
echo "===== Generated yaml templates for OpenShift ====="

# Take giant yaml file output by kustomize and separate it into a file per k8s object
for dir in kubernetes openshift; do
echo "===== Parsing files from ${OUTPUT_DIR}/${dir}/combined.yaml ====="
pushd "${OUTPUT_DIR}/${dir}" &>/dev/null
mkdir -p objects

# Split combined.yaml into separate files for each record, with names temp01, temp02, etc.
# Then rename each temp file according to the .metadata.name and .kind of the object
csplit -s -f "temp" --suppress-matched "combined.yaml" '/^---$/' '{*}'
for file in temp*; do
name_kind=$(yq -r '"\(.metadata.name).\(.kind)"' "$file")
mv "$file" "objects/${name_kind}.yaml"
done

popd &>/dev/null
done

# Compress files into a tarball
cd "$BASE_DIR"
tar -czvf "$TARBALL_PATH" deploy
36 changes: 36 additions & 0 deletions build/yaml-templates/yaml-builder.Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
FROM alpine AS builder

ARG DEVWORKSPACE_BRANCH=master
ARG NAMESPACE=devworkspace-controller
ARG IMG=quay.io/devfile/devworkspace-controller:next
ARG PULL_POLICY=Always
ARG DEFAULT_ROUTING=basic
ARG DEVWORKSPACE_API_VERSION=aeda60d4361911da85103f224644bfa792498499

WORKDIR /build

RUN apk add --no-cache \
bash \
coreutils \
curl \
gettext \
git \
jq \
python3 \
py-pip \
tar \
&& pip install yq \
&& curl -sL "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv3.9.2/kustomize_v3.9.2_linux_amd64.tar.gz" -o kustomize.tar.gz \
&& tar -xvf kustomize.tar.gz --directory /usr/bin

COPY ["prepare_templates.sh", "/"]
RUN /prepare_templates.sh

FROM alpine

COPY --from=builder "/build/devworkspace_operator_templates.tar.gz" /devworkspace_operator_templates.tar.gz
CMD echo "To extract yaml files from this repo, run:" &&\
echo " docker create --name builder <this-image>" &&\
echo " docker cp builder:/devworkspace_operator_templates.tar.gz ./devworkspace_operator_templates.tar.gz" &&\
echo " docker rm builder" &&\
echo " tar -xzf devworkspace_operator_templates.tar.gz"
2 changes: 1 addition & 1 deletion config/base/config.properties
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
devworkspace.routing.cluster_host_suffix=${ROUTING_SUFFIX}
controller.plugin_registry.url=${PLUGIN_REGISTRY_URL}
controller.plugin_artifacts_broker.image=quay.io/eclipse/che-plugin-artifacts-broker:v3.4.0
controller.webhooks.enabled=${WEBHOOK_ENABLED}
controller.webhooks.enabled=true
devworkspace.default_routing_class=${DEFAULT_ROUTING}
# image pull policy that is applied to all workspace's containers
devworkspace.sidecar.image_pull_policy=${PULL_POLICY}
Expand Down