Tiller sidecar (#48)

* feat: run tiller as a subprocess, exit on tiller error.
* examples: add ADDON_OPERATOR_NAMESPACE variable, use alpine image, add Enabled keys
* fix: run onStartup hooks for newly enabled modules

Dockerfile

@@ -11,8 +11,8 @@ RUN apt-get update && \
     rm -rf /var/lib/apt/lists && \
     wget https://storage.googleapis.com/kubernetes-release/release/v1.13.5/bin/linux/amd64/kubectl -O /bin/kubectl && \
     chmod +x /bin/kubectl && \
-    wget https://storage.googleapis.com/kubernetes-helm/helm-v2.12.1-linux-amd64.tar.gz -O /helm.tgz && \
-    tar -z -x -C /bin -f /helm.tgz --strip-components=1 linux-amd64/helm && \
+    wget https://storage.googleapis.com/kubernetes-helm/helm-v2.14.3-linux-amd64.tar.gz -O /helm.tgz && \
+    tar -z -x -C /bin -f /helm.tgz --strip-components=1 linux-amd64/helm linux-amd64/tiller && \
     rm -f /helm.tgz && \
     helm init --client-only && \
     mkdir /hooks

Dockerfile-alpine3.9

@@ -10,7 +10,7 @@ RUN apk --no-cache add ca-certificates jq bash && \
     wget https://storage.googleapis.com/kubernetes-release/release/v1.13.5/bin/linux/amd64/kubectl -O /bin/kubectl && \
     chmod +x /bin/kubectl && \
     wget https://storage.googleapis.com/kubernetes-helm/helm-v2.12.1-linux-amd64.tar.gz -O /helm.tgz && \
-    tar -z -x -C /bin -f /helm.tgz --strip-components=1 linux-amd64/helm && \
+    tar -z -x -C /bin -f /helm.tgz --strip-components=1 linux-amd64/helm linux-amd64/tiller && \
     rm -f /helm.tgz && \
     helm init --client-only && \
     mkdir /hooks

HOOKS.md

@@ -126,11 +126,11 @@ Parameters:
 
 ## schedule
 
-[schedule binding](https://github.com/flant/shell-operator/blob/master/HOOKS.md#schedule).
+[schedule binding](https://github.com/flant/shell-operator/blob/v1.0.0-beta.5/HOOKS.md#schedule).
 
 ## onKubernetesEvent
 
-[onKubernetesEvent binding](https://github.com/flant/shell-operator/blob/master/HOOKS.md#onKubernetesEvent)
+[onKubernetesEvent binding](https://github.com/flant/shell-operator/blob/v1.0.0-beta.5/HOOKS.md#onKubernetesEvent)
 
 > Note: Addon-operator requires a ServiceAccount with the appropriate [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) permissions. See `addon-operator-rbac.yaml` files in [examples](/examples).
 

LIFECYCLE.md

@@ -6,8 +6,6 @@ The module files are located in the `/modules` directory. The directory can be s
 
 During the start, Addon-operator finds and initializes all global hooks. For more info, see [HOOKS](HOOKS.md#initialization-of-global-hooks).
 
-Addon-operator installs a separate instance of tiller in the namespace where the addon-operator is running to store the information about helm-releases of modules.
-
 # Main loop
 
 After initialization Addon-operator executes all global `onStartup` hooks and starts the modules discovery process.
@@ -54,22 +52,6 @@ If an error occurs during the modules discovery process, then the module discove
 
 As a result of a module discovery process the tasks for the execution of all *enabled* modules, deletion of all *disabled* modules, and execution of all global hook with the `afterAll` binding are added to the queue.
 
-### Module enablement examples
-
-### 1
-
-modules/values.yaml
-```
-moduleNameEnabled: true
-```
-
-modules/001-moduleName/values.yaml
-```
-moduleNameEnabled: false
-```
-
-Module is disabled
-
 ## Enabled script
 
 A script or an executable file that returns a status of the module. The script has access to the module values in $VALUES_PATH and $CONFIG_VALUES_PATH files, more details about the values are available in [VALUES](VALUES.md#using-values-in-enabled-script). The variable $MODULE_ENABLED_RESULT passes the path to the file into which the script should write the module status: true or false.
@@ -90,18 +72,38 @@ fi
 
 ```
 
-## Example
+## Examples
+
+### keys in values.yaml files
+
+Let’s assume the module named `nginx-ingress` and the following values are defined:
+
+```
+$ cat modules/values.yaml
+
+nginxIngressEnabled: true
+
+$ cat modules/001-nginx-ingress/values.yaml
+
+nginxIngressEnabled: false
+```
+
+Module `nginx-ingress` is disabled in `modules/values.yaml` but enabled in `modules/001-nginx-ingress/values.yaml`. The final result is that the module is disabled.
+
+Also note that module's directory name is kebab-cased but keys in values.yaml are camelCased (see [VALUES](VALUES.md#values-storage)).
+
+### values.yaml and ConfigMap 
 
 Let’s assume the following values and enabled script are defined:
 
 ```
-$ cat modules/values.yaml:
+$ cat modules/values.yaml
 
 global:
   param1: 100
 someModuleEnabled: false
 
-$ cat modules/01-some-module/values.yaml
+$ cat modules/001-some-module/values.yaml
 
 someModule:
   param1: "String"
@@ -124,7 +126,7 @@ $ cat modules/01-some-module/enabled
 echo false > $MODULE_ENABLED_RESULT
 ```
 
-Module some-module is explicitly disabled in modules/values.yaml and enabled by `someModuleEnabled` key in ConfigMap/addon-operator. So enabled script is executed. Script returns `false` and the final result is that the module is disabled.
+Module `some-module` is explicitly disabled in `modules/values.yaml` but enabled by `someModuleEnabled` key in ConfigMap/addon-operator. Thus enabled script is executed and returns `false`. So the final result is that the module is disabled.
 
 # Tasks queue
 
@@ -140,12 +142,10 @@ Several tools are available for the debugging of addon-operator and hooks:
 
 - You can get logs of an Addon-operator’s pod for analysis (by executing `kubectl logs -f po/POD_NAME`)
 - You can set the environment variable RLOG_LOG_LEVEL=DEBUG to include the detailed debugging data into logs
-- You can run the Addon-operator with the `--debug=yes` argument to obtain even more detailed debugging information and save it to logs
 - You can view the contents of the working queue via the HTTP request of `/queue` endpoint:
 
 ```bash
 kubectl port-forward po/addon-operator 9115:9115
 
 curl localhost:9115/queue
 ```
-

METRICS.md

@@ -1,6 +1,6 @@
 # Addon-operator metrics
 
-Addon-operator implements Prometheus target at `/metrics` endpoint. Default port is 9115.
+Addon-operator implements Prometheus target at `/metrics` endpoint. Default port is `9650`.
 
 __addon_operator_module_hook_errors{module=”module-name”, hook="hook-name"}__
 

MODULES.md

@@ -39,10 +39,14 @@ We recommend to define the `version` field in your Chart.yaml as "0.0.1" and use
 
 A module’s execution might be triggered by an event that does not change the parameters of the module (see [modules discovery](LIFECYCLE.md#modules-discovery)). Re-running Helm will lead to an "empty" release. To avoid this, Addon-operator compares values’ checksums and starts the installation of a Helm chart only if there are some changes.
 
-# Workarounds for Helm issues
+## Workarounds for Helm issues
 
 Helm badly handles failed chart installations ([PR#4871](https://github.com/helm/helm/pull/4871)). A workaround has been added to Addon-operator to reduce the number of manual interventions in such situations: automatic deletion of the single failed release. In the future, in addition to this mechanism, we plan to add a few improvements to the interaction with Helm. In particular, we plan to port related algorithms (how the interaction with Helm is done) from werf — [ROADMAP](https://github.com/flant/addon-operator/issues/17).
 
+## Tiller
+
+Tiller is started as subprocess. It listens on 127.0.0.1 and use two ports: one for gRPC connectivity with helm and one for cluster probes. These settings can be changed with environment variables (See [RUNNING](RUNNING.md)). If Tiller process suddenly exits, Addon-operator process also exits.
+
 # Next
 
 - addon-operator [lifecycle](LIFECYCLE.md)

README.md

@@ -9,7 +9,7 @@
 </p>
 
 
-The **Addon-operator** adds hooks and values to helm charts in order to enhance the capabilities of helm and transform charts into smart modules that configure themselves and respond to changes in the cluster.
+The **Addon-operator** combines helm charts with hooks and values storage to transform charts into smart modules that configure themselves and respond to changes in the cluster.
 
 # Features
 
@@ -25,7 +25,7 @@ Also, Addon-operator provides:
 - the possibility of *dynamic enabling/disabling* of a module (depending on detected parameters);
 - the ability to tie *conditions of module activation* to the activation of other modules;
 - *the unified ConfigMap* for the configuration of all settings;
-- the ability to run helm only if the parameters have changed. In this case, the release list would contain the time of build of the modified release;
+- the ability to run helm only if parameters have changed. In this case, `helm history` would output only releases with changes;
 - *global hooks* for figuring out parameters and performing actions that affect several dependent modules;
 - off-the-shelf *metrics* for monitoring via Prometheus.
 
@@ -42,7 +42,7 @@ A hook is an executable file that can make changes to Kubernetes and set values
 
 ![A hook is an executable file](docs/readme-2.gif)
 
-Hooks are a part of the module. Also, there is a helm chart in the module. If the hook makes changes to values, then Addon-operator would start the reinstallation of the helm chart.
+Hooks are a part of the module. Also, there is a helm chart in the module. If the hook makes changes to values, then Addon-operator would upgrade the release of the helm chart.
 
 ![Hook is a part of the module](docs/readme-3.gif)
 
@@ -54,27 +54,29 @@ There can be many modules.
 
 In addition to modules, the Addon-operator supports global hooks and global values. They have their own storage of values. Global hooks are triggered by events and when active they can:
 
-- Make changes to Kubernetes
+- Make changes to Kubernetes cluster
 - Make changes to global values storage
 
 ![Global hooks and global values](docs/readme-5.gif)
 
-If the global hook changes values in the global storage, then the Addon-operator starts the reinstallation of all helm charts.
+If the global hook changes values in the global storage, then the Addon-operator triggers upgrade of releases of all helm charts.
 
 ![Changes in global values cause reinstallation](docs/readme-6.gif)
 
 
 # Installation
 
-You may use the prepared image [flant/addon-operator](https://hub.docker.com/r/flant/addon-operator) to install Addon-operator in a cluster. The image comprises a binary addon-operator file as well as several required tools: helm, kubectl, jq, bash.
+You may use the prepared image [flant/addon-operator](https://hub.docker.com/r/flant/addon-operator) to install Addon-operator in a cluster. The image comprises a binary addon-operator file as well as several required tools: helm, tiller, kubectl, jq, bash.
 
-The installation incorporates the image building process with *files of modules and hooks*, adding the necessary RBAC rights and launching image in the cluster. You may find a preshaped files and commands in the /examples directory.
+The installation incorporates the image building process with *files of modules and hooks*, applying the necessary RBAC rights and deploying the image in the cluster.
 
+To experiment with modules, hooks and values we are prepared some [examples](/examples).
 
 # What's next?
-- Find out more on [lifecycle](LIFECYCLE.md) of Addon-operator and how to use [modules](MODULES.md) and [values](VALUES.md) in documentation.
+- Find out more on [lifecycle](LIFECYCLE.md) of Addon-operator and how to use [modules](MODULES.md), [hooks](HOOKS.md) and [values](VALUES.md).
 - `/metrics` endpoint is implemented. See [METRICS](METRICS.md) for details.
-- More examples can be found in [examples](/examples/) directory.
+- Explore Shell-operator documentation, especialy [hooks](https://github.com/flant/shell-operator/blob/v1.0.0-beta.5/HOOKS.md) section.
+- See how to tune [deploy settings](RUNNING.md).
 
 ## License
 

RUNNING.md

@@ -0,0 +1,68 @@
+# Running Addon-operator
+
+## Environment variables
+
+**ADDON_OPERATOR_NAMESPACE** — a required parameter with namespace where Addon-operator is deployed.
+
+**ADDON_OPERATOR_CONFIG_MAP** — a name of ConfigMap to store values. Default is `addon-operator`.
+
+Namespace and config map name are used to watch for ConfigMap changes. 
+
+Example of container:
+
+```
+containers:
+- image: addon-operator-image:latest
+  env:
+  - name: ADDON_OPERATOR_NAMESPACE
+    valueFrom:
+      fieldRef:
+        fieldPath: metadata.namespace
+  - name: ADDON_OPERATOR_CONFIG_MAP
+    value: my-values   
+```
+
+With this variables Addon-operator would monitor ConfigMap/my-values object. 
+
+**ADDON_OPERATOR_LISTEN_ADDRESS** — address for http server. Default is `0.0.0.0`
+
+**ADDON_OPERATOR_LISTEN_PORT** — port for http server. Default is `9650`.
+
+Addon-operator starts http server and listens on `ADDRESS:PORT`. There is a liveness probe and `/metrics` endpoint.
+
+```
+  env:
+  ...
+  - name: ADDON_OPERATOR_LISTEN_ADDRESS
+    valueFrom:
+      fieldRef:
+        fieldPath: status.podIP
+  - name: ADDON_OPERATOR_LISTEN_PORT
+    value: 9090
+  livenessProbe:
+    httpGet:
+      path: /healthz
+      port: 9090      
+``` 
+
+**ADDON_OPERATOR_PROMETHEUS_METRICS_PREFIX** — a prefix for Prometheus metrics. Default is `addon_operator_`.
+
+```
+  env
+  - name: ADDON_OPERATOR_PROMETHEUS_METRICS_PREFIX
+    value: dev_cluster_  
+```
+
+```
+curl localhost:9650/metrics
+
+...
+dev_cluster_live_ticks 32
+...
+```
+
+
+**ADDON_OPERATOR_TILLER_LISTEN_PORT** — a port used for communication with helm (-listen flag). Default is 44435.
+**ADDON_OPERATOR_TILLER_PROBE_LISTEN_PORT** — a port used for Tiller probes (-probe-listen flag). Default is 44434.
+
+Tiller starts as a subprocess and listens on 127.0.01 address. Defaults are good, but if Addon-operator should start with `hostNetwork: true`, then these variables will come in handy.

cmd/addon-operator/main.go

@@ -5,23 +5,18 @@ import (
 	"fmt"
 	"os"
 
-	"github.com/romana/rlog"
+	"github.com/flant/shell-operator/pkg/executor"
 	"gopkg.in/alecthomas/kingpin.v2"
 
-	shell_operator_app "github.com/flant/shell-operator/pkg/app"
-	"github.com/flant/shell-operator/pkg/executor"
 	utils_signal "github.com/flant/shell-operator/pkg/utils/signal"
 
 	operator "github.com/flant/addon-operator/pkg/addon-operator"
 	"github.com/flant/addon-operator/pkg/app"
-
 )
 
 func main() {
-
 	kpApp := kingpin.New(app.AppName, fmt.Sprintf("%s %s: %s", app.AppName, app.Version, app.AppDescription))
 
-
 	// global defaults
 	app.SetupGlobalSettings(kpApp)
 
@@ -42,19 +37,11 @@ func main() {
 			// in case if addon-operator is a PID 1 process.
 			go executor.Reap()
 
-			operator.InitHttpServer(app.ListenAddress)
-
-			rlog.Infof("addon-operator %s, shell-operator %s", app.Version, shell_operator_app.Version)
-
-			err := operator.Init()
-			if err != nil {
-				os.Exit(1)
-			}
-
-			operator.Run()
+			operator.Start()
 
 			// Block action by waiting signals from OS.
 			utils_signal.WaitForProcessInterruption()
+
 			return nil
 		})
 

examples/001-startup-global/Dockerfile

@@ -1,3 +1,3 @@
-FROM flant/addon-operator:latest
+FROM flant/addon-operator:latest-alpine3.9
 ADD modules /modules
-ADD global-hooks /global-hooks
+ADD global-hooks /global-hooks

examples/001-startup-global/README.md

@@ -7,37 +7,37 @@ Example of a global hook written as bash script.
 Build addon-operator image with custom scripts:
 
 ```
-$ docker build -t "registry.mycompany.com/addon-operator:startup-global" .
-$ docker push registry.mycompany.com/addon-operator:startup-global
+docker build -t "registry.mycompany.com/addon-operator:startup-global" .
+docker push registry.mycompany.com/addon-operator:startup-global
 ```
 
 Edit image in addon-operator-pod.yaml and apply manifests:
 
 ```
-$ kubectl create ns example-startup-global
-$ kubectl -n example-startup-global apply -f addon-operator-rbac.yaml
-$ kubectl -n example-startup-global apply -f addon-operator-pod.yaml
+kubectl create ns example-startup-global
+kubectl -n example-startup-global apply -f addon-operator-rbac.yaml
+kubectl -n example-startup-global apply -f addon-operator-pod.yaml
 ```
 
-See in logs that hook.sh was run:
+See in logs that hook.sh was run at startup:
 
 ```
-$ kubectl -n example-startup-global logs -f po/addon-operator
+kubectl -n example-startup-global logs pod/addon-operator -f
 ...
 INFO     : Initializing global hooks ...
-INFO     : Initializing global hook 'global-hooks/hook.sh' ...
+INFO     : INIT: global hook 'hook.sh' ...
 ...
-INFO     : TASK_RUN GlobalHookRun@ON_STARTUP global-hooks/hook.sh
-INFO     : Running global hook 'global-hooks/hook.sh' binding 'ON_STARTUP' ...
+INFO     : TASK_RUN GlobalHookRun@ON_STARTUP hook.sh
+INFO     : Running global hook 'hook.sh' binding 'ON_STARTUP' ...
 OnStartup global hook
 ...
 ```
 
 ### cleanup
 
 ```
-$ kubectl delete clusterrolebinding/addon-operator
-$ kubectl delete clusterrole/addon-operator
-$ kubectl delete ns/example-startup-global
-$ docker rmi registry.mycompany.com/addon-operator:startup-global
+kubectl delete clusterrolebinding/addon-operator
+kubectl delete clusterrole/addon-operator
+kubectl delete ns/example-startup-global
+docker rmi registry.mycompany.com/addon-operator:startup-global
 ```