Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
docs: fixes of links, headers and slightly of texts
  • Loading branch information
diafour committed May 21, 2019
1 parent 14c7c56 commit 01279e8
Show file tree
Hide file tree
Showing 5 changed files with 90 additions and 95 deletions.
41 changes: 21 additions & 20 deletions HOOKS.md
Expand Up @@ -2,23 +2,23 @@

A hook is an executable file that Addon-operator executes when some event occurs. It can be a script or a compiled program written in any programming language.

Addon-operator pursues an agreement stating that the information is transferred to hooks via files and results of hooks execution are also stored in files. Paths to files are passed via environment variables. The output to stdout will be written to the log, except for the case with the configuration output (--config). Such an agreement simplifies the work with the input data and reporting the results of the hook execution.
Addon-operator pursues an agreement stating that the information is transferred to hooks via files and results of hooks execution are also stored in files. Paths to files are passed via environment variables. The output to stdout will be written to the log, except for the case with the configuration output (run with `--config` flag). Such an agreement simplifies the work with the input data and reporting the results of the hook execution.

# Initialization of global hooks

Global hooks can read and modify values in the global values storage which is available to all modules (see [VALUES](VALUES.md)). The global hook performs actions and discovers the parameters required by several modules.
Global hooks can read and modify values in the global values storage which is available to all modules (see [VALUES](VALUES.md)). The global hook performs actions and discovers the values required by several modules.

Global hooks are stored in the $GLOBAL_HOOKS_DIR directory. The Addon-operator recursively searches all executable files in it and runs them with the `--config` flag. Each hook prints its events binding configuration in the JSON format to stdout. If the execution fails, the Addon-operator terminates with the code of 1.

Bindings from [shell-operator](https://github.com/flant/shell-operator) are available for global hooks: [onStartup](# onstartup), [schedule](# schedule) and [onKubernetesEvent](# onkubernetesevent). The bindings of the modules discovery process are also available: [beforeAll](# beforeall) and [afterAll](# afterall) - see [modules discovery](LIFECYCLE.md#modules-discovery).
Bindings from [shell-operator](https://github.com/flant/shell-operator) are available for global hooks: [onStartup](#onstartup), [schedule](#schedule) and [onKubernetesEvent](#onkubernetesevent). The bindings to the events of the modules discovery process are also available: [beforeAll](#beforeall) and [afterAll](#afterall) (see [modules discovery](LIFECYCLE.md#modules-discovery)).

# Initialization of module hooks

Module hooks can get values from the global values storage. Also, they can read and modify values in the module values storage. For details on values storage, see [VALUES](VALUES.md)). Hooks are initialized for all enabled modules in the process of [modules discovery](LIFECYCLE.md#modules-discovery).
Module hooks can get values from the global values storage. Also, they can read and modify values in the module values storage. For details on values storage, see [VALUES](VALUES.md). Hooks are initialized for all enabled modules within the process of [modules discovery](LIFECYCLE.md#modules-discovery).

Module hooks are executable files stored in the `hooks` subdirectory. During the module discovery process Addon-operator searhes for executable files in this directory and all found files are executed with `--config` flag. Each hooks should print to stdout its event binding configuration in JSON format. The module discovery process restarts if an error occurs.
Module hooks are executable files stored in the `hooks` subdirectory of module. During the modules discovery process Addon-operator searches for executable files in this directory and all found files are executed with `--config` flag. Each hook prints its event binding configuration in JSON format to stdout. The module discovery process restarts if an error occurs.

Bindings from [shell-operator](https://github.com/flant/shell-operator) are available for module hooks: [schedule](#schedule) and [onKubernetesEvent](#onkubernetesevent). The bindings of the module lifecycle are also available: `onStartup`, `beforeHelm`, `afterHelm`, `afterDeleteHelm` — see [Module lifecycle](LIFECYCLE.md#module-lifecycle).
Bindings from [shell-operator](https://github.com/flant/shell-operator) are available for module hooks: [schedule](#schedule) and [onKubernetesEvent](#onkubernetesevent). The bindings of the module lifecycle are also available: `onStartup`, `beforeHelm`, `afterHelm`, `afterDeleteHelm` — see [module lifecycle](LIFECYCLE.md#module-lifecycle).

# Bindings

Expand Down Expand Up @@ -70,7 +70,7 @@ Parameters:

## afterAll

The execution of global hooks right after running and removing modules (after modules discovery process).
The execution of global hooks after running and removing modules.

Syntax:

Expand Down Expand Up @@ -126,44 +126,45 @@ Parameters:

## schedule

[schedule binding](https://github.com/flant/shell-operator/blob/master/HOOKS.md#schedule)
[schedule binding](https://github.com/flant/shell-operator/blob/master/HOOKS.md#schedule).

## onKubernetesEvent

[onKubernetesEvent binding](https://github.com/flant/shell-operator/blob/master/HOOKS.md#onKubernetesEvent)

> Note: Addon-operator requires a ServiceAccount with the appropriate [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) permissions. See examples with RBAC: [monitor-pods](examples/101-monitor-pods) and [monitor-namespaces](examples/102-monitor-namespaces).
> Note: Addon-operator requires a ServiceAccount with the appropriate [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) permissions. See `addon-operator-rbac.yaml` files in [examples](/examples).
# Execution on event

When an event associated with a hook is triggered, Addon-operator executes the hook without arguments and passes the module parameters from the values storage via temporary files. In response, a hook could return JSON patches to modify module parameters. The detailed description of the values storage is available at [VALUES](VALUES.md).
When an event associated with a hook is triggered, Addon-operator executes the hook without arguments and passes the global or module values from the values storage via temporary files. In response, a hook could return JSON patches to modify values. The detailed description of the values storage is available at [VALUES](VALUES.md).

## Binding context

Binding context is information about the event which caused the hook execution.

The $BINDING_CONTEXT_PATH environment variable contains the path to a file with JSON array of structures with the following fields:

- `binding` is a string from the `name` parameter for `schecdule` or `onKubernetesEvent` or a binding type if parameter is not set for other hooks. For example, file content for `beforeAll` hook:
```
{“binding”:”beforeAll”}
- `binding` is a string from the `name` parameter for `schecdule` or `onKubernetesEvent` or a binding type if parameter is not set and for other hooks. For example, binding context for `beforeAll` hook:

```json
[{"binding":"beforeAll"}]
```

There are some extra fields for `onKubernetesEvent` hooks:

- `resourceEvent` — the event type is identical to the values in the `event` parameter: add”, “update or delete.
- `resourceEvent` — the event type is identical to the values in the `event` parameter: "add", "update" or "delete".
- `resourceNamespace`, `resourceKind`, `resourceName` — the information about the Kubernetes object associated with an event.

For example, if you have the following binding configuration of a hook:

```json
{
 "schedule": [
   {
     "name": "incremental",
     "crontab": "0 2 */3 * * *",
     "allowFailure": true
   } ]
"schedule": [
{
"name": "incremental",
"crontab": "0 2 */3 * * *",
"allowFailure": true
}]
}
```

Expand Down
23 changes: 12 additions & 11 deletions LIFECYCLE.md
@@ -1,10 +1,10 @@
# Structure

The modules files are located in the `/modules` directory. The directory can be set via $MODULES_DIR variable. The global hooks files are located in the `/global-hooks` directory (you can set your own directory with the $GLOBAL_HOOKS_DIR variable).
The module files are located in the `/modules` directory. The directory can be set via $MODULES_DIR variable. The global hook files are located in the `/global-hooks` directory (you can set your own directory with the $GLOBAL_HOOKS_DIR variable).

# Initialization

During the start, Addon-operator finds and initializes all global hooks. For more info, see [HOOKS](HOOKS.md#global-hooks).
During the start, Addon-operator finds and initializes all global hooks. For more info, see [HOOKS](HOOKS.md#initialization-of-global-hooks).

Addon-operator installs a separate instance of tiller in the namespace where the addon-operator is running to store the information about helm-releases of modules.

Expand All @@ -16,7 +16,7 @@ The modules discovery process finds all enabled modules and starts them.

During start-up, the module executes its `onStartup` hook and initializes the installation of a helm chart. Prior to the installation of a helm chart, the `beforeHelm` hook is executed. The `afterHelm` hook is executed after the installation.

Then the main cycle starts. It processes scheduled and Kubernetes objects’ events, as well as a module activation event when parameters change, or the modules restart event.
Then the main loop starts. It processes scheduled and Kubernetes objects’ events, as well as a module activation event when values change and the modules restart event.

# Module lifecycle

Expand All @@ -27,11 +27,11 @@ Next, the modules are run in alphabetical order with `helm upgrade --install`. P
After the launch module would start to respond to two types of events:

- `schedule` — events that are generated by the crontab scheduler built in the addon-operator;
- `onKubernetesEvent` — events within the cluster that apiserver announces to Addon-operator.
- `onKubernetesEvent` — events within the cluster that api-server announces to Addon-operator.

When module is deactivated, Addon-operator launch command `helm delete --purge` and after the release deletion, the `afterDeleteHelm` hooks are executed.

All necessary hooks will be restarted if there are errors during the module activation or deactivation. For example, if an error occurred in the hook with `afterHelm` binding during the first module execution, then after a 5 seconds delay the `onStartup`, `beforeHelm` hooks are restarted, the helm chart is installed and `afterHelm` hooks are executed.
All necessary hooks will be restarted if there are errors during the module activation or deactivation. For example, if an error occurred in the hook with `afterHelm` binding during the first module execution, then after a 5 seconds delay the `onStartup` and `beforeHelm` hooks are executed, the helm chart is installed and then `afterHelm` hooks are executed.

# Modules discovery

Expand All @@ -47,29 +47,30 @@ For each module in the values.yaml and in the ConfigMap/addon-operator there is

The key may be absent. In this case, the `true` value is considered.

If combined values from values.yaml and ConfigMap/addon-operator equal to false, then the module is disabled.
If combined values from values.yaml and ConfigMap/addon-operator equal to `false`, then the module is disabled.

If the value is not false, the additional check is conducted – the `enabled` script is executed (see below). If there is a script and it returns false, then the module is considered disabled.
If the value is not `false`, the additional check is conducted – the `enabled` script is executed (see below). If there is a script and it returns `false`, then the module is considered disabled.

If an error occurs during the modules discovery process, then the module discovery is restarted every 5 seconds until successful execution. In this case, the execution of hooks with `schedule` and `onKubernetesEvent` bindings will be blocked.

As a result of a module discovery process the tasks for the execution of all *enabled* modules, deletion of all *disabled* modules, and execution of all global hook with the `afterAll` binding are added to the queue.

## Enabled script

A script or an executable file that returns status of the module. The script has access to the module parameters in $VALUES_PATH и $CONFIG_VALUES_PATH files, similar to the module [hook](HOOKS.md#module-hook), more details about the parameters are available in [VALUES](VALUES.md#working-with-values-in-enabled-script). The variable $MODULE_ENABLED_RESULT passes the path to the file into which the script should save the module status: true or false.
A script or an executable file that returns a status of the module. The script has access to the module values in $VALUES_PATH and $CONFIG_VALUES_PATH files, more details about the values are available in [VALUES](VALUES.md#using-values-in-enabled-script). The variable $MODULE_ENABLED_RESULT passes the path to the file into which the script should write the module status: true or false.

Below is an example of the enabled script that disables the module when parameter `param2` set to "stopMePlease".
Below is an example of the `enabled` script that disables the module when parameter `param2` is set to "stopMePlease".


```bash
#!/usr/bin/env bash

param2=$(jq -r '.simpleModule.param2' $VALUES_PATH)

if [[ $param2 == "stopMePlease" ]] ; then
 echo "false" > $MODULE_ENABLED_RESULT
echo "false" > $MODULE_ENABLED_RESULT
else
 echo "true" > $MODULE_ENABLED_RESULT
echo "true" > $MODULE_ENABLED_RESULT
fi

```
Expand Down
21 changes: 10 additions & 11 deletions METRICS.md
Expand Up @@ -4,7 +4,7 @@ Addon-operator implements Prometheus target at `/metrics` endpoint. Default port

__addon_operator_module_hook_errors{module=”module-name”, hook="hook-name"}__

The counter of hooks’ execution errors. It only tracks errors of hooks with the disabled allowFailure (allowFailure: false).
The counter of hooks’ execution errors. It only tracks errors of hooks with the disabled `allowFailure` (allowFailure: false).


__addon_operator_module_hook_allowed_errors{module=”module-name”, hook="hook-name"}__
Expand All @@ -22,16 +22,6 @@ __addon_operator_global_hook_allowed_errors{hook="hook-name"}__
The counter of execution errors of global hooks for which execution errors are allowed (allowFailure: true).


__addon_operator_tasks_queue_length__

An indicator of a working queue length. This metric can be used to warn about stuck hooks. It has no labels.


__addon_operator_live_ticks__

A counter that increases every 10 seconds.


__addon_operator_module_discover_errors__
A counter of errors during the [modules discover](LIFECYCLE.md#modules-discover) process. It increases every time when there are errors in the enabled-scripts running, configuration of module hooks, errors when viewing the helm releases or accessing the K8s API.

Expand All @@ -42,3 +32,12 @@ Counter of error on module [start-up](LIFECYCLE.md#modules-lifecycle).
__addon_operator_module_delete_errors{module=x}__
Counter of errors on module [deletion](LIFECYCLE.md#modules-lifecycle).


__addon_operator_tasks_queue_length__

An indicator of a working queue length. This metric can be used to warn about stuck hooks. It has no labels.

__addon_operator_live_ticks__

A counter that increases every 10 seconds.

15 changes: 7 additions & 8 deletions MODULES.md
@@ -1,10 +1,7 @@
# Modules

# Module scructure

A module is a directory with files. Addon-operator searches for the modules directories in `/modules` or in the path specified by $MODULES_DIR variable. The module has the same name as the corresponding directory excluding the numeric prefix.


The file structure of the module’s directory:

```
Expand All @@ -26,27 +23,29 @@ The file structure of the module’s directory:
- `README.md` — module description;
- `values.yaml` – default values for chart in a [special format](VALUES.md).

Module name in this case is `simple-module`.

# Notes on how Helm is used

## values.yaml

Addon-operator does not use values.yaml as the only source of values for the chart. It generates a new file with a combined set of parameters (also mixing values from this file (see [VALUES](VALUES.md)).
Addon-operator does not use values.yaml as the only source of values for the chart. It generates a new file with a merged set of values (also mixing values from this file (see [VALUES](VALUES.md#merged-values)).

## Chart.yaml

We recommend to define the `version` field in your chart.yaml as "0.0.1" and use VCS to control versions. We also recommend to explicitly specify the `name` field even despite it is ignored: Addon-operator passes the module name to the Helm as a release name.
We recommend to define the `version` field in your Chart.yaml as "0.0.1" and use VCS to control versions. We also recommend to explicitly specify the `name` field even despite it is ignored: Addon-operator passes the module name to the Helm as a release name.

## Releases deduplication

A module’s execution might be triggered by an event that does not change the parameters of the module (see [modules discovery](LIFECYCLE.md#modules-discovery)). Re-running Helm will lead to an "empty" release. To avoid this, Addon-operator compares parameters checksums and starts the installation of a Helm chart only if there are some changes.
A module’s execution might be triggered by an event that does not change the parameters of the module (see [modules discovery](LIFECYCLE.md#modules-discovery)). Re-running Helm will lead to an "empty" release. To avoid this, Addon-operator compares values’ checksums and starts the installation of a Helm chart only if there are some changes.

# Workarounds for Helm issues

Helm badly handles failed chart installations [PR#4871](https://github.com/helm/helm/pull/4871). A workaround has been added to Addon-operator to reduce the number of manual interventions in such situations: automatic deletion of the single failed release. In the future, in addition to this mechanism, we plan to add a few improvements to the interaction with Helm. In particular, we plan to port related algorithms (how the interaction with Helm is done) from werf — [ROADMAP](https://github.com/flant/addon-operator/issues/17).
Helm badly handles failed chart installations ([PR#4871](https://github.com/helm/helm/pull/4871)). A workaround has been added to Addon-operator to reduce the number of manual interventions in such situations: automatic deletion of the single failed release. In the future, in addition to this mechanism, we plan to add a few improvements to the interaction with Helm. In particular, we plan to port related algorithms (how the interaction with Helm is done) from werf — [ROADMAP](https://github.com/flant/addon-operator/issues/17).

# Next

- [addon-operator lifecycle](LIFECYCLE.md)
- addon-operator [lifecycle](LIFECYCLE.md)
- [values](VALUES.md)
- [hooks](HOOKS.md)

0 comments on commit 01279e8

Please sign in to comment.