[meta] Workflow Charts for Kubernetes Helm

[slack]

This is a meta issue to organize work for transitioning Workflow to use kubernetes/helm. We've been doing some exploratory work and have some early successes. Wanted to consolidate our thinking here!

High-level goals for the transition. How do operators of Workflow use Helm to:

1. deploy a configured workflow install
2. change configuration of a release (Helm release) e.g. set REGISTRATION_MODE
3. cleanly and easily upgrade with custom configuration in place
4. customize/tweak components: custom images, custom configuration

I'm using configuration to mean turning a known knob or dial for one of Workflow's components. For example: wiring up off-cluster registry, object storage settings, setting cpu/memory limits, etc.

Customization represents a deeper change to the instance of Workflow, including adding additional resources, etc. Deep customization is most likely out of scope for now, but it is useful to categorize those types of activities separately from "configuration".

Installation && Configuration

As much configuration as possible should come via a single values.yaml that can be version controlled and handed to helm with helm install -f values.yaml <workflow release> and later helm upgrade -f

Configurations supported through initial installation and configuration should match helm classic and our test matrix.

• off-cluster registry leaves out the registry component
• off-cluster storage leaves out the minio component

Update Configuration

helm upgrade -f new-values.yaml should work for simple configuration changes. This includes setting/configuring well-known knobs and dials like REGISTRATION_MODE, etc.

Configuration should also allow for tweaking (via values.yaml), the following options across all Workflow components. Most likely limited to simple interpolation in charts. Anything more complicated, like merging a structured annotation would fall under "Deeper Customization" below.

Configuration Use Cases:

• running a custom docker image (host, tag) for a given component
• adding arbitrary annotations to configure the router

Upgrade

Upgrades of a configured Workflow install should be as painless as possible. Let's make sure this totally rocks. To really test this out, we should create charts for 2.2, 2.3, 2.4 and 2.5 releases. We should lean on lifecycle hooks to make sure in-place upgrades are amazing.

Goals for upgrade are:

• preserve in-place configuration
• provide for zero-downtime (post 2.3) upgrades
• clean up Kubernetes resources after upgrade (e.g. migration from ReplicationController to Deployments)

Deeper Customization

Deeper customization includes complicated modification of a resource that is part of Workflow charts. For now, we'll leave this out of scope. If you need to run forked or custom versions of the Workflow Components that

Generic Cleanup

We made a bit of a mess in generate_params.toml, this would be a good opportunity to clean up the charts as long as it doesn't make things extremely complicated from a maintenance standopint.

A few rules of thumb, as a place to start:

• consistent variable naming (snake_case or camelCase-- we should pick one and stick to it #333)
• values sections organized by component (we should alphabetize the generate_params file #321)
• documented configuration options

[slack]

/cc @helgi @kmala Let's get the convo rolllin'

[kmala]

@slack i had tackled most of the issues with respect to installation, configuration and upgrade and written a gist of all the things https://gist.github.com/kmala/6479ebf6189debdfb6890585450f1195

[slack]

Yeah, your gist motivated me to get the rest written down!

[kmala]

deis/workflow#564