Skip to content
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.

Commit

Permalink
Merge pull request #1200 from thojkooi/tiller-tls
Browse files Browse the repository at this point in the history 
helm-op: Add support for connecting to tiller using tls
  • Loading branch information
@stefanprodan
stefanprodan committed Jul 5, 2018
2 parents 34e4f93 + 8fdf917 commit 783f23e
Show file tree
Hide file tree
Showing 7 changed files with 310 additions and 20 deletions.
191 changes: 191 additions & 0 deletions chart/flux/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,13 @@ The following tables lists the configurable parameters of the Weave Flux chart a
| `helmOperator.repository` | Helm operator image repository | `quay.io/weaveworks/helm-operator`
| `helmOperator.tag` | Helm operator image tag | `0.1.0-alpha`
| `helmOperator.pullPolicy` | Helm operator image pull policy | `IfNotPresent`
| `helmOperator.tillerNamespace` | Namespace in which the Tiller server can be found | `kube-system`
| `helmOperator.tls.enable` | Enable TLS for communicating with Tiller | `false`
| `helmOperator.tls.verify` | Verify the Tiller certificate, also enables TLS when set to true | `false`
| `helmOperator.tls.secretName` | Name of the secret containing the TLS client certificates for communicating with Tiller | `helm-client-certs`
| `helmOperator.tls.keyFile` | Name of the key file within the k8s secret | `tls.key`
| `helmOperator.tls.certFile` | Name of the certificate file within the k8s secret | `tls.crt`
| `helmOperator.tls.caContent` | Certificate Authority content used to validate the Tiller server certificate | None
| `token` | Weave Cloud service token | None

Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example:
Expand All @@ -126,5 +133,189 @@ helm upgrade --reuse-values flux \
weaveworks/flux
```

## Installing Weave Flux helm-operator and Helm with TLS enabled

### Installing Helm / Tiller

Generate certificates for Tiller and Flux. This will provide a CA, server certs for tiller and client certs for helm / weave flux.

The following script can be used for that (requires [cfssl](https://github.com/cloudflare/cfssl)):

```bash
export TILLER_HOSTNAME=tiller-server
export TILLER_SERVER=server
export USER_NAME=flux-helm-operator

mkdir tls
cd ./tls

# Prep the configuration
echo '{"CN":"CA","key":{"algo":"rsa","size":4096}}' | cfssl gencert -initca - | cfssljson -bare ca -
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json

# Create the tiller certificate
echo '{"CN":"'$TILLER_SERVER'","hosts":[""],"key":{"algo":"rsa","size":4096}}' | cfssl gencert \
-config=ca-config.json -ca=ca.pem \
-ca-key=ca-key.pem \
-hostname="$TILLER_HOSTNAME" - | cfssljson -bare $TILLER_SERVER

# Create a client certificate
echo '{"CN":"'$USER_NAME'","hosts":[""],"key":{"algo":"rsa","size":4096}}' | cfssl gencert \
-config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem \
-hostname="$TILLER_HOSTNAME" - | cfssljson -bare $USER_NAME
```

Alternatively, you can follow the [Helm documentation for configuring TLS](https://docs.helm.sh/using_helm/#using-ssl-between-helm-and-tiller).


Next deploy Helm with TLS and RBAC enabled;

Create a file called `helm-rbac.yaml`. This contains all the RBAC configuration for Tiller:
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system

---
# Helm client serviceaccount
apiVersion: v1
kind: ServiceAccount
metadata:
name: helm
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: tiller-user
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- pods/portforward
verbs:
- create
- apiGroups:
- ""
resources:
- pods
verbs:
- list
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: tiller-user-binding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: tiller-user
subjects:
- kind: ServiceAccount
name: helm
namespace: kube-system
```

Deploy Tiller:

```bash
kubectl apply -f helm-rbac.yaml

# Deploy helm with mutual TLS enabled
helm init --upgrade --service-account tiller \
--override 'spec.template.spec.containers[0].command'='{/tiller,--storage=secret}' \
--tiller-tls \
--tiller-tls-cert ./tls/server.pem \
--tiller-tls-key ./tls/server-key.pem \
--tiller-tls-verify \
--tls-ca-cert ./tls/ca.pem
```

To check if Tiller installed succesfully with TLS enabled, try `helm ls`. This should give an error:

```bash
# Should give an error
$ helm ls
Error: transport is closing
```

When providing the certificates, it should work correctly:

```bash
helm --tls \
--tls-ca-cert ./tls/ca.pem \
--tls-cert ./tls/flux-helm-operator.pem \
--tls-key ././tls/flux-helm-operator-key.pem \
ls
```

## deploy weave flux helm-operator

First create a new Kubernetes TLS secret for the client certs;

```bash
kubectl create secret tls helm-client --cert=tls/flux-helm-operator.pem --key=./tls/flux-helm-operator-key.pem
```

> note; this has to be in the same namespace as the helm-operator is deployed in.
Deploy flux with Helm;

```bash
helm repo add weaveworks https://weaveworks.github.io/flux

helm upgrade --install \
--set helmOperator.create=true \
--set git.url=$YOUR_GIT_REPO \
--set helmOperator.tls.enable=true \
--set helmOperator.tls.verify=true \
--set helmOperator.tls.secretName=helm-client \
--set helmOperator.tls.caContent="$(cat ./tls/ca.pem)" \
flux \
weaveworks/flux
```

### Check if it worked

Use `kubectl logs` on the helm-operator and observe the helm client being created.

### Debugging

#### Error creating helm client: failed to append certificates from file: /etc/fluxd/helm-ca/ca.crt

Your CA certificate content is not set correctly, check if your configMap contains the correct values. Example:

```bash
$ kubectl get configmaps flux-helm-tls-ca-config -o yaml
apiVersion: v1
data:
ca.crt: |
-----BEGIN CERTIFICATE-----
....
-----END CERTIFICATE-----
kind: ConfigMap
metadata:
creationTimestamp: 2018-07-04T15:27:25Z
name: flux-helm-tls-ca-config
namespace: helm-system
resourceVersion: "1267257"
selfLink: /api/v1/namespaces/helm-system/configmaps/flux-helm-tls-ca-config
uid: c106f866-7f9e-11e8-904a-025000000001
```
32 changes: 32 additions & 0 deletions chart/flux/templates/helm-operator-deployment.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,18 @@ spec:
secret:
secretName: {{ template "flux.fullname" . }}-git-deploy
defaultMode: 0400
{{- if .Values.helmOperator.tls.enable }}
- name: helm-tls-certs
secret:
secretName: {{ .Values.helmOperator.tls.secretName }}
defaultMode: 0400
{{- if .Values.helmOperator.tls.verify }}
- name: helm-tls-ca
configMap:
name: {{ template "flux.fullname" . }}-helm-tls-ca-config
defaultMode: 0600
{{- end }}
{{- end }}
containers:
- name: flux-helm-operator
image: "{{ .Values.helmOperator.repository }}:{{ .Values.helmOperator.tag }}"
Expand All @@ -44,8 +56,28 @@ spec:
- name: git-key
mountPath: /etc/fluxd/ssh
readOnly: true
{{- if .Values.helmOperator.tls.enable }}
- name: helm-tls-certs
mountPath: /etc/fluxd/helm
readOnly: true
{{- if .Values.helmOperator.tls.verify }}
- name: helm-tls-ca
mountPath: /etc/fluxd/helm-ca
readOnly: true
{{- end }}
{{- end }}
args:
- --git-url={{ .Values.git.url }}
- --git-branch={{ .Values.git.branch }}
- --git-charts-path={{ .Values.git.chartsPath }}
- --tiller-namespace={{ .Values.helmOperator.tillerNamespace }}
{{- if .Values.helmOperator.tls.enable }}
- --tiller-tls-enable={{ .Values.helmOperator.tls.enable }}
- --tiller-tls-key-path=/etc/fluxd/helm/{{ .Values.helmOperator.tls.keyFile }}
- --tiller-tls-cert-path=/etc/fluxd/helm/{{ .Values.helmOperator.tls.certFile }}
{{- if .Values.helmOperator.tls.verify }}
- --tiller-tls-verify={{ .Values.helmOperator.tls.verify }}
- --tiller-tls-ca-cert-path=/etc/fluxd/helm-ca/ca.crt
{{- end }}
{{- end }}
{{- end -}}
11 changes: 11 additions & 0 deletions chart/flux/templates/helm-tls.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{{- if .Values.helmOperator.tls.enable -}}
{{- if .Values.helmOperator.tls.verify -}}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "flux.fullname" . }}-helm-tls-ca-config
data:
ca.crt: |
{{ .Values.helmOperator.tls.caContent | indent 4 }}
{{- end -}}
{{- end -}}
8 changes: 8 additions & 0 deletions chart/flux/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,14 @@ helmOperator:
repository: quay.io/weaveworks/helm-operator
tag: 0.1.0-alpha
pullPolicy: IfNotPresent
tillerNamespace: kube-system
tls:
secretName: 'helm-client-certs'
verify: false
enable: false
keyFile: 'tls.key'
certFile: 'tls.crt'
caContent: ''

rbac:
# Specifies whether RBAC resources should be created
Expand Down
24 changes: 23 additions & 1 deletion cmd/helm-operator/main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,12 @@ var (
tillerPort *string
tillerNamespace *string

tillerTLSVerify *bool
tillerTLSEnable *bool
tillerTLSKey *string
tillerTLSCert *string
tillerTLSCACert *string

chartsSyncInterval *time.Duration
chartsSyncTimeout *time.Duration
eventHandlerWorkers *uint
Expand Down Expand Up @@ -91,6 +97,12 @@ func init() {
tillerPort = fs.String("tiller-port", "", "Tiller port.")
tillerNamespace = fs.String("tiller-namespace", "kube-system", "Tiller namespace. If not provided, the default is kube-system.")

tillerTLSVerify = fs.Bool("tiller-tls-verify", false, "Verify TLS certificate from Tiller. Will enable TLS communication when provided.")
tillerTLSEnable = fs.Bool("tiller-tls-enable", false, "Enable TLS communication with Tiller. If provided, requires TLSKey and TLSCert to be provided as well.")
tillerTLSKey = fs.String("tiller-tls-key-path", "/etc/fluxd/helm/tls.key", "Path to private key file used to communicate with the Tiller server.")
tillerTLSCert = fs.String("tiller-tls-cert-path", "/etc/fluxd/helm/tls.crt", "Path to certificate file used to communicate with the Tiller server.")
tillerTLSCACert = fs.String("tiller-tls-ca-cert-path", "", "Path to CA certificate file used to validate the Tiller server. Required if tiller-tls-verify is enabled.")

chartsSyncInterval = fs.Duration("charts-sync-interval", 3*time.Minute, "Interval at which to check for changed charts")
chartsSyncTimeout = fs.Duration("charts-sync-timeout", 1*time.Minute, "Timeout when checking for changed charts")
eventHandlerWorkers = fs.Uint("event-handler-workers", 2, "Number of workers processing events for Flux-Helm custom resources")
Expand Down Expand Up @@ -162,7 +174,17 @@ func main() {
}

// HELM ---------------------------------------------------------------------------------
helmClient := fluxhelm.ClientSetup(log.With(logger, "component", "helm"), kubeClient, fluxhelm.TillerOptions{IP: *tillerIP, Port: *tillerPort, Namespace: *tillerNamespace})
helmClient := fluxhelm.ClientSetup(log.With(logger, "component", "helm"), kubeClient, fluxhelm.TillerOptions{
IP: *tillerIP,
Port: *tillerPort,
Namespace: *tillerNamespace,

TLSVerify: *tillerTLSVerify,
TLSEnable: *tillerTLSEnable,
TLSKey: *tillerTLSKey,
TLSCert: *tillerTLSCert,
TLSCACert: *tillerTLSCACert,
})

// The status updater, to keep track the release status for each
// FluxHelmRelease. It runs as a separate loop for now.
Expand Down
23 changes: 22 additions & 1 deletion integrations/helm/helm.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,18 @@ import (
"k8s.io/client-go/kubernetes"
k8shelm "k8s.io/helm/pkg/helm"
rls "k8s.io/helm/pkg/proto/hapi/services"
"k8s.io/helm/pkg/tlsutil"
)

type TillerOptions struct {
IP string
Port string
Namespace string
TLSVerify bool
TLSEnable bool
TLSKey string
TLSCert string
TLSCACert string
}

// Helm struct provides access to helm client
Expand All @@ -32,7 +38,22 @@ func newClient(kubeClient *kubernetes.Clientset, opts TillerOptions) (*k8shelm.C
return &k8shelm.Client{}, err
}

return k8shelm.NewClient(k8shelm.Host(host)), nil
options := []k8shelm.Option{k8shelm.Host(host)}
if opts.TLSVerify || opts.TLSEnable {
tlscfg, err := tlsutil.ClientConfig(tlsutil.Options{
KeyFile: opts.TLSKey,
CertFile: opts.TLSCert,
InsecureSkipVerify: !opts.TLSVerify,
CaCertFile: opts.TLSCACert,
})

if err != nil {
return &k8shelm.Client{}, err
}
options = append(options, k8shelm.WithTLS(tlscfg))
}

return k8shelm.NewClient(options...), nil
}

func ClientSetup(logger log.Logger, kubeClient *kubernetes.Clientset, tillerOpts TillerOptions) *k8shelm.Client {
Expand Down

0 comments on commit 783f23e

Please sign in to comment.