Install in an isolated namespace

[retr0h]

I'm attempting to install sealed secrets helm chart into a namespace where my user is "admin" at the role level not the cluster level. I was hoping to install sealed secrets in this namespace to isolate the users from needing access to other namespaces/secrets.

Am I being a numbskull here?

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: {{ item.namespace }}-manager
  namespace: {{ item.namespace }}
rules:
- apiGroups:
  - "*"
  resources:
  - "*"
  verbs:
  - "*"

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: {{ item.namespace }}-binding
  namespace: {{ item.namespace }}
subjects:
{% for item in item.users | default([]) %}
- kind: User
  name: {{ item }}
  apiGroup: rbac.authorization.k8s.io
{% endfor -%}
roleRef:
  kind: Role
  name: {{ item.namespace }}-manager
  apiGroup: rbac.authorization.k8s.io

[retr0h]

I guess this just isn't the way to go about it. I am running the service in kube-system which is the intended way to do this.

The user I have has the access above, but needs access to services/proxy in cluster role.

kubeseal <mysecret.json >mysealedsecret.json
panic: Error fetching certificate: services "http:sealed-secrets-controller:" is forbidden: User "xxx" cannot get resource "services/proxy" in API group "" in the namespace "kube-system"

Any suggestion on cluster role bindings?

[retr0h]

Added 'services/proxy' to list of resources read-only user can access.

[mdraijer]

Exactly my point also. Our users have only access to their own namespace(s) and cannot use kubeseal without the extra rolebinding in the kube-system namespace (or at least: the namespace where the controller is running).
This information should at least be in the README.md (I didn't find it there).
Also I do not know what else might be opened up in kube-system for regular users with that role binding: is it safe enough?

Can this issue be reopened to answer my questions?

[mkmik]

Yes

[mkmik]

@mdraijer just to make sure I understand the problem you're facing:

You have the rights to install the sealed-secret-controller in kube-system or another namespace
but the users of your clusters are unprivileged and kubeseal fails to fetch the controller certificate.

If yes, you have two options:

1. use offline certificates
2. configure RBAC so your unprivileged users can access the service via the k8s proxy.

I'll improve the README and find a way to simplify all of this if possible, but in the meantime I hope this can unblock you:

---

use offline certificates

kubeseal doesn't really need to talk to the cluster. All it does is encrypt your secrets a public key so that only the controller running in the cluster can decrypt it.

At least one user has privileged access

As long as you can provide that key to your users (and they can trust it's the right key), then they can generate sealed-secrets without any access to the cluster.

$ kubeseal --cert /tmp/sealed-cert.pem <secret.yaml >sealed-secret.json

FWIW, I have exactly this situation at work: some of our clusters are configured in such a way and we share a certificate file with our team with a secure mechanism (in our case that's keybase team signed filesystem)

In order to get the certificate you need to do this once from an account that has the rights:

$ kubeseal --controller-namespace=mysealed --fetch-cert >/tmp/sealed-cert.pem

or if you installed it in a dedicated namespace (possibly with a custom controller name, as happens when you use the helm chart):

$ kubeseal --controller-namespace=the-namespace-where-you-installed-it --controller-name=name-of-your-controller --fetch-cert >/tmp/sealed-cert.pem

You have access to logs

$ kubectl -n kube-system get logs -lname=sealed-secrets-controller
....
-----BEGIN CERTIFICATE-----
MIIErTCCApWgAwI
...
$ kubectl -n the-namespace-where-you-installed-it ...

You don't have privileged access at hand and there are no strict network policies

As long as you can run some code in the cluster, you can try to directly curl the service endpoint:

$ kubectl run curl --generator=run-pod/v1 --image=everpeace/curl-jq
pod/curl created
$ kubectl exec -ti curl curl http://sealed-secrets-controller.kube-system.svc.cluster.local:8080/v1/cert.pem
-----BEGIN CERTIFICATE-----
MIIErTCCApWgAwI
$ kubectl delete pod curl

[olliebun]

Food for thought: a variant on offline certificates is to add an Ingress to the namespace that runs the sealed-secrets controller, exposing the /v1/cert.pem endpoint.

Rather than mess around with role bindings or distributing the certificate to our developers directly, we've simply documented the URL for the certificate for each cluster.

[mkmik]

@ceralena that's a reasonable approach.

I'm struggling to figure out what is the best option that would work out of the box. Ingresses unfortunately don't seem to fit the bill. It's hard to provide instructions for setting up TLS ingress that would work out of the box everywhere (people have different ingress controllers, different load balancers).

Perhaps there is just no one size fits all solution.
Perhaps we should improve and document the way to deal with two main scenarios:

a) a person who can talk to the cluster with kubectl and push sealed secret resources there. The kubectl tool should obtain the certificate using the least amount of privileges possible.

b) a person who cannot talk to the cluster (e.g. all interactions with the cluster are done by a GitOps style interaction mediated by a CI tool which has actual access to the cluster).

Both scenarios have to be secure, i.e. it should be very hard for an attacker to trick the user into encrypting the secret belonging to the attacker.

It's easy for the sealed secret controller to post the cert somewhere; what's harder is for the user to establish that it's the right certificate to use.

[mdraijer]

Thanks @mkmik, that helps. We will distribute the public key to the users.

[mdraijer]

Related to the original question in this issue: install in an isolated namespace, i.e. some other namespace than kube-system:
I can't get the sealed-secrets-controller pod running in another namespace. It is always hanging when it tries to find a master key (which is not there: after every attempt I clean the system of everything sealed-secrets related).
Tried with kubectl install, with helm install, with v0.7.0, with v0.8.1, in kube-system and in other namespace. Everything works except the other namespace (both v0.7.0 and v0.8.1, of course only with helm install).
Can you give any pointers as to what I'm doing wrong, or where I can look for clues?

[mkmik]

@mdraijer Hmm, it should work. Could you show me the exact commands you issued to install it with kubectl in another namespace?

[mdraijer]

I did not use kubectl, because that would mean editing the yaml first (hard coded namespace).
With helm the command was: helm install --namespace $NAMESPACE --name sealed-secrets stable/sealed-secrets
Logging of the pod:

2019/07/30 08:32:44 Starting sealed-secrets controller version: v0.8.1
2019/07/30 08:32:44 Searching for existing private keys

Thought it could be the resource quota, they were initially exactly the resources requested by the pod. But I made that 10 times as high now:

spec:
  hard:
    limits.cpu: '1'
    limits.memory: 5Gi
    requests.cpu: 500m
    requests.memory: 1280Mi
    requests.storage: 100Mi

[mkmik]

I'm not familiar with the helm chart and it's been released independently so I'd prefer to not add a variable here.

Kubectl has a builtin feature that allows you to apply config overlays called kustomize. It's a bit better than helm in that it works on any k8s config file.

$ wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.8.1/controller.yaml
$ kubectl create ns mysealed
$ cat >kustomization.yaml <<EOF
namespace: mysealed
resources:
  - controller.yaml
$ kubectl apply -k .

(you can clean up with kubectl delete -k ., and see diffs between local conf and cluster with kuvectl diff -k .)

It would really help if you could try this (possibly after making sure there is no other sealed secret instance left over)

[mdraijer]

For now we have settled with an installation in kube-system, but thanks for your help.

[mkmik]

@mdraijer thanks anyway; if you'll have some extra time, I'd really appreciate if you manage to reproduce the issue without the helm chart; I'd love to know if there is a bug.

[mkmik]

I'll keep this issue open because I think the certificate fetching issue can be improved with some tweaks to RBAC

[mkmik]

Since #208 the sealed secret controller is accessible even if users have no access to the namespace it's deployed into, I think this issue can be considered closed.

(Keep in mind that until we directly release helm charts from this project, there is no guarantee that config changes as this one will be reflected soon in the helm chart. Please consider using kustomize)