Chapter 8 Sealed Secrets: No endpoints available

[diunox]

Following the guide in chapter 8, using an Ubuntu 20 Droplet, followed the guide to the letter. Had helm, kubeseal, kubectl(configured via doctl) set up prior to beginning the guide. It works well till I try to fetch a cert, like so:

root@K8s-bastion-host:~/Kubernetes-Starter-Kit-Developers# kubeseal --fetch-cert --controller-namespace=sealed-secrets > pub-sealed-secrets.pem
error: cannot fetch certificate: no endpoints available for service "http:sealed-secrets-controller:"
root@K8s-bastion-host:~/Kubernetes-Starter-Kit-Developers# kubeseal --fetch-cert --controller-namespace=kube-system > pub-sealed-secrets.pem
error: cannot fetch certificate: services "sealed-secrets-controller" not found

Pod description:

root@K8s-bastion-host:~/Kubernetes-Starter-Kit-Developers# kubectl describe pods -n=sealed-secrets
Name:         sealed-secrets-controller-6857477d7b-p7zbl
Namespace:    sealed-secrets
Priority:     0
Node:         pool-10nacqz6k-cnr75/10.124.0.14
Start Time:   Tue, 15 Mar 2022 18:49:03 +0000
Labels:       app.kubernetes.io/instance=sealed-secrets-controller
              app.kubernetes.io/name=sealed-secrets
              pod-template-hash=6857477d7b
Annotations:  <none>
Status:       Running
IP:           10.244.1.103
IPs:
  IP:           10.244.1.103
Controlled By:  ReplicaSet/sealed-secrets-controller-6857477d7b
Containers:
  controller:
    Container ID:  containerd://ca391a34d1ab73586a15cd62180a6e00abc783edb2c23a0b1515a7a9b0329d92
    Image:         quay.io/bitnami/sealed-secrets-controller:v0.17.1
    Image ID:      quay.io/bitnami/sealed-secrets-controller@sha256:6c10080d6cf09cc823b333a9085d80799c95ddbd1ccac9a5db04daa89285c374
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      controller
    Args:
      --key-prefix
      sealed-secrets-key
    State:          Running
      Started:      Tue, 15 Mar 2022 18:49:06 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:http/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:http/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lrsb2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-lrsb2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  4m4s                default-scheduler  Successfully assigned sealed-secrets/sealed-secrets-controller-6857477d7b-p7zbl to pool-10nacqz6k-cnr75
  Normal   Pulling    4m3s                kubelet            Pulling image "quay.io/bitnami/sealed-secrets-controller:v0.17.1"
  Normal   Pulled     4m1s                kubelet            Successfully pulled image "quay.io/bitnami/sealed-secrets-controller:v0.17.1" in 2.366086755s
  Normal   Created    4m1s                kubelet            Created container controller
  Normal   Started    4m1s                kubelet            Started container controller
  Warning  Unhealthy  3m59s (x2 over 4m)  kubelet            Readiness probe failed: Get "http://10.244.1.103:8080/healthz": dial tcp 10.244.1.103:8080: connect: connection refused

Please let me know if you need more details, I will keep the host and cluster up.

[v-ctiutiu]

Hi @diunox,

Let me check and get back to you. Thanks.

[v-ctiutiu]

@diunox

I managed to reproduce your issue and it seems to be a bug with the kubeseal CLI. Can you try with the latest version please? You can fetch it from here (depending on your OS).

Seems that they fixed it starting with version 0.17.2. More info can be found here.

I tested using the latest version and it seems to work just fine. Please try and let me know.

Thanks a lot.

[diunox]

@v-ctiutiu thank you for this! I've tried again, and I'm past that issue now. Unfortunately, a new one has come up.

kubeseal --fetch-cert --controller-namespace=sealed-secrets > pub-sealed-secrets.pem

Results in an empty file.

When I try to run kubeseal --format=yaml --cert=pub-sealed-secrets.pem --secret-file your-app-secret.yaml --sealed-secret-file your-app-sealed.yaml, no 'your-app-sealed-yaml' file is generated due to the .pem file being empty. My controller looks healthy, i'm really not sure why this is failing, there are no errors anywhere as far as I can tell.

root@ablack-k8s-secrets-bastion:~/Kubernetes-Starter-Kit-Developers# kubectl get all -n sealed-secrets
NAME                                             READY   STATUS    RESTARTS   AGE
pod/sealed-secrets-controller-6857477d7b-8crbd   1/1     Running   0          3m10s

NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/sealed-secrets-controller   ClusterIP   10.245.203.112   <none>        8080/TCP   3m10s

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/sealed-secrets-controller   1/1     1            1           3m10s

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/sealed-secrets-controller-6857477d7b   1         1         1       3m10s
root@ablack-k8s-secrets-bastion:~/Kubernetes-Starter-Kit-Developers# kubectl get pods -n sealed-secrets
NAME                                         READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-6857477d7b-8crbd   1/1     Running   0          3m31s

I'm really not sure why this is failing because everything else looks healthy. I've tried with both 0.17.1 and 0.17.3, and both fail.

[v-ctiutiu]

Strange. I have one more solution for the empty file result. It's a workaround that I used in the flux cd chapter. I will add it in the sealed secrets chapter as well, if it's more viable.

If for some reason the kubeseal certificate fetch command hangs (or you get an empty file), you can use the following steps to work around this issue:

• First, open a terminal window, and expose the Sealed Secrets Controller service on your localhost (you can use CTRL - C to terminate, after fetching the public key):

  kubectl port-forward service/sealed-secrets-controller 8080:8080 -n sealed-secrets

• Then, in another terminal window, fetch the public key (please replace the <> placeholders accordingly):

  curl --retry 5 --retry-connrefused localhost:8080/v1/cert.pem > pub-sealed-secrets.pem

@diunox Can you try the above steps please and let me know the result. Thanks a lot.

[diunox]

Realized I've been testing the wrong chapter out... I'll leave this discussion for posterity, but I'm going to walk away from sealed secrets to focus on the scaling chapter.