Cannot fetch certificate: no endpoints available for service "http:sealed-secrets-controller:"

[seansabour]

Hello!
We've installed the latest sealed-secrets via helm charts and noticed an issue when trying to use kubseal.

kubeseal --fetch-cert
error: cannot fetch certificate: no endpoints available for service "http:sealed-secrets-controller:"

After digging a bit deeper i uncovered a PR that was merged that fixed this exact issue, but it doesn't look like it's made its way into this repo:
helm/charts#22097

This can be checked by running:

1. kubectl proxy

curl http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/http:sealed-secrets:/proxy/v1/cert.pem
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
  },
  "status": "Failure",
  "message": "no endpoints available for service \"http:sealed-secrets:\"",
  "reason": "ServiceUnavailable",
  "code": 503
}%

If you add the http protocol for port it works: curl http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/http:sealed-secrets-controller:http/proxy/v1/cert.pem

[GiGurra]

Same issue here

[glitchcrab]

Also seeing the same problem after upgrading to the latest version. Manually removing the name and setting targetPort: 8080 on the service got things working again.

[albanduval]

Same issue here after helm upgrade.

Thanks @glitchcrab for the workaround, it made the trick for a quick patch.

[jmpalomar]

Same here anyone tried downgrading?

[juan131]

Hi everyone!

Thanks so much for reporting this! The changes we recently introduced at #690 (introducing a name for the http port exposed in the service) broke the compatibility with kubeseal.

This should be fixed by this PR: #648

In the meantime, you can workaround this by removing the name and setting targetPort: 8080 as @glitchcrab pointed out.

[demisx]

Same here anyone tried downgrading?

I've downgraded helm release to v1.16.1 and it works again.

[dirien]

Hi @demisx, can confirm v1.16.1 works perfect. Just run into the same issue with >v2

[kerossin]

To easily apply the aforementioned fix to the service:

kubectl -n kube-system patch svc sealed-secrets-controller --type='json' -p='[{"op": "remove", "path": "/spec/ports/0/name"}, {"op": "replace", "path": "/spec/ports/0/targetPort", "value":8080}]'

[mtt88]

To easily apply the aforementioned fix to the service:

kubectl -n kube-system patch svc sealed-secrets-controller --type='json' -p='[{"op": "remove", "path": "/spec/ports/0/name"}, {"op": "replace", "path": "/spec/ports/0/targetPort", "value":8080}]'

If you're using the prom servicemonitor in the chart don't forget to update this after patching the service. I.e

kubectl -n <namespace> patch servicemonitor sealed-secrets-controller --type='json' -p='[{"op": "remove", "path": "/spec/endpoints/0/port"}, {"op": "replace", "path": "/spec/endpoints/0/targetPort", "value":8080}]'

[juan131]

Thanks for sharing the commands to patch both the service and the servicemonitor @kerossin @mtt88

[juan131]

Hi everyone! I’m glad to announce that we just released a new version of Sealed Secrets that address this issue: v0.17.2
Please give it a try when you have a chance! Find more info at:

• https://github.com/bitnami-labs/sealed-secrets/releases/tag/v0.17.2
• https://github.com/bitnami-labs/sealed-secrets/blob/main/RELEASE-NOTES.md

[cdenneen]

@juan131 I have v0.17.2 installed and still having the same error:
error: cannot fetch certificate: no endpoints available for service "http:sealed-secrets-controller:"

[juan131]

@cdenneen are you using v0.17.2 in the controller or kubeseal? Please note it's kubeseal the tool including the fix.

[dirien]

Just a FYI: In the current project I am working, we switched to "offline" sealing. Means we fetch the public certs and use this in the kubeseal commmand! So much better, as we don't need to handle the kubeconfig stuff, and can share the key with the Team!

Back to topic: the version v0.17.2 works flawless!

[cdenneen]

@juan131 I had old version of kubeseal still. I only had updated the sealed-secrets-controller to v0.17.2. Thanks for that clarification.

[emenylouu]

I am facing another related issue on the latest version v0.18.2. I'm using this version on both the controller and the kubeseal

Output of this cmd after kubectl proxy is
curl http://127.0.0.1:8001/api/v1/namespaces/flux-system/services/sealed-secrets-controller:8001/proxy/v1/cert.pem

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "services \"sealed-secrets-controller\" not found",
  "reason": "NotFound",
  "details": {
    "name": "sealed-secrets-controller",
    "kind": "services"
  },
  "code": 404
}

kubeseal --version
output:
kubeseal version: v0.18.2

kubeseal --fetch-cert -n <mynamespace>
Output is:
error: cannot fetch certificate: no endpoints available for service "sealed-secrets-controller"

Controller.yml service :

apiVersion: v1
kind: Service
metadata:
  annotations: {}
  labels:
    name: sealed-secrets-controller
  name: sealed-secrets-controller
  namespace: <mynamespace>
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    name: sealed-secrets-controller
  type: ClusterIP 

Any idea on what I'm missing or doing wrong here?

[dirien]

Hi @emenylouu,

I installed sealed-secret via the helm chart on a fresh k8s cluster (docker-desktop 1.25) and it works fine:

helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller --version 2.6.1 sealed-secrets/sealed-secrets
NAME: sealed-secrets
LAST DEPLOYED: Mon Sep  5 12:36:23 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

You should now be able to create sealed secrets.

1. Install the client-side tool (kubeseal) as explained in the docs below:

    https://github.com/bitnami-labs/sealed-secrets#installation-from-source

2. Create a sealed secret file running the command below:

    kubectl create secret generic secret-name --dry-run=client --from-literal=foo=bar -o [json|yaml] | \
    kubeseal \
      --controller-name=sealed-secrets-controller \
      --controller-namespace=kube-system \
      --format yaml > mysealedsecret.[json|yaml]

The file mysealedsecret.[json|yaml] is a commitable file.

If you would rather not need access to the cluster to generate the sealed secret you can run:

    kubeseal \
      --controller-name=sealed-secrets-controller \
      --controller-namespace=kube-system \
      --fetch-cert > mycert.pem

to retrieve the public cert used for encryption and store it locally. You can then run 'kubeseal --cert mycert.pem' instead to use the local cert e.g.

    kubectl create secret generic secret-name --dry-run=client --from-literal=foo=bar -o [json|yaml] | \
    kubeseal \
      --controller-name=sealed-secrets-controller \
      --controller-namespace=kube-system \
      --format [json|yaml] --cert mycert.pem > mysealedsecret.[json|yaml]

3. Apply the sealed secret

    kubectl create -f mysealedsecret.[json|yaml]

Running 'kubectl get secret secret-name -o [json|yaml]' will show the decrypted secret that was generated from the sealed secret.

Both the SealedSecret and generated Secret must have the same name and namespace.

on ⛵ docker-desktop on docker-desktop in docker-desktop ()civogo on  master [!⇡] via 🐹 v1.19 took 3s
❯ kubeseal --fetch-cert -n kube-system
-----BEGIN CERTIFICATE-----
MIIEzDCCArSgAwIBAgIQPHJbXfIvad2JL0Xb5cXw9jANBgkqhkiG9w0BAQsFADAA
MB4XDTIyMDkwNTEwMzIzM1oXDTMyMDkwMjEwMzIzM1owADCCAiIwDQYJKoZIhvcN
AQEBBQADggIPADCCAgoCggIBAO0tcugE2TAMA/9V2l/03ykfbyQsLXpnJNmdJ6Np
RrbjK6VKAtIRgaw7cHF1eznkJpIy9yxsGjavrAFSO4mEtR94L8wYpBKRfe06wLMg
CKajkapU18X3ldatITdLAZj2p9bCxGv+5/pF97a5rtS/h1ofOOMLtcHOJnjkzxMG
tOGGWngc2hfp0ruO1LyS+R8TqDDARiLW+aTMXIuaguEIQ9aq40Asg6uprIkJ7Jp6
X+QwvLHL9PI6NIoDNtwukTH9kfqmduYFoalMkILFzqKL7HwY1796Qq6FY2GD/RtF
27vAI+Akn5qf1VWRVSeuVxA8qYhZkElXI3Q7vbbP/gJgVc49wpMfTC7TFs0+cdAx
mvy5ooiXuHZwwoKEHMAM8Io+cU4eSypgFwQD1EMQclhDdbiSWJ9sZUBO+h5F8Wq8
TlemMffyhdMMIbwU/oNKGEkR9RhnLhQyNejIrOpYZ4HcGdYxhR0ErYzylTiE1bmc
03iUY3y2Zi6ic8C4etv4T3LeEynd+icSZrAxZU09JnAQAuvonTOQ+RF4S+tsq29w
3VeBvBotlXsZVvJWtvrqlaGCrMeCNhO8EWDkYVbOCoq+96cMEOCiOe/NFgL42ztA
Sw4Z9lD2dTme3LehFge3pq+jLza6J2Zf1Xe/WZvPOl91lnw6MofQI8v/YMwNzXE6
tLKjAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIAATAPBgNVHRMBAf8EBTADAQH/MB0G
A1UdDgQWBBRVwr/zCuwZ79ydHZbJyMttT/CdhzANBgkqhkiG9w0BAQsFAAOCAgEA
s09KWlMfzCNKs6/+cjt5Yv7JVuZdJSVc0IHfja3av8O/y5wmli1wF+M0uH9VJeGO
+hXNcppqUfkCEMNen82pMZoYIajh9bwhQTXbfJyc9qqJWHo0zJinMqxDjY3ENOGw
wLbAVJFGdYQvbpXKxU/yYXQJvYkKgM5G6BlSflqpcV9uuLUZ3vV14v2DHiZJP6ej
emoIipiA0hxyAUWoeO9iPMDIaaUqBDqsyYGCnQ7+jEVZ1RyPA6ZaumO4e9kX5N/+
kL9N7zk+NEnJh+tUNQKRApfZnYwgi1FxxFfJ66HzLFc1zq+bPAC3VG0MOba73rgi
OsFd97Lg8ONu0f6gLqfC+B02BsmLAi0/vx9cJpF5AilS3f0VRSGSkHI6Rwl9wRsk
pydi1Tdn2RiWO2TJIVfh4hpj+c2M8dSIU42mMwnMi95swYYMOPtt0WYZ1odpMMVP
1SS14zWHk+xzneQnPcbCtvvJ/D6e4maWDSTnKu7V8AhONtWLvDzdMza7sGD7q/Dv
v3FUTLZ0GHFSOODmFhfAqBfhB28EHPbvL+nrnfeSzUnFqc/byEOQ2OYbns9g5Yro
REK9xkZF5+09uAxIZVFfNZAxEpsEQXPe409Lmu+FfnV5hVLhmw//1Ct/yhCzeyeM
JMdmzhCaNXJdScaWEGy6vZcqTIAhYziIMPoXXhUHCFY=
-----END CERTIFICATE-----

Would need more input from you regarding the way you installed.

Check maybe the endpoints?

❯ k get endpoints -n kube-system
NAME                        ENDPOINTS                                               AGE
docker.io-hostpath          <none>                                                  46d
kube-dns                    10.1.2.202:53,10.1.2.203:53,10.1.2.202:53 + 3 more...   46d
sealed-secrets-controller   10.1.2.205:8080                                         2m52s

And the endpoint itself:

k get  endpoints sealed-secrets-controller -o yaml -n kube-system
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2022-09-05T10:36:29Z"
  creationTimestamp: "2022-09-05T10:36:23Z"
  labels:
    app.kubernetes.io/instance: sealed-secrets
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: sealed-secrets
    app.kubernetes.io/version: v0.18.2
    helm.sh/chart: sealed-secrets-2.6.1
  name: sealed-secrets-controller
  namespace: kube-system
  resourceVersion: "44162"
  uid: ffe04034-42fe-4559-9fdc-4163f6540bac
subsets:
- addresses:
  - ip: 10.1.2.205
    nodeName: docker-desktop
    targetRef:
      kind: Pod
      name: sealed-secrets-controller-68dd8fb447-xvwh9
      namespace: kube-system
      uid: 3863e0e7-07e2-4dd9-ad3e-44284011c2a8
  ports:
  - name: http
    port: 8080
    protocol: TCP

Hope it helps to solve the issue.

[emenylouu]

Thanks for sharing all the info and steps @dirien!! after multiple attempts, I tested the configuration and came to the conclusion that sealed-secret-controller only runs in kube-system namespace as a part of the cluster configuration. what I was trying to do is to install it on another namespace where my secret is. I haven't seen any official documentation around this though, but I've seen another github thread that recommends this.