New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Could not find nodeport for backend #281

Open
korovaisdead opened this Issue Jan 26, 2018 · 13 comments

Comments

Projects
None yet
7 participants
@korovaisdead

korovaisdead commented Jan 26, 2018

After creating the all required resources I've got the keypair stored inside the secret.
But the ingress shows an error:

Could not find nodeport for backend {ServiceName:cm-bla-bla-com-qwtvo ServicePort:{Type:0 IntVal:8089 StrVal:}}: service default/cm-bla-bla-com-qwtvo not found in store

Environment:

  • Kubernetes version (use kubectl version): 1.9
  • Cloud provider or hardware configuration**: GKE
  • Install tools:
  • Others:
@munnerz

This comment has been minimized.

Member

munnerz commented Jan 26, 2018

That seems like a potentially old, transient error.

service default/cm-bla-bla-com-qwtvo not found in store implies that the service 'cm-bla-bla-com-qwtvo' in namespace 'default' no longer exists, which is expected as the validation has completed successfully (so cert-manager has cleaned up the resources it created to validate the challenge).

Are you seeing any actual problems, aside from this error message? Is the error message being printed continuously or the like?

Can you also provide the output of:

kubectl get ing,svc,po -o wide -n default

I'd expect the cm- prefixed entries to have been deleted since the challenge validation is complete 😄

@dguettler

This comment has been minimized.

dguettler commented Mar 13, 2018

I'm running into the same issue.

Are you seeing any actual problems, aside from this error message?

Yes, the ingress is not binding to its static IP and is "stuck" on "Creating ingress"

Is the error message being printed continuously or the like?

Yes the error message happens continuously on low frequency

Here is the output from above command

$ kubectl get ing,svc,po -o wide -n development
NAME                                                  HOSTS                             ADDRESS   PORTS     AGE
ing/services-development-my-domain                    my-domain                                   80, 443   7h

NAME                                                  CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE       SELECTOR
svc/cm-tls-secret-bfmcr                               [REDACTED]      <nodes>       8089:31941/TCP   7m        certmanager.k8s.io/certificate=tls-secret,certmanager.k8s.io/domain=my-domain,certmanager.k8s.io/id=ihgzp,certmanager.k8s.io/managed=true
svc/cm-tls-secret-ofwah                               [REDACTED]      <nodes>       8089:30166/TCP   4h        certmanager.k8s.io/certificate=tls-secret,certmanager.k8s.io/domain=my-domain,certmanager.k8s.io/id=tjcsn,certmanager.k8s.io/managed=true
svc/services-development-my-domain                    [REDACTED]      <nodes>       80:32493/TCP     31d       app=my-app,env=development,release=services-development

NAME                                                  READY     STATUS    RESTARTS   AGE       IP            NODE
po/cm-tls-secret-golis                                1/1       Running   0          7m        [REDACTED]    gke-n1-standard-2-20dfa812-0rm6
po/cm-tls-secret-nergh                                1/1       Running   1          9h        [REDACTED]    gke-n1-standard-2-20dfa812-0rm6
po/cm-tls-secret-yuiil                                1/1       Running   1          4h        [REDACTED]    gke-n1-standard-2-20dfa812-0rm6
po/services-development-my-domain-f5474fc57-ftbbt     1/1       Running   0          7h        [REDACTED]    gke-n1-standard-2-20dfa812-8mgg

@korovaisdead were you able to resolve this and if so how.

@abevoelker

This comment has been minimized.

abevoelker commented Mar 23, 2018

Also seeing this issue. The challenge validation does not complete (no secret in kubectl get secret); cm- prefixed pods keep being recreated every 10 mins or so.

$ kubectl get ing,svc,po -o wide -n default
NAME                                HOSTS                                                                    ADDRESS              PORTS     AGE
ing/captioned-images-ipv4-ingress   captioned-images.abevoelker.com,assets-captioned-images.abevoelker.com   35.201.64.7          80        2d
ing/captioned-images-ipv6-ingress   captioned-images.abevoelker.com,assets-captioned-images.abevoelker.com   2600:1901:0:439d::   80        2d

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE       SELECTOR
svc/captioned-images-assets         NodePort    10.51.247.191   <none>        80:31095/TCP     2d        run=captioned-images-web
svc/captioned-images-web            NodePort    10.51.251.87    <none>        80:31821/TCP     2d        run=captioned-images-web
svc/cm-captioned-images-tls-hfqnz   NodePort    10.51.241.16    <none>        8089:30162/TCP   1m        certmanager.k8s.io/certificate=captioned-images-tls,certmanager.k8s.io/domain=assets-captioned-images.abevoelker.com,certmanager.k8s.io/id=gvkkm,certmanager.k8s.io/managed=true
svc/cm-captioned-images-tls-zhqbx   NodePort    10.51.240.182   <none>        8089:30976/TCP   1m        certmanager.k8s.io/certificate=captioned-images-tls,certmanager.k8s.io/domain=captioned-images.abevoelker.com,certmanager.k8s.io/id=deuqz,certmanager.k8s.io/managed=true
svc/kubernetes                      ClusterIP   10.51.240.1     <none>        443/TCP          3d        <none>

NAME                                       READY     STATUS      RESTARTS   AGE       IP           NODE
po/captioned-images-db-migrate-qzkdt       1/2       Completed   0          2d        10.48.2.8    gke-captioned-images-app-default-pool-70b2da38-6k20
po/captioned-images-web-5b5b686768-hh5hw   3/3       Running     0          55m       10.48.4.7    gke-captioned-images-app-default-pool-70b2da38-sd7s
po/captioned-images-web-5b5b686768-wbqh9   3/3       Running     0          55m       10.48.4.8    gke-captioned-images-app-default-pool-70b2da38-sd7s
po/cm-captioned-images-tls-dasls           1/1       Running     0          1m        10.48.0.15   gke-captioned-images-app-default-pool-70b2da38-bk19
po/cm-captioned-images-tls-odqrm           1/1       Running     0          1m        10.48.2.12   gke-captioned-images-app-default-pool-70b2da38-6k20
$ kubectl describe ing
Name:             captioned-images-ipv4-ingress
Namespace:        default
Address:          35.201.64.7
Default backend:  default-http-backend:80 (10.48.0.5:8080)
Rules:
  Host                                    Path  Backends
  ----                                    ----  --------
  captioned-images.abevoelker.com         
                                                                                                                    captioned-images-web:80 (<none>)
                                          /.well-known/acme-challenge/peYKD0frmNK0ufteWVZHXUuuilKFzmcbQTYSm2bzz3M   cm-captioned-images-tls-zhqbx:8089 (<none>)
  assets-captioned-images.abevoelker.com  
                                                                                                                    captioned-images-assets:80 (<none>)
                                          /.well-known/acme-challenge/yZekTtuiJAgpQBB-S-SnBS_PbFgFKKAEDWIl82vIW_Q   cm-captioned-images-tls-hfqnz:8089 (<none>)
Annotations:
  target-proxy:     k8s-tp-default-captioned-images-ipv4-ingress--3db116602c6fe7c2
  url-map:          k8s-um-default-captioned-images-ipv4-ingress--3db116602c6fe7c2
  backends:         {"k8s-be-30162--3db116602c6fe7c2":"Unknown","k8s-be-30882--3db116602c6fe7c2":"HEALTHY","k8s-be-30976--3db116602c6fe7c2":"Unknown","k8s-be-31095--3db116602c6fe7c2":"HEALTHY","k8s-be-31821--3db116602c6fe7c2":"HEALTHY"}
  forwarding-rule:  k8s-fw-default-captioned-images-ipv4-ingress--3db116602c6fe7c2
Events:
  Type     Reason   Age                From                     Message
  ----     ------   ----               ----                     -------
  Warning  Service  25m                loadbalancer-controller  Could not find nodeport for backend {ServiceName:cm-captioned-images-tls-iwnfa ServicePort:{Type:0 IntVal:8089 StrVal:}}: service default/cm-captioned-images-tls-iwnfa not found in store
  Warning  Service  18m                loadbalancer-controller  Could not find nodeport for backend {ServiceName:cm-captioned-images-tls-prcvz ServicePort:{Type:0 IntVal:8089 StrVal:}}: service default/cm-captioned-images-tls-prcvz not found in store
  Warning  Service  11m                loadbalancer-controller  Could not find nodeport for backend {ServiceName:cm-captioned-images-tls-mirsu ServicePort:{Type:0 IntVal:8089 StrVal:}}: service default/cm-captioned-images-tls-mirsu not found in store
  Warning  Service  2m                 loadbalancer-controller  Could not find nodeport for backend {ServiceName:cm-captioned-images-tls-smoeg ServicePort:{Type:0 IntVal:8089 StrVal:}}: service default/cm-captioned-images-tls-smoeg not found in store
  Normal   Service  1m (x354 over 2d)  loadbalancer-controller  no user specified default backend, using system default


Name:             captioned-images-ipv6-ingress
Namespace:        default
Address:          2600:1901:0:439d::
Default backend:  default-http-backend:80 (10.48.0.5:8080)
Rules:
  Host                                    Path  Backends
  ----                                    ----  --------
  captioned-images.abevoelker.com         
                                             captioned-images-web:80 (<none>)
  assets-captioned-images.abevoelker.com  
                                             captioned-images-assets:80 (<none>)
Annotations:
  forwarding-rule:  k8s-fw-default-captioned-images-ipv6-ingress--3db116602c6fe7c2
  target-proxy:     k8s-tp-default-captioned-images-ipv6-ingress--3db116602c6fe7c2
  url-map:          k8s-um-default-captioned-images-ipv6-ingress--3db116602c6fe7c2
  backends:         {"k8s-be-30882--3db116602c6fe7c2":"HEALTHY","k8s-be-31095--3db116602c6fe7c2":"HEALTHY","k8s-be-31821--3db116602c6fe7c2":"HEALTHY"}
Events:
  Type    Reason   Age                From                     Message
  ----    ------   ----               ----                     -------
  Normal  Service  5m (x338 over 2d)  loadbalancer-controller  no user specified default backend, using system default

Here's my issuer.yml, certificate.yml, ingress.yml ... not sure what else is helpful

(Throwaway project so no redactions needed)

Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.4", GitCommit:"bee2d1505c4fe820744d26d41ecd3fdd4a3d6546", GitTreeState:"clean", BuildDate:"2018-03-12T16:29:47Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.8-gke.0", GitCommit:"6e5b33a290a99c067003632e0fd6be0ead48b233", GitTreeState:"clean", BuildDate:"2018-02-16T18:26:58Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: GKE

Install tools: Followed these instructions: https://github.com/ahmetb/gke-letsencrypt/

@abevoelker

This comment has been minimized.

abevoelker commented Apr 4, 2018

Anything else I can provide to help debug this?

I see "Got successful challenge request, writing key" many times in my GKE logs:

E  2018/04/04 17:33:12 [captioned-images.abevoelker.com] Responding OK to health check '/'
E  2018/04/04 17:33:12 [captioned-images.abevoelker.com] Responding OK to health check '/'
E  2018/04/04 17:33:12 [captioned-images.abevoelker.com] Validating request. basePath=/.well-known/acme-challenge, token=qB7qTGkLRJgvZxzsT1liqSqn7txkpqvqpj-IphBu7ds
E  2018/04/04 17:33:12 [captioned-images.abevoelker.com] Comparing actual host 'captioned-images.abevoelker.com' against expected 'captioned-images.abevoelker.com'
E  2018/04/04 17:33:12 [captioned-images.abevoelker.com] Got successful challenge request, writing key...
E  2018/04/04 17:33:12 [captioned-images.abevoelker.com] Responding OK to health check '/'
E  2018/04/04 17:33:12 [captioned-images.abevoelker.com] Responding OK to health check '/'

But the cert-manager pod logs show "error waiting for authorization" errors

I0404 17:33:17.917221       1 helpers.go:165] Setting lastTransitionTime for Certificate "captioned-images-tls" condition "Ready" to 2018-04-04 17:33:17.917200666 +0000 UTC m=+2075.300618827
I0404 17:33:17.918350       1 sync.go:242] Error preparing issuer for certificate: [error waiting for authorization for domain "assets-captioned-images.abevoelker.com": acme: authorization error for : , error waiting for authorization for domain "captioned-images.abevoelker.com": acme: authorization error for : ]
E0404 17:33:17.927408       1 sync.go:190] [default/captioned-images-tls] Error getting certificate 'captioned-images-tls': secret "captioned-images-tls" not found
E0404 17:33:17.927684       1 controller.go:196] certificates controller: Re-queuing item "default/captioned-images-tls" due to error processing: [error waiting for authorization for domain "assets-captioned-images.abevoelker.com": acme: authorization error for : , error waiting for authorization for domain "captioned-images.abevoelker.com": acme: authorization error for : ]
I0404 17:33:17.927824       1 controller.go:187] certificates controller: syncing item 'default/captioned-images-tls'
I0404 17:33:17.927952       1 sync.go:107] Error checking existing TLS certificate: secret "captioned-images-tls" not found
I0404 17:33:17.928091       1 sync.go:238] Preparing certificate with issuer
I0404 17:33:17.928892       1 prepare.go:239] Compare "" with "https://acme-v01.api.letsencrypt.org/acme/reg/32447217"
I0404 17:33:17.929565       1 prepare.go:239] Compare "" with "https://acme-v01.api.letsencrypt.org/acme/reg/32447217"
@abevoelker

This comment has been minimized.

abevoelker commented Apr 4, 2018

I also tried creating the Certificate using ingressClass: nginx but that hasn't worked either; probably predictably since my site Ingresses are using the GLBC load balancer with static IP addresses

$ kubectl get ing
NAME                            HOSTS                                                                    ADDRESS            PORTS     AGE
captioned-images-ipv4-ingress   captioned-images.abevoelker.com,assets-captioned-images.abevoelker.com   130.211.47.102     80        8d
captioned-images-ipv6-ingress   captioned-images.abevoelker.com,assets-captioned-images.abevoelker.com   2600:1901:0:f...   80        8d
cm-captioned-images-tls-agodk   assets-captioned-images.abevoelker.com                                                      80        4m
cm-captioned-images-tls-nnqdb   captioned-images.abevoelker.com                                                             80        4m
I0404 18:09:23.016456       1 helpers.go:165] Setting lastTransitionTime for Certificate "captioned-images-tls" condition "Ready" to 2018-04-04 18:09:23.016377632 +0000 UTC m=+4240.399795785
I0404 18:09:23.016535       1 sync.go:242] Error preparing issuer for certificate: [error waiting for key to be available for domain "captioned-images.abevoelker.com": context deadline exceeded, error waiting for key to be available for domain "assets-captioned-images.abevoelker.com": context deadline exceeded]
E0404 18:09:23.027129       1 sync.go:190] [default/captioned-images-tls] Error getting certificate 'captioned-images-tls': secret "captioned-images-tls" not found
E0404 18:09:23.027183       1 controller.go:196] certificates controller: Re-queuing item "default/captioned-images-tls" due to error processing: [error waiting for key to be available for domain "captioned-images.abevoelker.com": context deadline exceeded, error waiting for key to be available for domain "assets-captioned-images.abevoelker.com": context deadline exceeded]
I0404 18:09:23.027212       1 controller.go:187] certificates controller: syncing item 'default/captioned-images-tls'
I0404 18:09:23.027260       1 sync.go:107] Error checking existing TLS certificate: secret "captioned-images-tls" not found
I0404 18:09:23.027348       1 sync.go:238] Preparing certificate with issuer
I0404 18:09:23.029402       1 prepare.go:239] Compare "" with "https://acme-v01.api.letsencrypt.org/acme/reg/32447217"
I0404 18:09:23.029565       1 prepare.go:239] Compare "" with "https://acme-v01.api.letsencrypt.org/acme/reg/32447217"
I0404 18:24:28.702385       1 sync.go:242] Error preparing issuer for certificate: [error waiting for key to be available for domain "assets-captioned-images.abevoelker.com": context deadline exceeded, error waiting for key to be available for domain "captioned-images.abevoelker.com": context deadline exceeded]
E0404 18:24:28.724243       1 sync.go:190] [default/captioned-images-tls] Error getting certificate 'captioned-images-tls': secret "captioned-images-tls" not found
E0404 18:24:28.724894       1 controller.go:196] certificates controller: Re-queuing item "default/captioned-images-tls" due to error processing: [error waiting for key to be available for domain "assets-captioned-images.abevoelker.com": context deadline exceeded, error waiting for key to be available for domain "captioned-images.abevoelker.com": context deadline exceeded]
I0404 18:24:28.726046       1 controller.go:187] certificates controller: syncing item 'default/captioned-images-tls'
I0404 18:24:28.726446       1 sync.go:107] Error checking existing TLS certificate: secret "captioned-images-tls" not found
I0404 18:24:28.726903       1 sync.go:238] Preparing certificate with issuer
I0404 18:24:28.727782       1 prepare.go:239] Compare "" with "https://acme-v01.api.letsencrypt.org/acme/reg/32447217"
I0404 18:24:28.728491       1 prepare.go:239] Compare "" with "https://acme-v01.api.letsencrypt.org/acme/reg/32447217"

$ kubectl describe certificate output

$ kubectl describe ing output

If I should open a new issue or if this is not valuable info let me know.

@boykom

This comment has been minimized.

boykom commented May 2, 2018

Hi there,

I can confirm the same issue.

@munnerz

This comment has been minimized.

Member

munnerz commented Jun 5, 2018

Are you still seeing this issue with cert-manager v0.3.0?

@m1kola

This comment has been minimized.

m1kola commented Jul 18, 2018

@munnerz I didn't try v0.3.0, but I'm having this issue with cert-manager v0.4.0 on Kubernetes v1.10.2-gke.3 with Google Cloud Load Balancer.

service default/cm-bla-bla-com-qwtvo not found in store implies that the service 'cm-bla-bla-com-qwtvo' in namespace 'default' no longer exists, which is expected as the validation has completed successfully (so cert-manager has cleaned up the resources it created to validate the challenge).

I don't have much details, at the moment (trying to find any traces like logs), but it seems like cert-manager cleaned up resources (services and a pod), but didn't update the ingress resource: I see see the /.well-known/acme-challenge/bla-bla-bla pointing to the cm-acme-http-solver-j9p82:8089 services that doesn't really exist.

Here are example manifests that I use to get a certificate:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "some-loadbalancer-ip"
  name: some_app
  namespace: some_app_namespace
spec:
  backend:
    serviceName: some_app
    servicePort: 80
  tls:
  - secretName: some_app-letsencrypt
    hosts:
    - some.app.com

---

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: some_app-tls
  namespace: some_app_namespace
spec:
  secretName: some_app-letsencrypt
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: some.app.com
  dnsNames:
  - some.app.com
  acme:
    config:
    - http01:
        ingress: app
      domains:
      - some.app.com
@shalkam

This comment has been minimized.

shalkam commented Oct 9, 2018

in my case I see that gke is still referencing the cm- services ...

  rules:
  - host: www.domain.io
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-zh285
          servicePort: 8089
        path: /.well-known/acme-challenge/Xf46CzbeX7OXfmx0U_ULMTK-sQ67qWSEqs6vh3NeJ_s
  - host: domain.io
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-xd6hg
          servicePort: 8089
        path: /.well-known/acme-challenge/mNt42U_E8HLoZ6wwargrpa4766eSDjI5d7UUwcG883U

even that my local ingress file doesn't have this part

@m1kola

This comment has been minimized.

m1kola commented Oct 9, 2018

@shalkam this looks very similar to the issues I mentioned before. I think, it was fixed in #831. At least, I do not see this in v0.5.0.

Which version do you use? If it's less than 0.5.0, try to update and let us know if it helps.

@shalkam

This comment has been minimized.

shalkam commented Oct 9, 2018

@m1kola It's 0.4 ... it's because I was following this tutorial here https://github.com/ahmetb/gke-letsencrypt/blob/master/20-install-cert-manager.md

I will try upgrading, and see if the problem persists

@m1kola

This comment has been minimized.

m1kola commented Oct 9, 2018

Thanks! :)

@munnerz I think, you fixed this issue it in #831. We can probably close it. Especially if we get another confirmation from @shalkam or someone else who tried it with 0.5.0

@shalkam

This comment has been minimized.

shalkam commented Oct 9, 2018

@m1kola
Okay, I gave it a try using v0.5.0 and there is no mentioning of any cm-acme-http-solver service, and I have my certificate successfully issued.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment