Could not find nodeport for backend #281
Comments
|
That seems like a potentially old, transient error.
Are you seeing any actual problems, aside from this error message? Is the error message being printed continuously or the like? Can you also provide the output of: I'd expect the |
|
I'm running into the same issue.
Yes, the ingress is not binding to its static IP and is "stuck" on "Creating ingress"
Yes the error message happens continuously on low frequency Here is the output from above command @korovaisdead were you able to resolve this and if so how. |
|
Also seeing this issue. The challenge validation does not complete (no secret in $ kubectl get ing,svc,po -o wide -n default
NAME HOSTS ADDRESS PORTS AGE
ing/captioned-images-ipv4-ingress captioned-images.abevoelker.com,assets-captioned-images.abevoelker.com 35.201.64.7 80 2d
ing/captioned-images-ipv6-ingress captioned-images.abevoelker.com,assets-captioned-images.abevoelker.com 2600:1901:0:439d:: 80 2d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
svc/captioned-images-assets NodePort 10.51.247.191 <none> 80:31095/TCP 2d run=captioned-images-web
svc/captioned-images-web NodePort 10.51.251.87 <none> 80:31821/TCP 2d run=captioned-images-web
svc/cm-captioned-images-tls-hfqnz NodePort 10.51.241.16 <none> 8089:30162/TCP 1m certmanager.k8s.io/certificate=captioned-images-tls,certmanager.k8s.io/domain=assets-captioned-images.abevoelker.com,certmanager.k8s.io/id=gvkkm,certmanager.k8s.io/managed=true
svc/cm-captioned-images-tls-zhqbx NodePort 10.51.240.182 <none> 8089:30976/TCP 1m certmanager.k8s.io/certificate=captioned-images-tls,certmanager.k8s.io/domain=captioned-images.abevoelker.com,certmanager.k8s.io/id=deuqz,certmanager.k8s.io/managed=true
svc/kubernetes ClusterIP 10.51.240.1 <none> 443/TCP 3d <none>
NAME READY STATUS RESTARTS AGE IP NODE
po/captioned-images-db-migrate-qzkdt 1/2 Completed 0 2d 10.48.2.8 gke-captioned-images-app-default-pool-70b2da38-6k20
po/captioned-images-web-5b5b686768-hh5hw 3/3 Running 0 55m 10.48.4.7 gke-captioned-images-app-default-pool-70b2da38-sd7s
po/captioned-images-web-5b5b686768-wbqh9 3/3 Running 0 55m 10.48.4.8 gke-captioned-images-app-default-pool-70b2da38-sd7s
po/cm-captioned-images-tls-dasls 1/1 Running 0 1m 10.48.0.15 gke-captioned-images-app-default-pool-70b2da38-bk19
po/cm-captioned-images-tls-odqrm 1/1 Running 0 1m 10.48.2.12 gke-captioned-images-app-default-pool-70b2da38-6k20$ kubectl describe ing
Name: captioned-images-ipv4-ingress
Namespace: default
Address: 35.201.64.7
Default backend: default-http-backend:80 (10.48.0.5:8080)
Rules:
Host Path Backends
---- ---- --------
captioned-images.abevoelker.com
captioned-images-web:80 (<none>)
/.well-known/acme-challenge/peYKD0frmNK0ufteWVZHXUuuilKFzmcbQTYSm2bzz3M cm-captioned-images-tls-zhqbx:8089 (<none>)
assets-captioned-images.abevoelker.com
captioned-images-assets:80 (<none>)
/.well-known/acme-challenge/yZekTtuiJAgpQBB-S-SnBS_PbFgFKKAEDWIl82vIW_Q cm-captioned-images-tls-hfqnz:8089 (<none>)
Annotations:
target-proxy: k8s-tp-default-captioned-images-ipv4-ingress--3db116602c6fe7c2
url-map: k8s-um-default-captioned-images-ipv4-ingress--3db116602c6fe7c2
backends: {"k8s-be-30162--3db116602c6fe7c2":"Unknown","k8s-be-30882--3db116602c6fe7c2":"HEALTHY","k8s-be-30976--3db116602c6fe7c2":"Unknown","k8s-be-31095--3db116602c6fe7c2":"HEALTHY","k8s-be-31821--3db116602c6fe7c2":"HEALTHY"}
forwarding-rule: k8s-fw-default-captioned-images-ipv4-ingress--3db116602c6fe7c2
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Service 25m loadbalancer-controller Could not find nodeport for backend {ServiceName:cm-captioned-images-tls-iwnfa ServicePort:{Type:0 IntVal:8089 StrVal:}}: service default/cm-captioned-images-tls-iwnfa not found in store
Warning Service 18m loadbalancer-controller Could not find nodeport for backend {ServiceName:cm-captioned-images-tls-prcvz ServicePort:{Type:0 IntVal:8089 StrVal:}}: service default/cm-captioned-images-tls-prcvz not found in store
Warning Service 11m loadbalancer-controller Could not find nodeport for backend {ServiceName:cm-captioned-images-tls-mirsu ServicePort:{Type:0 IntVal:8089 StrVal:}}: service default/cm-captioned-images-tls-mirsu not found in store
Warning Service 2m loadbalancer-controller Could not find nodeport for backend {ServiceName:cm-captioned-images-tls-smoeg ServicePort:{Type:0 IntVal:8089 StrVal:}}: service default/cm-captioned-images-tls-smoeg not found in store
Normal Service 1m (x354 over 2d) loadbalancer-controller no user specified default backend, using system default
Name: captioned-images-ipv6-ingress
Namespace: default
Address: 2600:1901:0:439d::
Default backend: default-http-backend:80 (10.48.0.5:8080)
Rules:
Host Path Backends
---- ---- --------
captioned-images.abevoelker.com
captioned-images-web:80 (<none>)
assets-captioned-images.abevoelker.com
captioned-images-assets:80 (<none>)
Annotations:
forwarding-rule: k8s-fw-default-captioned-images-ipv6-ingress--3db116602c6fe7c2
target-proxy: k8s-tp-default-captioned-images-ipv6-ingress--3db116602c6fe7c2
url-map: k8s-um-default-captioned-images-ipv6-ingress--3db116602c6fe7c2
backends: {"k8s-be-30882--3db116602c6fe7c2":"HEALTHY","k8s-be-31095--3db116602c6fe7c2":"HEALTHY","k8s-be-31821--3db116602c6fe7c2":"HEALTHY"}
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Service 5m (x338 over 2d) loadbalancer-controller no user specified default backend, using system defaultHere's my issuer.yml, certificate.yml, ingress.yml ... not sure what else is helpful (Throwaway project so no redactions needed) Kubernetes version (use kubectl version): $ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.4", GitCommit:"bee2d1505c4fe820744d26d41ecd3fdd4a3d6546", GitTreeState:"clean", BuildDate:"2018-03-12T16:29:47Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.8-gke.0", GitCommit:"6e5b33a290a99c067003632e0fd6be0ead48b233", GitTreeState:"clean", BuildDate:"2018-02-16T18:26:58Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}Cloud provider or hardware configuration: GKE Install tools: Followed these instructions: https://github.com/ahmetb/gke-letsencrypt/ |
|
Anything else I can provide to help debug this? I see "Got successful challenge request, writing key" many times in my GKE logs: But the cert-manager pod logs show "error waiting for authorization" errors |
|
I also tried creating the Certificate using $ kubectl get ing
NAME HOSTS ADDRESS PORTS AGE
captioned-images-ipv4-ingress captioned-images.abevoelker.com,assets-captioned-images.abevoelker.com 130.211.47.102 80 8d
captioned-images-ipv6-ingress captioned-images.abevoelker.com,assets-captioned-images.abevoelker.com 2600:1901:0:f... 80 8d
cm-captioned-images-tls-agodk assets-captioned-images.abevoelker.com 80 4m
cm-captioned-images-tls-nnqdb captioned-images.abevoelker.com 80 4m
If I should open a new issue or if this is not valuable info let me know. |
|
Hi there, I can confirm the same issue. |
|
Are you still seeing this issue with cert-manager v0.3.0? |
munnerz
added
the
triage/needs-information
|
@munnerz I didn't try v0.3.0, but I'm having this issue with cert-manager v0.4.0 on Kubernetes v1.10.2-gke.3 with Google Cloud Load Balancer.
I don't have much details, at the moment (trying to find any traces like logs), but it seems like Here are example manifests that I use to get a certificate: apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.global-static-ip-name: "some-loadbalancer-ip"
name: some_app
namespace: some_app_namespace
spec:
backend:
serviceName: some_app
servicePort: 80
tls:
- secretName: some_app-letsencrypt
hosts:
- some.app.com
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: some_app-tls
namespace: some_app_namespace
spec:
secretName: some_app-letsencrypt
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
commonName: some.app.com
dnsNames:
- some.app.com
acme:
config:
- http01:
ingress: app
domains:
- some.app.com |
munnerz
added
kind/bug
|
in my case I see that gke is still referencing the even that my local ingress file doesn't have this part |
|
@m1kola It's 0.4 ... it's because I was following this tutorial here https://github.com/ahmetb/gke-letsencrypt/blob/master/20-install-cert-manager.md I will try upgrading, and see if the problem persists |
|
@m1kola |
|
Issues go stale after 90d of inactivity. |
jetstack-bot
added
the
lifecycle/stale
After creating the all required resources I've got the keypair stored inside the secret.
But the ingress shows an error:
Could not find nodeport for backend {ServiceName:cm-bla-bla-com-qwtvo ServicePort:{Type:0 IntVal:8089 StrVal:}}: service default/cm-bla-bla-com-qwtvo not found in storeEnvironment:
kubectl version): 1.9The text was updated successfully, but these errors were encountered: