Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cannot curl podinfo service - http 503 #80

Closed
itapai opened this issue Oct 3, 2022 · 1 comment
Closed

cannot curl podinfo service - http 503 #80

itapai opened this issue Oct 3, 2022 · 1 comment

Comments

@itapai
Copy link

itapai commented Oct 3, 2022

I forked this repo, added a new cluster in clusters/development and bootstraped flux on a local kind cluster.

flux bootstrap github  \
  --owner=itapai \
  --repository=flux2-kustomize-helm-example \
  --branch=master \
  --path=clusters/development \
  --personal

flux check

► checking prerequisites
✔ Kubernetes 1.25.2 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.25.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.29.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.27.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.30.0
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed


k get no -o wide

NAME                    STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
local-control-plane   Ready    control-plane   37m   v1.25.2   172.24.0.2    <none>        Ubuntu 22.04.1 LTS   5.19.12-arch1-1   containerd://1.6.8


k get svc -n nginx

NAME                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
nginx-ingress-controller                   NodePort    10.96.186.228   <none>        80:32642/TCP,443:30051/TCP   32m
nginx-ingress-controller-default-backend   ClusterIP   10.96.218.142   <none>        80/TCP                       32m

When I try to curl it with the url, I get this error:

curl 172.24.0.2:32642 -H "Host: podinfo.development"

<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx</center>
</body>
</html>

and this error log in the nginx pod

k logs -n nginx nginx-ingress-controller-598f55d78c-6pcbq

...
E1003 18:57:16.549786       1 reflector.go:140] k8s.io/client-go@v0.25.2/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:nginx:nginx-ingress-controller" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
W1003 18:57:47.749632       1 reflector.go:424] k8s.io/client-go@v0.25.2/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:nginx:nginx-ingress-controller" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
E1003 18:57:47.749654       1 reflector.go:140] k8s.io/client-go@v0.25.2/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:nginx:nginx-ingress-controller" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope

However, if I port forward and curl, it works.

 k port-forward -n podinfo svc/podinfo 8088:9898

 curl -H "Host: podinfo.development" localhost:8088
{
  "hostname": "podinfo-6cd678c96-tqqkg",
  "version": "6.2.1",
  "revision": "44157ecd84c0d78b17e4d7b43f2a7bb316372d6c",
  "color": "#34577c",
  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
  "message": "greetings from podinfo v6.2.1",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.19.1",
  "num_goroutine": "9",
  "num_cpu": "8"
}

I tried the same thing with a managed kubernetes service (digitalocean) with a LoadBalancer service, and had the same error in the nginx-ingress pod and curl returned Empty reply from server.
How can I access the service via url?
@itapai
Copy link
Author

itapai commented Oct 3, 2022

Adding this to the nginx-ingress-controller clusterrole fixed this for me.

- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch

Short

kubectl get clusterrole nginx-ingress-controller -o json | \
  jq '.rules += [{"apiGroups":["discovery.k8s.io"],"resources":["endpointslices"],"verbs":["list","watch"]}]' | \
  kubectl apply -f -

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@itapai @stefanprodan