Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ingress addon: "default backend - 404" when HTTPS is used. #1701

Open
veqryn opened this issue Jul 15, 2017 · 21 comments

Comments

@veqryn
Copy link

commented Jul 15, 2017

BUG REPORT

Minikube version: v0.20.0

Environment:

  • OS: Windows 10 Pro (Anniversary Edition)
  • VM Driver: hyperv
  • ISO version: minikube-v0.20.0.iso

What happened:

$ kubectl run hello-world --image=tutum/hello-world:latest --port=80
deployment "hello-world" created

$ kubectl expose deployment hello-world --type=NodePort
service "hello-world" exposed

$ curl $(minikube service hello-world --url)
<html>
<head>
        <title>Hello world!</title>
...

$ minikube addons enable ingress
ingress was successfully enabled

$ curl $(minikube ip)
default backend - 404

So far so good. Created a file minikube-ingress.yaml as so:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: minikube-ingress
  annotations:
spec:
  rules:
  - host: hello.world
    http:
      paths:
      - path: /*
        backend:
          serviceName: hello-world
          servicePort: 80

Added my minikube ip to my hosts file: 192.168.0.25 hello.world

$ kubectl apply -f minikube-ingress.yaml
ingress "minikube-ingress" created

$ curl http://hello.world
<html>
<head>
        <title>Hello world!</title>
...

$ curl -k https://hello.world
default backend - 404

Ok, so I should be able to hit the hello-world service with https. Supposedly the default configuration of the ingress is SSL Termination, with some auto-generated self-signed certs.
I did try hitting this in my browser as well, where I was asked to accept the cert first.
But no matter what, it still goes to the 'default backend' instead of my hello-world service.

Also tried the following configuration:

apiVersion: v1
kind: Secret
metadata:
  name: minikube-ingress-secret
  namespace: default
type: Opaque
data:
  tls.crt: LS0t...
  tls.key: LS0t...

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: minikube-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - secretName: minikube-ingress-secret
  backend:
    serviceName: hello-world
    servicePort: 80

The above ingress definition completely ignores whatever I put in "backend" and just sends everything to the "default backend 404".
I also tried adding the rule from the first config into this, and several other permutations, all with the same results.
I also tried deleting all the resources and creating them again, with no luck.

I've tried deleting all my dns and ingress pods, to see if when they came back it would work. That did not help.

What you expected to happen:
I would expect that I could curl https://hello.world and get back my hello world html results with 200, instead of going to default-backend-404.

I would also expect that I could overwrite what the default backend is by specifying the backend block in the above spec, but it is getting compltely ignored.

And I would also expect that the ssl-redirect would work, yet it seems to also be ignored (http continues to work just fine).

@carlpett

This comment has been minimized.

Copy link

commented Nov 6, 2017

I'm seeing the same thing. Interestingly, if I kubectl exec -it <ingress-controller-pod> bash and then curl from there, it works. So it seems something is mangling the request before it arrives at the ingress controller?

@paulczar

This comment has been minimized.

Copy link

commented Nov 8, 2017

I can confirm I'm also seeing this. not only does it 404, but it appears to ignore my TLS certs from secret.

@paulczar

This comment has been minimized.

Copy link

commented Nov 8, 2017

it seems adding the following made it work for me:

spec:
  tls:
  - hosts:
    - hello.world
  secretName: minikube-ingress-secret
@carlpett

This comment has been minimized.

Copy link

commented Nov 8, 2017

I'm wondering if minikube addons enable ingress creates some sort of hidden ingress somewhere? Because if I do kubectl port-forward to the ingress and point to localhost, it also works as intended...

@fejta-bot

This comment has been minimized.

Copy link

commented Feb 7, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

commented Mar 9, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@elgalu

This comment has been minimized.

Copy link

commented Mar 15, 2018

/remove-lifecycle rotten

@snorthov

This comment has been minimized.

Copy link

commented Apr 22, 2018

Does minikube ingress support https? No matter what I do, I cannot get it to work (I can get http to work). Is there a tutorial anywhere for minikube ingress that shows how to get https to work?

@fejta-bot

This comment has been minimized.

Copy link

commented Jul 21, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@darkedges

This comment has been minimized.

Copy link
Contributor

commented Jul 21, 2018

I am facing the same issue. my ingress is https://gist.github.com/darkedges/80def8628fa3faa5bb13f0c5d00ed36c

I know it is working as when I hit the http:// address it redirects to https://, but I get the backend not found response.

If I kubectl exec -it -n kube-system nginx-ingress-controller-67956bf89d-5c9zx bash I can see my certificates are there

root@nginx-ingress-controller-67956bf89d-5c9zx:/ingress-controller/ssl# ls -lrt
total 20
-rw------- 1 root root 2933 Jul 21 19:48 default-fake-certificate.pem
-rw-r--r-- 1 root root 1659 Jul 21 19:48 default-darkedges.com-full-chain.pem
-rw------- 1 root root 3364 Jul 21 21:18 default-darkedges.com.pem
-rw-r--r-- 1 root root 1659 Jul 21 21:19 default-darkedges-com-tls-full-chain.pem
-rw------- 1 root root 3364 Jul 21 21:58 default-darkedges-com-tls.pem

but in /etc/nginx.conf I can see all my host entries on port 80, but nothing on 443.

Any ideas on what I am doing wrong?

Edit:

Found this via kubectl logs -n kube-system nginx-ingress-controller-67956bf89d-5c9zx

W0721 22:34:37.322162       6 controller.go:1032] ssl certificate default/darkedges-com-tls does not contain a Common Name or Subject Alternative Name for host as.tpp.forgerockdev.darkedges.com. Reason: x509: certificate is valid for *.darkedges.com, darkedges.com, not as.tpp.forgerockdev.darkedges.com
W0721 22:34:37.322388       6 controller.go:1026] unexpected error validating SSL certificate default/darkedges-com-tls for host as.bank.forgerockdev.darkedges.com. Reason: x509: certificate is valid for *.darkedges.com, darkedges.com, not as.bank.forgerockdev.darkedges.com```

So mine is an issue with my certificates not matching.
@jainishshah17

This comment has been minimized.

Copy link

commented Aug 17, 2018

I am having same issue

@fejta-bot

This comment has been minimized.

Copy link

commented Sep 16, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@snorthov

This comment has been minimized.

Copy link

commented Sep 17, 2018

Still happening. Please don't close. Can someone provide an example where this is working?

@tstromberg tstromberg changed the title Ingress HTTPS does not work (always gives default-backend-404) ingress addon: "default backend - 404" when HTTPS is used. Sep 19, 2018

@ametad

This comment has been minimized.

Copy link

commented Oct 10, 2018

A working example would be great indeed. Dropping a line to follow this...

@ametad

This comment has been minimized.

Copy link

commented Oct 10, 2018

This is working (well... almost):

it seems adding the following made it work for me:

spec:
  tls:
  - hosts:
    - hello.world
  secretName: minikube-ingress-secret

Little change:

spec:
  tls:
  - hosts:
    - hello.world
  - secretName: minikube-ingress-secret

(Notice the last line)

@sccoe

This comment has been minimized.

Copy link

commented Dec 28, 2018

Sweet, the - secretName fixed it for me too

@tstromberg

This comment has been minimized.

Copy link
Contributor

commented Jan 24, 2019

The workarounds seem reasonable. Help wanted to get this into the addon itself!

@snorthov

This comment has been minimized.

Copy link

commented Jan 24, 2019

Hilarious! After moving on to other things and months passing, I decided to try out the work around today. I was just about to report back here that is was working and tstromberg commented.

There is a Software God (or Gods) and a Single Global Cache!

@wavilov

This comment has been minimized.

Copy link

commented Mar 6, 2019

Sweet, the - secretName fixed it for me too
Mee too! After an evening of struggling with 404. Thanks, ametad!

@wavilov

This comment has been minimized.

Copy link

commented Mar 6, 2019

This is working (well... almost):

it seems adding the following made it work for me:

spec:
  tls:
  - hosts:
    - hello.world
  secretName: minikube-ingress-secret

Little change:

spec:
  tls:
  - hosts:
    - hello.world
  - secretName: minikube-ingress-secret

(Notice the last line)

New info! Ingress config change will force ingress to use fake certificate. Site will work but with big browser ssl alarm.

In my case true reason has been in certificate, that was stored in another kubernetes' namespace from ingress. After I put certificate in same namespace, HTTPS worked with default ingress config.

Idea taken from this commentary
kubernetes/ingress-nginx#1984 (comment)

@tstromberg tstromberg added the r/2019q2 label May 22, 2019

@fejta-bot

This comment has been minimized.

Copy link

commented Aug 20, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.