Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nginx returns the wrong certificate intermittently #4691

Open
juanxo opened this issue Oct 17, 2019 · 3 comments

Comments

@juanxo
Copy link

@juanxo juanxo commented Oct 17, 2019

Is this a request for help?: Yes

What keywords did you search in NGINX Ingress controller issues before filing this one?: ssl local resolution


Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

NGINX Ingress controller version: 0.26.1

Kubernetes version: 1.16.1 client/ 1.14 server

Environment:

  • Cloud provider or hardware configuration: AWS

What happened:
Using the default template for nginx, and a SSL certificate from LetsEncrypt obtained thru cert-manager, we have a service A that performs a request using https to another service B thru a domain that points to a balancer for service B.

Both services are in the same pod.

Intermittently, requests from service A to service B will fail. Trying to reproduce this error by doing manual requests outside of the pod with the following command never throws that same error

while true; do curl -I <domain>; sleep 2; done

But if we execute this same command inside the ingress-nginx pod, it will start throwing intermittent errors:

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
curl: (60) SSL certificate problem: unable to get local issuer certificate

Analyzing the openssl output, we see that sometimes its only returning the fake default certificate instead of the LetsEncrypt one, and those cases are the ones raising errors:

> openssl s_client -connect <domain>:443 -servername <domain>
CONNECTED(00000003)
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
...
> openssl s_client -connect <domain>:443 -servername <domain>
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
...

What you expected to happen:

The ssl certificate used for every request, both local and external, would be the LetsEncrypt one.

@ElvinEfendi

This comment has been minimized.

Copy link
Member

@ElvinEfendi ElvinEfendi commented Oct 18, 2019

Do you see this behaviour with only 0.26.1? Can you try 0.25.1 and see what happens with that.

Also please post the Nginx logs.

Most importantly the best possible way to make sure these kind of issues get addressed is to PR an e2e test that fails because of this.

@scherkan

This comment has been minimized.

Copy link

@scherkan scherkan commented Oct 21, 2019

I had the same problem on 0.26.1. After downgrade to 0.25.1 everything is OK.

On 0.26.1 it looks like the challenge URL wasn't available, so no certificate could be obtained from cert-manager. So there was only private key in Certificate resource(thus fake default certificate was received).

@pedrogimenez

This comment has been minimized.

Copy link

@pedrogimenez pedrogimenez commented Oct 21, 2019

In our case, it was a problem with our server resources. We needed more nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.