Upstream SSL certificate verify error on JWT validation

[mixerka]

I am using Google Cloud Endpoints with container engine.

My securityDefinition is:

  ridero_jwt:
    authorizationUrl: ""
    flow: "implicit"
    type: "oauth2"
    x-google-issuer: "https://pg.ridero.store"
    x-google-jwks_uri: "https://pg.ridero.store/auth/certs"
    x-google-audiences: "store-admin-pg"

When I am trying to use JWT token, I get the following error in ESP stdout:

2017/09/08 04:28:17 [error] 9#9: upstream SSL certificate verify error: (18:self signed certificate)
10.60.3.1 - - [08/Sep/2017:04:28:17 +0000] "POST /v1/sku HTTP/1.0" 401 212 "-" "curl/7.54.0"

In response I get:

{
 "code": 16,
 "message": "JWT validation failed: Unable to fetch verification key",
 "details": [
  {
   "@type": "type.googleapis.com/google.rpc.DebugInfo",
   "stackEntries": [],
   "detail": "auth"
  }
 ]
}

In tracer I get:

I have tried to connect into esp container with "kubectl run", install curl and make request and there was no errors.
I have checked trusted-ca-certificates.crt and it contains my CA: DST Root CA X3.
I have tried to set x-google-jwks_uri to http://auth.default.svc.cluster.local/certs, but esp couldn't resolve domain name.

[qiwzhang]

@lizan @Kuat Nginx experts, could you help? I think Nginx should use our build-in trusted-ca-certificates.crt for server certificates. http.cc used to fetch public should use the same certs. is that correct?

Hi @liminw could you try to use local Nginx to test it to see if it can talk to that server?

[liminw]

I see two errors here: JWT validation failed, and SSL certificate verify error.

For JWT validation failure, what "kid" did you use in your JWT? From https://pg.ridero.store/auth/certs, you try to support JWT signed by different signing algorithms, including ES256, ES384, ES521, RS256. ESP currently supports RS (e.g., RS256) and HS algorithms. In the next ESP release, ESP will start supporting ES256.

For SSL certificate error, it seems that the error complains that you have a self-signed certificate. Can you check to make sure that your certificate is not self-signed?

[mixerka]

@liminw Thanks for your questions.

I definitely use RS256 and as you could see https://pg.ridero.store signed by let's encrypt with DST Root CA X3.

Actually, first error you mentioned is "JWT validation failed: Unable to fetch verification key". There is no conflict between "unable to fetch verification key" and "ssl certificate verify error". Is it possible, that SSL error is reason for unable to fetch error?

[liminw]

Sorry for responding late. I somehow missed the message.

I think you are right that in your case the JWT validation error is caused by the SSL error. Can you check if your certificate is self-signed?

[mixerka]

@liminw
My certificate is not self-signed: https://www.ssllabs.com/ssltest/analyze.html?d=pg.ridero.store&latest

[qiwzhang]

@lizan could you help to debug Nginx for this one?

if https://pg.ridero.store is not self-signed, why Nginx said it is self-signed. Our certification file is here

https://github.com/cloudendpoints/endpoints-tools/blob/master/start_esp/trusted-ca-certificates.crt

[lizan]

Looks like the https://pg.ridero.store requires SNI to return proper certificate, and the HTTPS client in ESP doesn't support that.

$ openssl s_client -showcerts -connect pg.ridero.store:443

will result

    Verify return code: 18 (self signed certificate)

While

$ openssl s_client -showcerts -servername pg.ridero.store -connect pg.ridero.store:443

will return verify OK. Filed #271.

[lizan]

I have tried to set x-google-jwks_uri to http://auth.default.svc.cluster.local/certs, but esp couldn't resolve domain name.

@mixerka This is because by default ESP uses Public DNS resolver (8.8.8.8), you can use --dns flag to set it to your K8s cluster DNS.

[lizan]

@mixerka Do you maintain https://pg.ridero.store by yourself? If the IP only serve pg.ridero.store, try not require SNI at server-side, that should work too.

[mixerka]

@lizan thank you for investigation.

For now, I will use k8s cluster dns and local dns name without https.

[lei-tang]

The support of SNI (Server Name Indication) for https clients has been added to ESP by the commit 3fe87fd.

[lizan]

ESP 1.20.0 was released and contains the fix. Let me know if this solves your issue.

[bourquep]

Works for me! I believe this issue can be closed.

[lei-tang]

@bourquep Thank you very much for your confirmation! The issue is closed.