Route certificate ignored

[mvlach]

When I changed the certificate for the edge termination type, certificate didn't changed. Router still serves the default certificate.

Version

oc v1.3.0
kubernetes v1.3.0+52492b4
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server ********
openshift v1.3.0
kubernetes v1.3.0+52492b4

Steps To Reproduce

1. create master + node from the ansible - specify the default router certificate
2. create a route
3. change route (specify other certificate)

Current Result

Old certificate is returned

Expected Result

The new certificate will be returned

[knobunc]

@weliang1 can you reproduce this please?

[mvlach]

Maybe this can help...

https://lists.openshift.redhat.com/openshift-archives/users/2016-October/msg00130.html

[knobunc]

Thanks @mvlach. @ramr can you see if your other changes will have resolved this?

[weliang1]

@knobunc let me try

[weliang1]

Follow above steps, I can replicate same issue in my env:

[root@ip-172-18-8-180 ~]# CA=/etc/origin/master
[root@ip-172-18-8-180 ~]# oadm ca create-server-cert --signer-cert=$CA/ca.crt \

  --signer-key=$CA/ca.key --signer-serial=$CA/ca.serial.txt \
  --hostnames='hello-openshift.com' \
  --cert=cloudapps.crt --key=cloudapps.key

[root@ip-172-18-8-180 ~]# cat cloudapps.crt cloudapps.key $CA/ca.crt > cloudapps.router.pem
[root@ip-172-18-8-180 ~]# oc new-project https
Now using project "https" on server "https://ip-172-18-8-180.ec2.internal:8443".

You can add applications to this project with the 'new-app' command. For example, try:

oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git

to build a new example application in Ruby.
[root@ip-172-18-8-180 ~]# oadm policy add-scc-to-user privileged -z https-user
[root@ip-172-18-8-180 ~]# oadm router router1 --replicas=1 --service-account=https-user -n https --host-network=true --default-cert=cloudapps.router.pem
info: password for stats user admin has been set to fJxKsjo0Cx
--> Creating router router1 ...
secret "router1-certs" created
serviceaccount "https-user" created
error: rolebinding "router-router1-role" already exists
deploymentconfig "router1" created
service "router1" created
--> Failed
[root@ip-172-18-8-180 ~]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/test50.json
route "route-1" created
service "endpoints" created
pod "endpoint-1" created
[root@ip-172-18-8-180 ~]#
[root@ip-172-18-8-180 ~]# curl --resolve hello-openshift.com:443:172.18.0.203 https://hello-openshift.com --cacert cloudapps.router.pem -vvI

• Added hello-openshift.com:443:172.18.0.203 to DNS cache
• About to connect() to hello-openshift.com port 443 (#0)
• Trying 172.18.0.203...
• Connected to hello-openshift.com (172.18.0.203) port 443 (#0)
• Initializing NSS with certpath: sql:/etc/pki/nssdb
• CAfile: cloudapps.router.pem
  CApath: none
• SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
• Server certificate:
• subject: CN=hello-openshift.com
• start date: Oct 14 20:29:39 2016 GMT
• expire date: Oct 14 20:29:40 2018 GMT
• common name: hello-openshift.com
• issuer: CN=openshift-signer@1476454220

HEAD / HTTP/1.1
User-Agent: curl/7.29.0
Host: hello-openshift.com
Accept: /

< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Fri, 14 Oct 2016 20:31:02 GMT
Date: Fri, 14 Oct 2016 20:31:02 GMT
< Content-Length: 17
Content-Length: 17
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Set-Cookie: f5e796f4ec45a2ffe9e31550d256219b=d4ece23b270c46f66c1f289397a4d854; path=/; HttpOnly
Set-Cookie: f5e796f4ec45a2ffe9e31550d256219b=d4ece23b270c46f66c1f289397a4d854; path=/; HttpOnly
< Cache-control: private
Cache-control: private

<

• Connection #0 to host hello-openshift.com left intact
  [root@ip-172-18-8-180 ~]# curl --resolve hello-openshift.com:443:172.18.0.203 https://hello-openshift.com --cacert cloudapps.router.pem
  Hello OpenShift!
  [root@ip-172-18-8-180 ~]#

  Edit route to have new certificate

  [root@ip-172-18-8-180 ~]# oc get route
  NAME HOST/PORT PATH SERVICES PORT TERMINATION
  route-1 hello-openshift.com endpoints edge
  [root@ip-172-18-8-180 ~]# oc edit route route-1
  route "route-1" edited
  [root@ip-172-18-8-180 ~]#
  [root@ip-172-18-8-180 ~]# curl --resolve hello-openshift.com:443:172.18.0.203 https://hello-openshift.com --cacert cloudapps.router.pem
  Hello OpenShift!
  [root@ip-172-18-8-180 ~]# curl --resolve hello-openshift.com:443:172.18.0.203 https://hello-openshift.com --cacert cloudapps.router.pem -vvI
• Added hello-openshift.com:443:172.18.0.203 to DNS cache
• About to connect() to hello-openshift.com port 443 (#0)
• Trying 172.18.0.203...
• Connected to hello-openshift.com (172.18.0.203) port 443 (#0)
• Initializing NSS with certpath: sql:/etc/pki/nssdb
• CAfile: cloudapps.router.pem
  CApath: none
• SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
• Server certificate:
• subject: CN=hello-openshift.com
• start date: Oct 14 20:29:39 2016 GMT
• expire date: Oct 14 20:29:40 2018 GMT
• common name: hello-openshift.com
• issuer: CN=openshift-signer@1476454220

HEAD / HTTP/1.1
User-Agent: curl/7.29.0
Host: hello-openshift.com
Accept: /

< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Fri, 14 Oct 2016 20:34:07 GMT
Date: Fri, 14 Oct 2016 20:34:07 GMT
< Content-Length: 17
Content-Length: 17
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Set-Cookie: f5e796f4ec45a2ffe9e31550d256219b=d4ece23b270c46f66c1f289397a4d854; path=/; HttpOnly
Set-Cookie: f5e796f4ec45a2ffe9e31550d256219b=d4ece23b270c46f66c1f289397a4d854; path=/; HttpOnly
< Cache-control: private
Cache-control: private

<

• Connection #0 to host hello-openshift.com left intact
  [root@ip-172-18-8-180 ~]#
  [root@ip-172-18-8-180 ~]# oc version
  oc v3.3.1.1
  kubernetes v1.3.0+52492b4
  features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-8-180.ec2.internal:8443
openshift v3.3.1.1
kubernetes v1.3.0+52492b4
[root@ip-172-18-8-180 ~]#

[ramr]

@mvlach so when you used the default certificate - was it a wildcard one (e.g. *.router.test)? If so, then the route's certificate will be secondary (if its in the same subdomain as the wildcard - example foo.router.test) on the list and won't be used.

@weliang1 your test https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/test50.json doesn't specify any custom certificates, so the defaults would always be used.

[ramr]

@mvlach aah just saw your mailing list message as well. So here's a simple test I wrote up:
https://github.com/ramr/router-default-cert-test

The test assumes you have a service setup called header-test-insecure -- if you use something else, make sure you edit the routes or the json config files (route-using-custom-cert.json and route-using-default-cert.json).

If you set it all up properly, here's the output that you should see:

$ curl -s -k -v --resolve custom.cert.test:443:127.0.0.1 https://custom.cert.test/ 2>&1 | egrep -e 'subject|issuer|Host'  
* Hostname custom.cert.test was found in DNS cache  
*   subject: OU=OpenShift3,O=Security,C=US,ST=CA,CN=header.test  
*   issuer: CN=header.test CA,OU=OpenShift3 test CA,O=Security,ST=CA,C=US  
> Host: custom.cert.test  

$ curl -s -k -v --resolve default-router.cert.test:443:127.0.0.1 https://default-router.cert.test/ 2>&1 | egrep -e 'subject|issuer|Host'  
* Hostname default-router.cert.test was found in DNS cache  
*   subject: CN=router.test,OU=OpenShift3,O=Security,ST=CA,C=US  
*   issuer: CN=router.test,OU=OpenShift3,O=Security,ST=CA,C=US  
> Host: default-router.cert.test  

As you can see the custom.cert.test returns the certificate for header.test. And the other host returns the default router cert issued to router.test.

[mvlach]

@ramr yes all certificates was wildcard. We changed the certificate because of alphaSSL certification authority problems. The problem I think is only with wildcard certificates. HAProxy pick the first eligible certificate.

[ramr]

@mvlach yeah if they are all wildcard (for the same subdomain ala *.co.tld), then HAProxy would pick the first eligible certificate. So if the default certificate is a wildcard certificate as an example for *.co.tld, then the cert from route www.co.tld would not be used because the wildcard cert would be the first match.

Now there have been some changes recently onhow the certs are presented and used since PR #11217 merged. This is on master though and assuming you want a default certificate to be used if nothing matches, one solution I can think of is:

1. use the latest code.
2. Set the default certificate to something other than a wildcard.
3. Either modify the haproxy images - custom dockerfile to also add your wildcard certificate to a well known place/path or you could also do it via a secret and mount that secret into the router container. If you do an oc edit dc/router, you should be able to see how the router-secrets get used/mounted.
   The main thing here is to get the wildcard certificate into a well known place/path inside the router container (let's say its at /opt/router/custom.cert).
4. Modify the haproxy config template to also write a line at the end for your default cert.
   Example: add a line /opt/router/custom.cert *.co.tld after the end of the range loop at:
   https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L570

HTH