Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDS presents the wrong server certificate (resulting in 404 HTTP response) when multiple gateways are configured in the same namespace #21077

Open
balamuru opened this issue Feb 12, 2020 · 0 comments

Comments

@balamuru
Copy link

@balamuru balamuru commented Feb 12, 2020

Bug description
When multiple gateways are configured in the SAME namespace with TLS / SDS with different secrets and different port names , a HTTP call to a svc associated with one gateway will work , while HTTP calls to other services will fail with 404 .

Interestingly, when making calls to the misbehaving svc that return 404 , the server presents the wrong certificate (associated with the working service)

  • I suspect that this issue is the same as #20661 (which hasn't recieved a response) .
  • I was advised to make the port names unique as per #12500 but this doesn't help.
  • I upgraded istio versions from 1.4.3 to 1.4.4 but that didn't help

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[X ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior

Steps to reproduce the bug

  • Deploy 2 services (let's call them A and B)
  • Create and install 2 sets of secrets inthe istio-system namespace (intended for svc A and B respectively)
  • Create and deploy 2 sets of istio gateways and virtual services
  • make a test call to svc A
  • make a test call to svc B
  • One of the calls will fail with 404 (probably svc B if it's gateway was installed second)
  • Note that the server certificate for A will be presented , instead of the certificate for B
  • Delete the gateway for A
  • Repeat the call to svc B . It will succeed with the correct server certificate presented

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
1.4.3
How was Istio installed?
istioctl

Environment where bug was observed (cloud vendor, OS, etc)
Centos

Debug Session

Gateway and Virtual Service config

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: mygateway
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 80
      name: http-httpbin
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: https-httpbin
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: "somehost-credential" # must be the same as secret
    hosts:
    - "*"
EOF

---
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
[istio-dump.tar.gz](https://github.com/istio/istio/files/4195235/istio-dump.tar.gz)

metadata:
  name: httpbin
spec:
  hosts:
  - "*"
  gateways:
  - mygateway
  http:
  - match:
    - uri:
        prefix: /status
    - uri:
        prefix: /delay
    route:
    - destination:
        port:
          number: 8000
        host: httpbin
EOF
---

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: mygateway-nodejs
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 80
      name: http-nodejs
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: https-nodejs
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: "otherhost-credential" # must be the same as secret
    hosts:
    - "*"
EOF

---
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nodejs
spec:
  hosts:
  - "*"
  gateways:
  - mygateway-nodejs
  http:
  - match:
    - uri:
        prefix: /fooa
    route:
    - destination:
        port:
          number: 8080
        host: nodejs
EOF

---

Make a call to svc B, note the wrong server certificate is presented

[ocadmin@cmsplavm224 other.host.com]$ curl -v  -k   --noproxy "*"  https://cmsplavm224.oma2hpe.lan:31002/fooa
* About to connect() to cmsplavm224.oma2hpe.lan port 31002 (#0)
*   Trying 10.2.125.225...
* Connected to cmsplavm224.oma2hpe.lan (10.2.125.225) port 31002 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=some.host.com,O=Dis,L=Springfield,ST=Denial,C=US ======> should be "other.host.com
* 	start date: Feb 11 23:16:26 2020 GMT
* 	expire date: Feb 20 23:16:26 2021 GMT
* 	common name: some.host.com
* 	issuer: CN=some.host.com,O=Dis,ST=Denial,C=US
> GET /fooa HTTP/1.1
> User-Agent: curl/7.29.0
> Host: cmsplavm224.oma2hpe.lan:31002
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< date: Wed, 12 Feb 2020 21:41:54 GMT
< server: istio-envoy
< content-length: 0
< 
* Connection #0 to host cmsplavm224.oma2hpe.lan left intact

Delete the other gateway

[ocadmin@cmsplavm224 other.host.com]$ kubectl delete gw mygateway 
gateway.networking.istio.io "mygateway" deleted

Repeat call, will succeed

[ocadmin@cmsplavm224 other.host.com]$ curl -v  -k   --noproxy "*"  https://cmsplavm224.oma2hpe.lan:31002/fooa
* About to connect() to cmsplavm224.oma2hpe.lan port 31002 (#0)
*   Trying 10.2.125.225...
* Connected to cmsplavm224.oma2hpe.lan (10.2.125.225) port 31002 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=other.host.com,O=Dis,L=Springfield,ST=Denial,C=US ====> correct
* 	start date: Jan 29 22:04:24 2020 GMT
* 	expire date: Feb 07 22:04:24 2021 GMT
* 	common name: other.host.com
* 	issuer: CN=other.host.com,O=Dis,ST=Denial,C=US
> GET /fooa HTTP/1.1
> User-Agent: curl/7.29.0
> Host: cmsplavm224.oma2hpe.lan:31002
> Accept: */*
> 
< HTTP/1.1 200 OK
< x-powered-by: Express
< content-type: text/html; charset=utf-8
< content-length: 24
< etag: W/"18-qN7KJMo1QAhzIcYWh4eTqRZdsMk"
< date: Wed, 12 Feb 2020 21:44:28 GMT
< x-envoy-upstream-service-time: 3
< server: istio-envoy
< 
Hello foo from Server A
* Connection #0 to host cmsplavm224.oma2hpe.lan left intact
[ocadmin@cmsplavm224 other.host.com]$ 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.