"GET / HTTP/2" 404 NR when trying trying to go via ingressgateway

[MrVentzi]

In short, what we are trying to achieve is this:
Laptop > Ingress gateway > external service in the environment

So we have a VM that runs some services, that VM is accessible by the cluster. doing a curl from the ingress gateway podconfirms it can access the VM.

The VM is on an HTTPS endpoint.

The end goal is to have the VM accessable by hitting the ingress gateway as this is one of the entry points to our environment.

There is an HTTP as well as an HTTPS endpoint on the VM. Using the same (apart from changing the tls to http in the VS and https with http in the Service Entry) works with the HTTP endpoint, but once we try HTTPS, it just doesn't work at all.

I have added a service entry, a destination rule and

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: test1
  namespace: istio-system
spec:
  hosts:
  - VM.endpoint.com
  ports:
  - number: 8123
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: test1
  namespace: istio-system
spec:
  host: VM.endpoint.com
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: test1
  namespace: istio-system
spec:
  gateways:
  - istio-system/tls-gateway
  hosts:
  - subdom.maindom.com
  tls:
  - match:
    - port: 443
      sniHosts:
      - subdom.maindom.com
    route:
    - destination:
        host: VM.endpoint.com
        port:
          number: 8123

[howardjohn]

Please add config dump from Ingressgateway pod: https://github.com/istio/istio/wiki/Troubleshooting-Istio. And ideally access log of failed request and the curl command used to reproduce

[howardjohn]

Also try istioctl analyze -A to see if it detect any issues

[esnible]

You have sniHosts: ["subdom.maindom.com"]. Can you supply the exact curl parameters you are using? You must curl subdom.maindom.com. It will not work without DNS unless you use curl's --resolve.

[MrVentzi]

Sorry for taking ages guys. had to move to another cluster

Config dump:
config_dump.zip

curl:

$ curl https://k8s.REDACTED.com/ -v
* About to connect() to k8s.REDACTED.com port 443 (#0)
*   Trying 10.32.0.250...
* Connected to k8s.REDACTED.com (10.32.0.250) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: XXXXX
* 	start date: Jun 05 07:14:24 2020 GMT
* 	expire date: Jun 05 07:14:54 2021 GMT
* 	common name: kXXXXX
* 	issuer: XXXXXX
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: k8s.REDACTED.com
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< date: Wed, 19 Aug 2020 10:06:23 GMT
< server: istio-envoy
< content-length: 0
< 
* Connection #0 to host k8s.REDACTED.com left intact

Istio-proxy log:
[2020-08-19T10:06:21.354Z] "GET / HTTP/1.1" 404 NR "-" "-" 0 0 0 - "10.32.0.3" "curl/7.29.0" "db74d55d-c96c-462e-b596-1c991df72b51" "k8s.REDACTED.com" "-" - - 10.32.66.2:443 10.32.0.3:42776 k8s.REDACTED.com -

Clearer yaml than above to avoid confusion:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: notebook
  namespace: istio-system
spec:
  hosts:
  - ventsislav-m.c.green-1-cpt-jojo-dev-1.internal
  ports:
  - number: 8123
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: notebook
  namespace: istio-system
spec:
  host: ventsislav-m.c.green-1-cpt-jojo-dev-1.internal
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: notebook
  namespace: istio-system
spec:
  gateways:
  - istio-system/tls-gateway
  hosts:
  - k8s.REDACTED.com
  tls:
  - match:
    - port: 443
      sniHosts:
      - k8s.REDACTED.com
    route:
    - destination:
        host: ventsislav-m.c.green-1-cpt-jojo-dev-1.internal
        port:
          number: 8123

Curl from ingressgateway pod to the external service:

$ curl https://ventsislav-m.c.green-1-cpt-jojo-dev-1.internal:8123 -v -k
* Rebuilt URL to: https://ventsislav-m.c.green-1-cpt-jojo-dev-1.internal:8123/
*   Trying 10.32.3.195...
* TCP_NODELAY set
* Connected to ventsislav-m.c.green-1-cpt-jojo-dev-1.internal (10.32.3.195) port 8123 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=ventsislav-m.europe-west1-b.c.green-1-cpt-jojo-dev-1.internal
*  start date: Aug 19 09:29:37 2020 GMT
*  expire date: Aug 26 09:30:07 2020 GMT
*  issuer: XXXXXX
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: ventsislav-m.c.green-1-cpt-jojo-dev-1.internal:8123
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Server: TornadoServer/6.0.3
< Content-Type: text/html; charset=UTF-8
< Date: Wed, 19 Aug 2020 10:18:44 GMT
< Location: /tree?
< Content-Length: 0
<
* Connection #0 to host ventsislav-m.c.green-1-cpt-jojo-dev-1.internal left intact

[howardjohn]

NAME                                         DOMAINS              MATCH                  VIRTUAL SERVICE
https.443.https.tls-gateway.istio-system     k8s.REDACTED.com     /elasticsearch/*       elasticsearch.elasticsearch
https.443.https.tls-gateway.istio-system     k8s.REDACTED.com     /elasticsearch         elasticsearch.elasticsearch
https.443.https.tls-gateway.istio-system     k8s.REDACTED.com     /REDACTED/*            REDACTED.REDACTED
https.443.https.tls-gateway.istio-system     k8s.REDACTED.com     /prometheus/*          prometheus-server.prometheus
https.443.https.tls-gateway.istio-system     k8s.REDACTED.com     /airflow/*             airflow.airflow
https.443.https.tls-gateway.istio-system     k8s.REDACTED.com     /airflow               airflow.airflow

You are calling on / but there is no config for this.

[howardjohn]

Your recent post and original post don't seem to align though... it should be noted, tls route is used only if we do passthrough TLS. If you are terminating TLS then it should be doing http routing.

From the config you are not doing any passhtrough. If you want to terminate and re-originate TLS you can do so with a DR

[MrVentzi]

Ended up with an nginx in the middle as it was taking too long to find the proper solution.