HTTPS Endpoints not reachable: Connection reset by peer

[Johannes1509]

I am trying to install Contour/Enovy as an Ingress controller. I have configured Contour with a NodePort. The cluster is running on AWS EC2 instances, which are running as a target group of an AWS NLB (Network load balancers).

Now when I want to curl to an https endpoint from local, the connection fails:

curl -k -I -vv https://emoji.eks-bku-dev-default.dbv-test.comp.de                                                                                                                                                                                                                      Fri May 26 14:59:03 2023

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.107.197.189:443...
* Connected to emoji.eks-bku-dev-default.dbv-test.comp.de (10.107.197.189) port 443 (#0)
* ALPN: offers h2,http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [5047 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.eks-bku-dev-default.dbv-test.comp.de
*  start date: Mar 30 00:00:00 2023 GMT
*  expire date: Apr 27 23:59:59 2024 GMT
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
} [5 bytes data]
* using HTTP/2
* h2h3 [:method: HEAD]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: emoji.eks-bku-dev-default.dbv-test.comp.db.de]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x15b06e0)
} [5 bytes data]
> HEAD / HTTP/2
> Host: emoji.eks-bku-dev-default.dbv-test.comp.db.de
> user-agent: curl/7.88.1
> accept: */*
>
{ [5 bytes data]
* TLSv1.2 (IN), TLS alert, close notify (256):
{ [2 bytes data]
* HTTP/2 stream 1 was not closed cleanly before end of the underlying stream
   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (18) HTTP/2 stream 1 was not closed cleanly before end of the underlying stream

With the use of an Nginx Ingress Controller the connection works.

Contour version 1.23.5, envoy version 1.24.5, Kubernetes EKS v1.23

Contour logs:
https://pastebin.mozilla.org/n51NTd5q

Envoy logs:
https://pastebin.mozilla.org/TPUNxVmU

Envoy Admin config_dump:
https://pastebin.mozilla.org/fO5An5UA

Contour helm chart values.yaml:

contour:
  image:
    tag: 1.23.5
  ingressClass:
    name: nginx-bku
    create: false
    default: true
  debug: true
envoy:
  logLevel: debug
  useHostPort: false
  image:
    tag: 1.24.5
  service:
    type: "NodePort"
    nodePorts:
      https: 30444
      http: 30081
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: "node.kubernetes.io/role"
            operator: In
            values:
            - "infra"
  priorityClassName: "system-node-critical"
configInline:
  disablePermitInsecure: false
  tls:
    fallback-certificate: {}
  accesslog-format: envoy
  network:
    admin-port: 9001
    num-trusted-hops: 10000
  policy:
    response-headers:
      set:
        X-Envoy-Response-Flags: "%RESPONSE_FLAGS%"

Ingress which I am trying to reach (uses a certmanger self-signed certificate):

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      ingress.kubernetes.io/force-ssl-redirect: "true"
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/backend-protocol: HTTP
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    creationTimestamp: "2023-04-03T14:32:56Z"
    generation: 7
    name: web-svc
    namespace: emojivoto
    resourceVersion: "901670625"
    uid: 44ca2e14-a124-4f22-ba4f-cc87a58c7607
  spec:
    ingressClassName: nginx-bku
    rules:
    - host: emoji.eks-bku-dev-default.dbv-test.comp.de
      http:
        paths:
        - backend:
            service:
              name: web-svc
              port:
                number: 80
          path: /
          pathType: Prefix
    tls:
    - hosts:
      - emoji.eks-bku-dev-default.dbv-test.comp.de
      secretName: emojivoto-cert
  status:
    loadBalancer:
      ingress:
      - ip: 172.XX.XX.XX

[skriss]

From the below, it looks like there's a wildcard cert configured on the NLB?

* Server certificate:
*  subject: CN=*.eks-bku-dev-default.dbv-test.comp.de
*  start date: Mar 30 00:00:00 2023 GMT
*  expire date: Apr 27 23:59:59 2024 GMT
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

If the NLB is terminating TLS, are you trying to re-encrypt traffic to the backend service, or should that hop be plain HTTP? The existence of a TLS secret on the Ingress implies re-encryption between the NLB and Envoy. Possibly since the Ingress cert is self-signed, the NLB is not trusting it.

If you're not actually looking for re-encryption, but just TLS termination at the NLB, then https://projectcontour.io/guides/deploy-aws-tls-nlb/ may help.