Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version - protocol issue #11893

Closed
davidt-gh opened this issue Aug 27, 2024 · 6 comments
Labels
kind/support Categorizes issue or PR as a support question. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@davidt-gh
Copy link

davidt-gh commented Aug 27, 2024

My stack is EKS (1.30) with ingress-nginx-controller (chart 4.11.1) and cert-manager (chart 1.15.3)
Expected:
I want to have web application (in my example small nginx container). To do so, I have subdomain for my domain in route53 direct to my NLB, the NLB have 2 listeners TCP:443 and TCP:80 which forward to my nginx ingress controller pods

I created a new ingress, and got certificate from my cert-manager, example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kuard
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - test.domain.com
    secretName: quickstart-example-tls
  rules:
  - host: test.domain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kuard
            port:
              number: 80

Here is my nginx-ing-controller service annotation:

      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "external"
        service.beta.kubernetes.io/aws-load-balancer-name: eks-dev-nlb
        service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
        service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: false
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
        service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"

NGINX logs look like that:

10.1.1.51 - - [27/Aug/2024:13:31:45 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - a4458d9ce9cdf81b7f3747ed96dd64c2
10.1.1.51 - - [27/Aug/2024:13:31:45 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - aa77319d25fa6a6d3af33cb7975dac42
10.1.2.111 - - [27/Aug/2024:13:31:46 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - 0b90d43435be1d477f1610f9240b8797

When doing curl to my subdomain this is the result:

~ curl --tlsv1.2 -v https://test.domain.com

* Host test.domain.com:443 was resolved.
* IPv6: (none)
* IPv4: 44.221.31.228, 52.23.30.253
*   Trying 44.221.31.228:443...
* Connected to test.domain.com (44.221.31.228) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
* Closing connection
curl: (35) LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version

from browser I'm getting:

ERR_SSL_PROTOCOL_ERROR

Not sure how to debug this, and what to check next or why I'm getting this error. From the documenation I can understand NGINX use by default "tls1.2" or "tls1.3" so I'm not sure where am I failing.

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Aug 27, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority labels Aug 27, 2024
@strongjz
Copy link
Member

Just checking the version of openssl

docker run -it registry.k8s.io/ingress-nginx/controller:v1.11.1@sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a openssl version
OpenSSL 3.3.1 4 Jun 2024 (Library: OpenSSL 3.3.1 4 Jun 2024)

Can you output the nginx.conf and see how the directive for test.domain.com looks and what version of TLS is configured?

https://kubernetes.github.io/ingress-nginx/troubleshooting/#check-the-nginx-configuration

@davidt-gh
Copy link
Author

## start server test.domain.com
        server {
                server_name test.domain.com ;

                http2 on;

                listen 80  ;
                listen [::]:80  ;
                listen 443  ssl;
                listen [::]:443  ssl;

                set $proxy_upstream_name "-";

                ssl_certificate_by_lua_block {
                        certificate.call()
                }

                location / {

                        set $namespace      "default";
                        set $ingress_name   "kuard";
                        set $service_name   "kuard";
                        set $service_port   "80";
                        set $location_path  "/";
                        set $global_rate_limit_exceeding n;

                        rewrite_by_lua_block {
                                lua_ingress.rewrite({
                                        force_ssl_redirect = false,
                                        ssl_redirect = true,
                                        force_no_ssl_redirect = false,
                                        preserve_trailing_slash = false,
                                        use_port_in_redirects = false,
                                        global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },
                                })
                                balancer.rewrite()
                                plugins.run()
                        }

                        # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
                        # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
                        # other authentication method such as basic auth or external auth useless - all requests will be allowed.
                        #access_by_lua_block {
                        #}

                        header_filter_by_lua_block {
                                lua_ingress.header()
                                plugins.run()
                        }

                        body_filter_by_lua_block {
                                plugins.run()
                        }

                        log_by_lua_block {
                                balancer.log()

                                plugins.run()
                        }

                        port_in_redirect off;

                        set $balancer_ewma_score -1;
                        set $proxy_upstream_name "default-kuard-80";
                        set $proxy_host          $proxy_upstream_name;
                        set $pass_access_scheme  $scheme;

                        set $pass_server_port    $server_port;

                        set $best_http_host      $http_host;
                        set $pass_port           $pass_server_port;

                        set $proxy_alternative_upstream_name "";

                        client_max_body_size                    1m;

                        proxy_set_header Host                   $best_http_host;

                        # Pass the extracted client certificate to the backend

                        # Allow websocket connections
                        proxy_set_header                        Upgrade           $http_upgrade;

                        proxy_set_header                        Connection        $connection_upgrade;

                        proxy_set_header X-Request-ID           $req_id;
                        proxy_set_header X-Real-IP              $remote_addr;

                        proxy_set_header X-Forwarded-For        $remote_addr;

                        proxy_set_header X-Forwarded-Host       $best_http_host;
                        proxy_set_header X-Forwarded-Port       $pass_port;
                        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
                        proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

                        proxy_set_header X-Scheme               $pass_access_scheme;

                        # Pass the original X-Forwarded-For
                        proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

                        # mitigate HTTPoxy Vulnerability
                        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
                        proxy_set_header Proxy                  "";

                        # Custom headers to proxied server

                        proxy_connect_timeout                   5s;
                        proxy_send_timeout                      60s;
                        proxy_read_timeout                      60s;

                        proxy_buffering                         off;
                        proxy_buffer_size                       4k;
                        proxy_buffers                           4 4k;

                        proxy_max_temp_file_size                1024m;

                        proxy_request_buffering                 on;
                        proxy_http_version                      1.1;

                        proxy_cookie_domain                     off;
                        proxy_cookie_path                       off;

                        # In case of errors try the next upstream server before returning an error
                        proxy_next_upstream                     error timeout;
                        proxy_next_upstream_timeout             0;
                        proxy_next_upstream_tries               3;

                        # Custom Response Headers

                        proxy_pass http://upstream_balancer;

                        proxy_redirect                          off;

                }

        }
        ## end server test.domain.com

Sure @strongjz

@longwuyuan
Copy link
Contributor

The information you providing is indicating only but not ample data for analyzing or reproducing problem in minikube or kind cluster.

If you look at the template of a new bug report, and edit your issue description here, to answer all the questions asked in the template of a new bug report, then it provides data that can be analyzed.

/kind support
/triage needs-information

@k8s-ci-robot k8s-ci-robot added kind/support Categorizes issue or PR as a support question. triage/needs-information Indicates an issue needs more information in order to work on it. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Aug 27, 2024
@davidt-gh
Copy link
Author

my issue was in the full values file (which not provided here), I've added:

controller:
  service:
    targetPorts:
      https: http

which cause all https to use http port.
Thank you for your time and good willing.

@longwuyuan
Copy link
Contributor

Thanks for updating. Glad resolved. The static manifest published by project for termnating on LB is different from the manifest for terminating on controller. If you see the diff, those port related values are truly the significant ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/support Categorizes issue or PR as a support question. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
Development

No branches or pull requests

4 participants