Traefik cannot process CONNECT http request method.

[OrvilleQ]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

I was trying to deploy an applications which need CONNECT http request method to work, using Traefik as it's TLS layer and reverse proxy.

The application expose h2c on port 8080, and I was using this IngressRoute:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: h2c
  namespace: applications
spec:
  entryPoints:
  - websecure
  routes:
  - match: Host(`www.example.com`)
    kind: Rule
    services:
    - name: h2c
      kind: Service
      port: 8080
      scheme: h2c
  tls:
    secretName: com-example-www

What did you see instead?

1. If I PUT or GET the server, I'll see that the traffic been redirect to the application I deployed and the application reported a bug since it require CONNECT to work.

Log from Traefik:

time="2023-02-11T13:11:50Z" level=debug msg="'502 Bad Gateway' caused by: read tcp 10.244.3.203:59980->10.244.3.36:8080: read: connection reset by peer"
[IPADDRESS] - - [11/Feb/2023:13:11:50 +0000] "GET / HTTP/2.0" 502 11 "-" "-" 18553 "applications-h2c-9a98d4be767ad3f49819@kubernetescrd" "h2c://10.244.3.36:8080" 5ms

Log from the application:

+0000 2023-02-11 13:11:50 ERROR [1313487149]: process connection from 10.244.3.203:59980: not CONNECT request

2. If I CONNECT the server, I'll see that the traffic been dropped to nowhere.

Log form Traefik:

[IPADDRESS] - - [11/Feb/2023:13:14:02 +0000] "CONNECT - HTTP/2.0" 404 19 "-" "-" 18580 "-" "-" 0ms
[IPADDRESS] - - [11/Feb/2023:13:14:02 +0000] "CONNECT - HTTP/2.0" 404 19 "-" "-" 18581 "-" "-" 0ms

no log from the application.

3. If I add Method(`CONNECT`) to the match section following suggestion from here , PUT or GET will get 405 but CONNECT still got dropped.

Rule:

  - match: Host(`www.example.com`) && Method(`CONNECT`)

Log from Traefik:

[IPADDRESS] - - [11/Feb/2023:13:16:57 +0000] "GET / HTTP/2.0" 405 0 "-" "-" 18618 "-" "-" 0ms
[IPADDRESS] - - [11/Feb/2023:13:17:04 +0000] "CONNECT - HTTP/2.0" 404 19 "-" "-" 18619 "-" "-" 0ms
[IPADDRESS] - - [11/Feb/2023:13:17:04 +0000] "CONNECT - HTTP/2.0" 404 19 "-" "-" 18620 "-" "-" 0ms

What version of Traefik are you using?

Version:      2.9.6
Codename:     banon
Go version:   go1.19.4
Built:        2022-12-07T14:17:58Z
OS/Arch:      linux/amd64

What is your environment & configuration?

Traefik helm chart with default values on k3s v1.26.1+k3s1 using cilium and metallb.

If applicable, please paste the log output in DEBUG level

See above.

[prashant-warrier-echelonvi]

For the maintainers - we're currently on v2.7.0 of Traefik, and we're experiencing the same issue.

[rtribotte]

Hello @OrvilleQ @prashant-warrier-echelonvi,

Can you provide the full debug logs of the Traefik instance?
The 404 status code in the access log lines in both situations (with and without the Method(`CONNECT`) matcher) seems to indicate the request isn't matching any router.
The debug logs should show if anything went wrong during the application of the dynamic configuration.

As an alternative, it would also be great if you can share with us a short reproducible case (e.g.: a docker-compose) to help us to identify the issue.

[OrvilleQ]

@rtribotte Sorry for the late reply.

connect.zip

Inside this zip file is the yaml to deploy application that I use and how to connect to it.

Full debug log might have to wait till tonight if I have time, quiet busy these days.

[OrvilleQ]

Here is a full debug log. @rtribotte

debug.log

The log should me GET the server first and got 502 bad gateway, Then I CONNECT the server using the method I provided and the traffic got dropped to no where.

I masked some personal related data hope you don't mind.

[rtribotte]

Hello @OrvilleQ,

At glance, I maintain that the access logs indicate that the 404 response is produced by Traefik because no router matched the request.

I noticed that the router applications-sing-box-test has TLS enabled, which would only match a TLS request.
Since you are doing h2c, are you sure that the client is issuing a TLS request?
Can you share with us an example request/the client you are using?

[OrvilleQ]

@rtribotte Thanks for your reply. I'm pretty sure the client is issuing a TLS request since the client does not support plain http for security reason.

Also I've already provide the config file of the client and how to set it up in connect.zip

[aofei]

If you take a closer look at the spec (RFC 9110, section 9.3.6), I'm sure you'll realize that Traefik or any other reverse proxy cannot route CONNECT requests based only on the Host:

CONNECT server.example.com:80 HTTP/1.1
Host: server.example.com

Because the server.example.com in the request line and the Host header are the same, they both refer to the target to be connected. Not the domain name you configured in Traefik using something like match: Host(`www.example.com`).

So the fact is that it's not that Traefik cannot process CONNECT requests, but that you're using it wrong.

I believe instead of dedicating a separate TCP port to your service, you can use a SNI-based TCP router (of course, that also requires TLS to work):

tcp:
  routers:
    connect:
      rule: "HostSNI(`www.example.com`)"

Or, if you insist on using an HTTP router, then I think you can only do routing based on the request path or request headers. In any case, you can't achieve your purpose with Host.

So I believe this is not a bug.

[kahidna]

any updates on this?
I'm trying to run a Squid proxy behind Traefik.
since I'm new to the proxying world I just noticed that Squid uses the CONNECT method for HTTP requests.
and I can reproduce this issue as well

[rtribotte]

Hello,

@OrvilleQ Sorry for the late answer.
I think @aofei made a very good point.

Traefik is forwarding CONNECT as is, and do not act as an HTTP proxy itself.

Making Traefik an HTTP Proxy would defeat the reverse proxy configuration and would let the client target any backend, which is not desirable and could be a vulnerability.

On the other hand, reverse proxying an HTTP proxy, by making Traefik able to act as the client initiating the tunnel with an HTTP proxy backend, is not possible with a layer 7 reverse proxy (HTTP routers).
In that case, the layer 4 reverse proxy is the solution, as suggested by @aofei.

We are wondering if it would be better to refuse CONNECT HTTP method and document it.

[rtribotte]

Hello,

We conclude this to be a bug and it would be better to refuse HTTP CONNECT method and document it.
Unfortunately, this would not make it to our roadmap for a while as we are focused elsewhere. If any community member would like to build it, let us know, and we will work with you to ensure you have all the information needed to merge it.

Don't forget to check out the contributor docs and link the PR to this issue.