bogus greeting in traefik -> traefik communication

[dodgex]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

We have a central traefik instance that routes requests to traefik instances running on the server that is serving the desired applicaton. recently we updated our traefik instance from 2.10.7 to 2.11.

After this we started to get Internal Server Error or Bad Gateway errors from traefik but nothing in the logs at this point. adding --log.level=DEBUG to the traefik instances result to the errors pasted below. The errors appeared in the second traefik instance.

The communication chain is Browser -HTTPS-> Traefik1 -HTTPS-> Traefik2 -HTTP-> local app container. Both traefik use a valid wildcard certificate.

When using a serversTransports with disableHTTP2 on the service loadbalancer configuration OR downgrading the first traefik to 2.10.7 it works fine. using 3.0.0-rc3 on the first traefik instance has the same issue.

The issue is not always present (works on first opening and after a browser restart; most of the times), and has a high chance to also be related to the fact, that the application that has been reported as affected are using kerberos/spnego authentication. The easiest way to enforce the issue is to use the logout button (present in case a user is logged in not via kerberos/spnego).

While this makes it hard if not impossible to reproduce, I still decided to open this issue in case you have an idea what change might have caused it...

What did you see instead?

Instead of a working application as before the upgrade, we now get Internal Server Error and Bad Gateway errors.

What version of Traefik are you using?

• Does not work when both instances are using this version
  Version: 2.11.0
  Codename: cheddar
  Go version: go1.22.0
  Built: 2024-02-12T15:26:45Z
  OS/Arch: linux/amd64

• When first or both instances using this version it works
  Version: 2.10.7
  Codename: saintmarcelin
  Go version: go1.21.5
  Built: 2023-12-06T15:54:59Z
  OS/Arch: linux/amd64

• Also tested with the first server beeing 3.0.0-rc3
  Version: 3.0.0-rc3
  Codename: beaufort
  Go version: go1.22.1
  Built: 2024-03-13T15:49:38Z
  OS/Arch: linux/amd64

What is your environment & configuration?

# docker copose command and labels for both
    command:
      - --api.insecure=true
      - --entryPoints.http.address=:80
      - --entryPoints.https.address=:443
      - --metrics.prometheus=true
      - --providers.docker
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=web
      - --providers.file.directory=/opt/traefik/conf
      - --providers.file.watch=true
      - --log.level=DEBUG # only for current testing

    labels:
      - "traefik.enable=true"
      - "traefik.http.services.traefik.loadbalancer.server.port=8080"
      - "traefik.http.routers.traefik.rule=Host(`traefik.server1.our.network`)" # server2 for second
      - "traefik.http.routers.traefik.entrypoints=http"

      - "traefik.http.middlewares.traefik-auth.basicauth.users=credentials"
      - "traefik.http.routers.traefik.middlewares=traefik-auth@docker"

# https-redirect.yaml (exists on both servers)
http:
  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https
        permanent: true

# reverse-proxy.yaml (exists only on server1)
http:
  services:
    srv-server2-http:
      loadBalancer:
        servers:
          - url: "http://server2.our.network"
    srv-server2-https:
      loadBalancer:
        servers:
          - url: "https://server2.our.network"

  routers:
    app-router:
      rule: "Host(`app.our.network`)"
      service: srv-server2-http
      entrypoints: http
      middlewares:
        - "https-redirect"
    app-secure-router:
      rule: "Host(`app.our.network`)"
      service: srv-server2-https
      entrypoints: https
      tls: {}

# docker compose for the target app (only on server2)
services:
  frontend:
    labels:
      - traefik.enable=true
      - traefik.http.services.app-frontend.loadbalancer.server.port=8080
      - traefik.http.routers.app-frontend-https.rule=Host(`app.our.network`)
      - traefik.http.routers.app-frontend-https.tls=true
      - traefik.http.routers.app-frontend-https.entrypoints=https

  backend:
    labels:
      - traefik.enable=true
      - traefik.http.services.app-backend.loadbalancer.server.port=8080
      - traefik.http.routers.app-backend-https.rule=Host(`app.our.network`) && PathPrefix(`/backend/`)
      - traefik.http.routers.app-backend-https.tls=true
      - traefik.http.routers.app-backend-https.entrypoints=https

If applicable, please paste the log output in DEBUG level

time="2024-03-14T10:13:12+01:00" level=debug msg="http2: server: error reading preface from client 10.101.x.y:41280: bogus greeting "GET /favicon.svg HTTP/1.""
time="2024-03-14T10:13:31+01:00" level=debug msg="http2: server: error reading preface from client 10.101.x.y:41432: bogus greeting "GET / HTTP/1.1\r\nHost: sc""

[bozzo]

I have the same issue with 2.11 on Kerberos authentication with 2 Keycloak instances behind Traefik (3 instances load balanced with round robin algorithm).
All requests work fine except Kerberos authentication. In private browsing this also works correctly (Firefox does not send a Kerberos ticket in private browsing, so authentication uses the Keycloak login/password form).
With 2.10.7, no issues.
I guess it's related to PR #10405, would a configuration option to enable sticky sessions for NTLM and Kerberos be an acceptable solution?

[juliens]

Thanks for reporting this issue.

I guess it's related to PR #10405, would a configuration option to enable sticky sessions for NTLM and Kerberos be an acceptable solution?

I agree that it seems linked to this PR, however, instead of relying on manual configuration, I'm wondering if there's a way to fix this automatically. To check, could you provide a reproducible case, like a docker-compose setup, to help me understand and debug the issue?

[dodgex]

I got a repro setup working. In the attached zip is a docker-compose.yml (starting two traefik and one nginx) and config files for each container.

Ideally you only need to unzip and run docker compose up. Then you should be able to open https://app-127-0-0-1.nip.io (depending on your dns setup this SOULD not work). If not either add it to etc/hosts or change the traefik yaml configs to something that works for you.

When I started the stack and open the url, the first time it shows the nginx welcome page, and as soon as you F5 the error happens in traefik-sub.

traefik_10522.zip

[42wim]

Got the same issue here with 2.11.0 and traefik going directly to a golang app with echo framework.
No kerberos enabled/used.
Only seem to happen with around 0.01% of the hits.

[42wim]

still happening with traefik v3.1.2, problem is that it's hard to reproduce, it only happens when there's a lot of load

[42wim]

I've rebuild v3.1.2 with the kerberos changes reverted (without doing the cloning)
Now the issue is gone, maybe there goes something wrong with the cloning of the roundtripper under a lot of load.

(maybe related to golang/go#21336)

Below is the patch I used

diff --git a/pkg/server/service/roundtripper.go b/pkg/server/service/roundtripper.go
index 06b6473c0..73ca22a9d 100644
--- a/pkg/server/service/roundtripper.go
+++ b/pkg/server/service/roundtripper.go
@@ -183,24 +183,10 @@ func (r *RoundTripperManager) createRoundTripper(cfg *dynamic.ServersTransport)

        // Return directly HTTP/1.1 transport when HTTP/2 is disabled
        if cfg.DisableHTTP2 {
-               return &KerberosRoundTripper{
-                       OriginalRoundTripper: transport,
-                       new: func() http.RoundTripper {
-                               return transport.Clone()
-                       },
-               }, nil
+               return transport, nil
        }

-       rt, err := newSmartRoundTripper(transport, cfg.ForwardingTimeouts)
-       if err != nil {
-               return nil, err
-       }
-       return &KerberosRoundTripper{
-               OriginalRoundTripper: rt,
-               new: func() http.RoundTripper {
-                       return rt.Clone()
-               },
-       }, nil
+       return newSmartRoundTripper(transport, cfg.ForwardingTimeouts)
 }

[dodgex]

@juliens Any news about this? The Issue is now almost 1 year old, and we honestly would love to upgrade all affected traefik instances to be on the same version as all of the others.

If the issue is to complex to find a proper fix in a reasonable time, that works for both cases (the one fixed in #10405 and the traefik->traefik case here), we would be happy to have to set a property to disable the fix from the PR to have the old behaviour for traefik->traefik communication.

Thanks!

[42wim]

@dodgex did you try my patch to see if it fixes your issue?

[dodgex]

@42wim your patch is simply reverting the part of #10405 that broke it in the first place, or am i wrong? also we dont want (and dont have the resources) to maintain a patched version of traefik.