Traefik tries to use HTTP/2 with NTLM

[tfenster]

Do you want to request a feature or report a bug?

Bug

What did you do?

I have a service behind Traefik that uses Windows Auth. As stated here https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-10/http2-on-iis#when-is-http2-not-supported, Windows Auth is not supported with HTTP/2. Unfortunately it seems to me like Traefik always uses HTTP/2 when the service scheme is https. This leads to the following error in the Traefik debug log:

traefik    | time="2020-03-31T16:31:51Z" level=debug msg="'500 Internal Server Error' caused by: stream error: stream ID 3; HTTP_1_1_REQUIRED"

If I run the backend service with http instead of https, then Windows Auth succeeds.

What did you expect to see?

Windows Authentication goes through, even when the backend service is running https. It would be good to have an option to e.g. force http/1.1 between Traefik and backend when needed

What did you see instead?

Windows Authentication fails

Output of traefik version: (What version of Traefik are you using?)

Version:      2.2.0
Codename:     chevrotin
Go version:   go1.14.1
Built:        2020-03-25T17:17:27Z
OS/Arch:      windows/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

My docker-compose.yml looks like this when it works. Note that I disable SSL for the backend service and then use scheme http and port 80 on the service loadBalancer

  bc:
    image: mcr.microsoft.com/businesscentral/onprem:1810-ltsc2019
    container_name: bc
    hostname: traeftest
    environment:
      - accept_eula=y
      - clickonce=y
      - webserverinstance=bc
      - publicdnsname=traeftest.westeurope.cloudapp.azure.com
      - user=VM-Administrator
      - password=...
      - auth=Windows
#      - usessl=y
    labels:
      - traefik.enable=true
      - traefik.http.routers.bc.rule=Host(`traeftest.westeurope.cloudapp.azure.com`) && PathPrefix(`/bc`)
      - traefik.http.routers.bc.entrypoints=websecure
      - traefik.http.routers.bc.tls.certresolver=myresolver
      - traefik.http.routers.bc.service=bc@docker
      - traefik.http.services.bc.loadBalancer.server.scheme=http
      - traefik.http.services.bc.loadBalancer.server.port=80

If docker-compose.yml looks like this then it doesn't work. Note that I enable SSL for the backend service and then use scheme https and port 443 on the service loadBalancer

  bc:
    image: mcr.microsoft.com/businesscentral/onprem:1810-ltsc2019
    container_name: bc
    hostname: traeftest
    environment:
      - accept_eula=y
      - clickonce=y
      - webserverinstance=bc
      - publicdnsname=traeftest.westeurope.cloudapp.azure.com
      - user=VM-Administrator
      - password=...
      - auth=Windows
      - usessl=y
    labels:
      - traefik.enable=true
      - traefik.http.routers.bc.rule=Host(`traeftest.westeurope.cloudapp.azure.com`) && PathPrefix(`/bc`)
      - traefik.http.routers.bc.entrypoints=websecure
      - traefik.http.routers.bc.tls.certresolver=myresolver
      - traefik.http.routers.bc.service=bc@docker
      - traefik.http.services.bc.loadBalancer.server.scheme=https
      - traefik.http.services.bc.loadBalancer.server.port=443

The traefik config itself looks like this, unchanged in both cases. I've set insecureSkipVerify to true as the backend service uses self-signed certificates:

  traefik:
    image: traefik:2.2-windowsservercore-1809
    container_name: traefik
    command:
      - --log.level=DEBUG
      - --api.insecure=true
#      - --certificatesresolvers.myresolver.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --providers.docker.endpoint=npipe:////./pipe/docker_engine
      - --entrypoints.websecure.address=:443
      - --entrypoints.sql.address=:1433
      - --entrypoints.rtc.address=:7046
      - --certificatesresolvers.myresolver.acme.tlschallenge=true
      - --certificatesresolvers.myresolver.acme.email=tobias.fenster@cosmoconsult.com
      - --certificatesresolvers.myresolver.acme.storage=c:/le/acme.json
      - --serversTransport.insecureSkipVerify=true
    ports:
      - "8080:8080"
      - "443:443"
      - "1433:1433"
      - "7046:7046"
    volumes:
      - c:\users\tfenster8982\traefik:c:/le
      - type: npipe
        source: \\.\pipe\docker_engine
        target: \\.\pipe\docker_engine

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

see above

[tfenster]

On a side note: This used to work with v1.7

[mpl]

@tfenster We will try to investigate, but if it turns out that it definitely is not a configuration problem, we might have a hard time reproducing anyway since we do not run on windows.

[tfenster]

@mpl I am happy to provide whatever environment you need because it is very easily reproducible

[tfenster]

I would also try to fix it myself, but I can't figure out how to build on or for Windows. Is there any documentation or something that I missed?

[tfenster]

@mpl did you make any progress with this or can I provide you with a Windows-based environment so that you can repro?

[mpl]

@tfenster no progress, and it's pretty unlikely that we'll make any in the short term, sorry.
Your best bet is probably to post on the https://community.containo.us/ forum in order to find help about building Traefik for windows, so that you can indeed work on a fix yourself.

[tfenster]

@mpl obviously not the answer I was hoping for, but I appreciate your openness. I'll try to find some time to figure this

[pascalberger]

This is not depending on Traefik running on Windows, you can reproduce the behavior with Traefik running on Linux while forwarding requests to an IIS using NTLM (or probably any endpoint not supporting HTTP2 while using HTTPS).

[tfenster]

@mpl I tried to fix this in #7179, please let me know if I missed anything

[jspdown]

I took some time to investigate the issue. This can definitely be reproduced and it's not windows specific. This is due to the backend returning a HTTP_1_1_REQUIRED. As stated by RFC- (https://tools.ietf.org/html/rfc7540#section-7:

HTTP_1_1_REQUIRED (0xd): The endpoint requires that HTTP/1.1 be used
instead of HTTP/2.

Since Traefik does not handle this status code it just get interpreted as an internal error.

[tfenster]

@jspdown very interesting! This should make it possible to selectively handle this cased based on the backend return instead of completely disabling it as suggested in #7645?