Backend for custom error page has no RemoteAddr/Scheme

[bonndan]

Do you want to request a feature or report a bug?

Bug

What did you do?

Trying to use custom error pages for a service "gateway" from the backend "keycloak". The browser shows "Internal server error" and the traefik logs indicate the backend is not resolved properly.

Environment is traefik:1.7 docker image running in swarm mode.

What did you expect to see?

A HTML page from the "keycloak" service.

What did you see instead?

Internal Server Error

Output of traefik version: (What version of Traefik are you using?)

Version:      v1.7.0-rc2
Codename:     maroilles
Go version:   go1.10.3
Built:        2018-07-17_12:42:38PM
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

version: "3.3"

networks:
  public:
    external:
      name: 

services:
  traefik:
    image: traefik:1.7 # 1.7 supports per-frontend auth forwarding
    command: --api --docker --docker.domain=xxx --docker.swarmmode --docker.watch 
    ports:
      - "80:80"     # The HTTP port
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock 
    networks:
      - public

version: '3.3'

services:

  gateway:
    image: keycloak-proxy:latest
    deploy:
      labels:
        - "traefik.backend=gateway"
        - "traefik.frontend.rule=PathPrefixStrip:/gateway"
        - "traefik.port=8080"
        - "traefik.docker.network=public"
        - "traefik.frontend.errors.all.backend=keycloak"
        - "traefik.frontend.errors.all.status=400-599"
        - "traefik.frontend.errors.all.query=/auth/realms/master/account"
      placement:
        constraints:
          - node.role == worker
      mode: replicated
      replicas: 1
    networks:
      - public

networks:
  public:
    external:
      name: public

version: '3'

volumes:
  postgres_data:
      driver: local
    
networks:
  public:
    external:
      name: 
  keycloak:

services:
  postgres:
      image: postgres
      volumes:
        - postgres_data:/var/lib/postgresql/data
      environment:
        XXX:YYY
      networks:
        - keycloak
  keycloak:
      image: jboss/keycloak
      environment:
        XXX:YYY
      healthcheck:
        test: ["CMD", "curl", "-f", "http://localhost:8080/auth/"]
        interval: 5s
        timeout: 2s
        retries: 15
      deploy:
        labels:
          - "traefik.backend=keycloak"
          - "traefik.port=8080"
          - "traefik.frontend.rule=PathPrefix:/auth"
          - "traefik.docker.network=public" #needed because we have two networks
      networks:
        - public
        - keycloak
      depends_on:
        - postgres

If applicable, please paste the log output in DEBUG level (--logLevel=DEBUG switch)

globalservices_traefik.0.vzd4fktlfe17@docker-32    | time="2018-07-19T13:18:14Z" level=debug msg="Upstream ResponseWriter of type *pipelining.writerWithoutCloseNotify does not implement http.CloseNotifier. Returning dummy channel."

...

globalservices_traefik.0.vzd4fktlfe17@xxx    | time="2018-07-19T13:18:14Z" level=debug msg="'500 Internal Server Error' caused by: unsupported protocol scheme \"\""
globalservices_traefik.0.vzd4fktlfe17@xxx    | time="2018-07-19T13:18:14Z" level=debug msg="vulcand/oxy/forward/http: Round trip: /auth/realms/master/account, code: 500, Length: 21, duration: 100.339µs"

...

globalservices_traefik.0.vzd4fktlfe17@xxx    | time="2018-07-19T13:00:34Z" level=debug msg="vulcand/oxy/forward: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/auth/realms/master/account\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Cookie\":[\"_ga=GA1.1.1559381502.1530883005; _gid=GA1.1.1615818971.1531902865; kc-access=; kc-state=\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36\"],\"X-Forwarded-Prefix\":[\"/gateway\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"\",\"RequestURI\":\"/auth/realms/master/account\",\"TLS\":null}"

[kachkaev]

I guess I'm facing the same thing – see my comment in #3678

[bonndan]

I also got this error in a rancher environment now: The backend was resolved to a non-existent IP, resulting in bad gateway.

[ldez]

If you have this problem, could you give us the version of Traefik you are using.

[bonndan]

I opened this issue, it's still 1.7-RC2. Haven't tried RC3, but according to the changelog nothing has been done regarding this issue.

[signaleleven]

1.7-RC3

[kachkaev]

Also 1.7-RC3 (details in #3678 (comment)). I could not make custom error pages working regardless of what I tried. Is there an example ingress rule that I can test in my cluster? I don't exclude that the error occurs because of a clash with other annotations that I've been using.

[ldez]

I confirm the bug, it's a regression related to #3312

[kachkaev]

@ldez do you think the bug occurs in all scenarios or are there certain combinations of ignress annotations where a custom error would be shown? If yes, I'd be keen to taste the flavour of this feature.

[ldez]

the bug occurs in all scenarios and all providers, I don't think there is any workaround. Sorry.

[traefiker]

Closed by #3894.