v2: chain middleware removes headers added by headers middleware

[JamesKyburz]

Do you want to request a feature or report a bug?

Bug

What did you do?

Expect to see custom headers when the headers middleware is used.

What did you expect to see?

custom headers

What did you see instead?

When the chain middleware is used custom headers are not present in the response.

Output of traefik version: (What version of Traefik are you using?)

v2.1

(paste your output here)

What is your environment & configuration (arguments, toml, provider, platform, ...)?

# (paste your configuration here)

Using chain the custom headers are not present.

Example docker-compose-bug.yml

~/Documents/src/traefik-bug/root1 ᐅ cat docker-compose-bug.yml

version: '3'
services:
  reverse-proxy:
    image: traefik:v2.1
    networks:
      - traefik-bug-root1
    command:
      - '--providers.docker'
    ports:
    - 8095:80
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    restart: always

  test:
    networks:
    - traefik-bug-root1
    image: hashicorp/http-echo
    command: "-listen :5000 -text='hi'"
    labels:
    - 'traefik.enable=true'
    - 'traefik.http.routers.test.rule=PathPrefix(`/test`)'
    - 'traefik.http.routers.test.middlewares=test'
    - 'traefik.http.services.test.loadbalancer.server.port=5000'
    - 'traefik.http.middlewares.test.chain.middlewares=test-headers,test-compress'
    - 'traefik.http.middlewares.test-compress.compress=true'
    - 'traefik.http.middlewares.test-headers.headers.customresponseheaders.X-Custom-Response-Header=value'

networks:
  traefik-bug-root1:
    driver: bridge

If the chain middleware is not used custom headers are present.

~/Documents/src/traefik-bug/root1 ᐅ cat docker-compose-ok.yml

version: '3'
services:
  reverse-proxy:
    image: traefik:v2.1
    networks:
      - traefik-bug-root1
    command:
      - '--providers.docker'
    ports:
    - 8095:80
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    restart: always

  test:
    networks:
    - traefik-bug-root1
    image: hashicorp/http-echo
    command: "-listen :5000 -text='hi'"
    labels:
    - 'traefik.enable=true'
    - 'traefik.http.routers.test.rule=PathPrefix(`/test`)'
    - 'traefik.http.routers.test.middlewares=test-headers'
    - 'traefik.http.services.test.loadbalancer.server.port=5000'
    - 'traefik.http.middlewares.test-headers.headers.customresponseheaders.X-Custom-Response-Header=value'

networks:
  traefik-bug-root1:
    driver: bridge

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

(paste your output here)

[juliens]

Could you provide us the logs in debug mode, and could you enable the --api.insecure and send us the result of http://127.0.0.1:8080/api/rawdata

[robinwangrubin]

Yes, I met the same problem. When i used the middleware chain. The customresponseheaders doesn't work.

[dtomcej]

I think the issue may come from the responseModifiers not being properly applied when the headers middleware is included in a chain:
https://github.com/containous/traefik/blob/master/pkg/responsemodifiers/response_modifier.go#L31

[Dmry]

Ran into the same issue. In case it's still helpful, please find my raw data from http://127.0.0.1:8080/api/rawdata attached.

rawdata.txt

[pascalandy]

Hello folks,
Just tried to recreate the issue. chain middleware is working as expected for me. The headers are present.

compose nginx

labels:
  # ...
  - "traefik.http.routers.nginx.middlewares=RuleGrpMain"

compose traefik

labels:
  # ...
  - "traefik.http.middlewares.RuleGrpMain.chain.middlewares=compress_all,ratelimit_all,retry_all,header_all,slash_step1,slash_step2"
  ...
  - "traefik.http.middlewares.ratelimit_all.ratelimit.average=100"
  - "traefik.http.middlewares.ratelimit_all.ratelimit.burst=10"

logs

[pascalandy]

Here is a funny behavior. I have 3 webapps (whoami, nginx, caddy). All of them are using the same chain:

  #### Apply rules (middlewares)
  - "traefik.http.routers.home.middlewares=RuleGrpMain"

I cannot explain why caddy have no headers ¯_(ツ)_/¯. nginx and home headers are correct. See my compose files here. (You can see my domain is also behind Cloudflare.)

logs

caddy

no headers:

$ curl -I https://devkiwi.club/caddy/

HTTP/2 200
date: Fri, 17 Jan 2020 16:08:36 GMT
content-type: text/html; charset=utf-8
set-cookie: __cfduid=d40191a814363abcda73c7c498d55b0de1579277316; expires=Sun, 16-Feb-20 16:08:36 GMT; path=/; domain=.devkiwi.club; HttpOnly; SameSite=Lax
accept-ranges: bytes
last-modified: Mon, 13 Jan 2020 14:50:45 GMT
vary: Accept-Encoding
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 556996baabb0f4be-YVR

whoami

headers yeah :)

$ curl -I https://devkiwi.club

HTTP/2 200
date: Fri, 17 Jan 2020 16:09:10 GMT
content-type: text/plain; charset=utf-8
content-length: 483
set-cookie: __cfduid=d96da8169317396e0c45bd3719987421f1579277350; expires=Sun, 16-Feb-20 16:09:10 GMT; path=/; domain=.devkiwi.club; HttpOnly; SameSite=Lax
content-security-policy: default-src https:
feature-policy: accelerometer 'none'; ambient-light-sensor 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; usb 'none'; midi 'none'; sync-xhr 'none'; vr 'none'
referrer-policy: strict-origin-when-cross-origin
strict-transport-security: max-age=31536000; includeSubDomains; preload
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: DENY
x-xss-protection: 1; mode=block
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 5569978f3a7d3b4e-YVR

nginx

headers yeah :)

$ curl -I https://devkiwi.club/nginx/

HTTP/2 200
date: Fri, 17 Jan 2020 16:06:48 GMT
content-type: text/html
set-cookie: __cfduid=d647e69770eefc87cdafcfdb4f5b53f511579277208; expires=Sun, 16-Feb-20 16:06:48 GMT; path=/; domain=.devkiwi.club; HttpOnly; SameSite=Lax
accept-ranges: bytes
content-security-policy: default-src https:
feature-policy: accelerometer 'none'; ambient-light-sensor 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; usb 'none'; midi 'none'; sync-xhr 'none'; vr 'none'
last-modified: Mon, 13 Jan 2020 14:50:45 GMT
referrer-policy: strict-origin-when-cross-origin
strict-transport-security: max-age=31536000; includeSubDomains; preload
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: DENY
x-xss-protection: 1; mode=block
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 556994175a1ff4b2-YVR

[pascalandy]

More tests...

A) I think the official caddy image is overriding the headers. Could it be possible?

Using a custom image of caddy (devmtl/caddyfire:1.0.3), where I defined the headers in Caddy's config, the headers are OK.

But yeah... this is a hack. The point is to have traefik caring about this.

B) I recreated your use case @JamesKyburz. No headers :-/
I don't think a basic server as hashicorp/http-echo do override the headers.

Conclusion: I'm confused.

compose-hello.yml

version: "3.3"

services:

  hello:
    image: hashicorp/http-echo
    container_name: hello
    hostname: hello
    restart: unless-stopped
    command: "-listen=':5001' -text='Hello Pascal'"

    labels:
      #### core configs
      - "traefik.enable=true"
      - "traefik.http.routers.hello.rule=Host(`devkiwi.club`) && Path(`/hello`)"
      - "traefik.http.services.hello.loadbalancer.server.port=5001"
      #### set TLS (https)
      - "traefik.http.routers.hello.entrypoints=websecure"
      - "traefik.http.routers.hello.tls=true"
      - "traefik.http.routers.hello.tls.certresolver=leresolver"
      #### Apply rules (middlewares)
      - "traefik.http.routers.hello.middlewares=RuleGrpMain"

logs

$ curl -I https://devkiwi.club/hello

HTTP/2 200
date: Fri, 17 Jan 2020 22:28:43 GMT
content-type: text/plain; charset=utf-8
content-length: 13
set-cookie: __cfduid=d03edbff6bb2f038ac4280db1f0df72261579300123; expires=Sun, 16-Feb-20 22:28:43 GMT; path=/; domain=.devkiwi.club; HttpOnly; SameSite=Lax
x-app-name: http-echo
x-app-version: 0.2.3
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 556bc389de28f4b6-YVR

[JamesKyburz]

docker-compose-bug.yml

version: '3'
services:
  reverse-proxy:
    image: traefik:v2.1
    networks:
      - traefik-bug-root1
    command:
      - '--providers.docker'
      - '--api.insecure'
    ports:
    - 8095:80
    - 8080:8080
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    restart: always

  test:
    networks:
    - traefik-bug-root1
    image: hashicorp/http-echo
    command: "-listen :5000 -text='hi'"
    labels:
    - 'traefik.enable=true'
    - 'traefik.http.routers.test.rule=PathPrefix(`/test`)'
    - 'traefik.http.routers.test.middlewares=test'
    - 'traefik.http.services.test.loadbalancer.server.port=5000'
    - 'traefik.http.middlewares.test.chain.middlewares=test-headers,test-compress'
    - 'traefik.http.middlewares.test-compress.compress=true'
    - 'traefik.http.middlewares.test-headers.headers.customresponseheaders.X-Custom-Response-Header=value'

networks:
  traefik-bug-root1:
    driver: bridge

docker-compose -f docker-compose-bug.yml up

http --headers http://localhost:8095/test              
HTTP/1.1 200 OK
Content-Length: 3
Content-Type: text/plain; charset=utf-8
Date: Sat, 18 Jan 2020 14:26:08 GMT
Vary: Accept-Encoding
X-App-Name: http-echo
X-App-Version: 0.2.3

curl http://localhost:8080/api/rawdata | jq

{
  "routers": {
    "api@internal": {
      "entryPoints": [
        "traefik"
      ],
      "service": "api@internal",
      "rule": "PathPrefix(`/api`)",
      "priority": 2147483646,
      "status": "enabled",
      "using": [
        "traefik"
      ]
    },
    "dashboard@internal": {
      "entryPoints": [
        "traefik"
      ],
      "middlewares": [
        "dashboard_redirect@internal",
        "dashboard_stripprefix@internal"
      ],
      "service": "dashboard@internal",
      "rule": "PathPrefix(`/`)",
      "priority": 2147483645,
      "status": "enabled",
      "using": [
        "traefik"
      ]
    },
    "reverse-proxy-root1@docker": {
      "service": "reverse-proxy-root1",
      "rule": "Host(`reverse-proxy-root1`)",
      "status": "enabled",
      "using": [
        "http",
        "traefik"
      ]
    },
    "test@docker": {
      "middlewares": [
        "test@docker"
      ],
      "service": "test",
      "rule": "PathPrefix(`/test`)",
      "status": "enabled",
      "using": [
        "http",
        "traefik"
      ]
    }
  },
  "middlewares": {
    "dashboard_redirect@internal": {
      "redirectRegex": {
        "regex": "^(http:\\/\\/[^:\\/]+(:\\d+)?)\\/$",
        "replacement": "${1}/dashboard/",
        "permanent": true
      },
      "status": "enabled",
      "usedBy": [
        "dashboard@internal"
      ]
    },
    "dashboard_stripprefix@internal": {
      "stripPrefix": {
        "prefixes": [
          "/dashboard/",
          "/dashboard"
        ]
      },
      "status": "enabled",
      "usedBy": [
        "dashboard@internal"
      ]
    },
    "test-compress@docker": {
      "compress": {},
      "status": "enabled"
    },
    "test-headers@docker": {
      "headers": {
        "customResponseHeaders": {
          "X-Custom-Response-Header": "value"
        }
      },
      "status": "enabled"
    },
    "test@docker": {
      "chain": {
        "middlewares": [
          "test-headers@docker",
          "test-compress@docker"
        ]
      },
      "status": "enabled",
      "usedBy": [
        "test@docker"
      ]
    }
  },
  "services": {
    "api@internal": {
      "status": "enabled",
      "usedBy": [
        "api@internal"
      ]
    },
    "dashboard@internal": {
      "status": "enabled",
      "usedBy": [
        "dashboard@internal"
      ]
    },
    "reverse-proxy-root1@docker": {
      "loadBalancer": {
        "servers": [
          {
            "url": "http://172.27.0.3:80"
          }
        ],
        "passHostHeader": true
      },
      "status": "enabled",
      "usedBy": [
        "reverse-proxy-root1@docker"
      ],
      "serverStatus": {
        "http://172.27.0.3:80": "UP"
      }
    },
    "test@docker": {
      "loadBalancer": {
        "servers": [
          {
            "url": "http://172.27.0.2:5000"
          }
        ],
        "passHostHeader": true
      },
      "status": "enabled",
      "usedBy": [
        "test@docker"
      ],
      "serverStatus": {
        "http://172.27.0.2:5000": "UP"
      }
    }
  }
}

[JamesKyburz]

cat docker-compose.ok.yaml

version: '3'
services:
  reverse-proxy:
    image: traefik:v2.1
    networks:
      - traefik-bug-root1
    command:
      - '--providers.docker'
      - '--api.insecure'
    ports:
    - 8095:80
    - 8080:8080
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    restart: always

  test:
    networks:
    - traefik-bug-root1
    image: hashicorp/http-echo
    command: "-listen :5000 -text='hi'"
    labels:
    - 'traefik.enable=true'
    - 'traefik.http.routers.test.rule=PathPrefix(`/test`)'
    - 'traefik.http.routers.test.middlewares=test-headers'
    - 'traefik.http.services.test.loadbalancer.server.port=5000'
    - 'traefik.http.middlewares.test-headers.headers.customresponseheaders.X-Custom-Response-Header=value'

networks:
  traefik-bug-root1:
    driver: bridge

docker-compose -f docker-compose-ok.yml up

http --headers http://localhost:8095/test              
HTTP/1.1 200 OK
Content-Length: 3
Content-Type: text/plain; charset=utf-8
Date: Sat, 18 Jan 2020 14:28:55 GMT
X-App-Name: http-echo
X-App-Version: 0.2.3
X-Custom-Response-Header: value

curl http://localhost:8080/api/rawdata | jq ''

{
  "routers": {
    "api@internal": {
      "entryPoints": [
        "traefik"
      ],
      "service": "api@internal",
      "rule": "PathPrefix(`/api`)",
      "priority": 2147483646,
      "status": "enabled",
      "using": [
        "traefik"
      ]
    },
    "dashboard@internal": {
      "entryPoints": [
        "traefik"
      ],
      "middlewares": [
        "dashboard_redirect@internal",
        "dashboard_stripprefix@internal"
      ],
      "service": "dashboard@internal",
      "rule": "PathPrefix(`/`)",
      "priority": 2147483645,
      "status": "enabled",
      "using": [
        "traefik"
      ]
    },
    "reverse-proxy-root1@docker": {
      "service": "reverse-proxy-root1",
      "rule": "Host(`reverse-proxy-root1`)",
      "status": "enabled",
      "using": [
        "http",
        "traefik"
      ]
    },
    "test@docker": {
      "middlewares": [
        "test-headers@docker"
      ],
      "service": "test",
      "rule": "PathPrefix(`/test`)",
      "status": "enabled",
      "using": [
        "http",
        "traefik"
      ]
    }
  },
  "middlewares": {
    "dashboard_redirect@internal": {
      "redirectRegex": {
        "regex": "^(http:\\/\\/[^:\\/]+(:\\d+)?)\\/$",
        "replacement": "${1}/dashboard/",
        "permanent": true
      },
      "status": "enabled",
      "usedBy": [
        "dashboard@internal"
      ]
    },
    "dashboard_stripprefix@internal": {
      "stripPrefix": {
        "prefixes": [
          "/dashboard/",
          "/dashboard"
        ]
      },
      "status": "enabled",
      "usedBy": [
        "dashboard@internal"
      ]
    },
    "test-headers@docker": {
      "headers": {
        "customResponseHeaders": {
          "X-Custom-Response-Header": "value"
        }
      },
      "status": "enabled",
      "usedBy": [
        "test@docker"
      ]
    }
  },
  "services": {
    "api@internal": {
      "status": "enabled",
      "usedBy": [
        "api@internal"
      ]
    },
    "dashboard@internal": {
      "status": "enabled",
      "usedBy": [
        "dashboard@internal"
      ]
    },
    "reverse-proxy-root1@docker": {
      "loadBalancer": {
        "servers": [
          {
            "url": "http://172.27.0.2:80"
          }
        ],
        "passHostHeader": true
      },
      "status": "enabled",
      "usedBy": [
        "reverse-proxy-root1@docker"
      ],
      "serverStatus": {
        "http://172.27.0.2:80": "UP"
      }
    },
    "test@docker": {
      "loadBalancer": {
        "servers": [
          {
            "url": "http://172.27.0.3:5000"
          }
        ],
        "passHostHeader": true
      },
      "status": "enabled",
      "usedBy": [
        "test@docker"
      ],
      "serverStatus": {
        "http://172.27.0.3:5000": "UP"
      }
    }
  }
}