Traefik >= 2.7.2 changes ";" to "&" in URL request string

[wollomatic]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

I have a request like https://whoami.example.com/?testurl=http://test/?a=1;b=2;c=3

I expect the request to be GET /?testurl=http://test/?a=1;b=2;c=3 HTTP/1.1

The issue can be reproduced with using

What did you see instead?

Request is GET /?testurl=http://test/?a=1&b=2&c=3 HTTP/1.1

What version of Traefik are you using?

Issue starts with version 2.7.2

Version:      2.7.3
Codename:     epoisses
Go version:   go1.17.11
Built:        2022-06-29T13:48:15Z
OS/Arch:      linux/amd64

What is your environment & configuration?

services:

  whoami:
    image: "containous/whoami"
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.wwhoami.entrypoints=web-secure"
      - "traefik.http.routers.wwhoami.rule=Host(`whoami.example.com`)"
      - "traefik.http.routers.wwhoami.tls=true"
      - "traefik.http.routers.wwhoami.tls.certresolver=acme"
      - "traefik.http.routers.wwhoami.middlewares=secHeaders@file"
      - "traefik.http.services.wwhoami.loadbalancer.server.port=80"
    networks:
      - traefik_servicenet

networks:
  traefik_servicenet:
    external: true
    name: traefik_servicenet

Configuration like https://github.com/wollomatic/traefik2-hardened with a few changes such as http3 disabled.

If applicable, please paste the log output in DEBUG level

No response

[No13]

I was literally just creating a bug report for this as well :)

Steps to reproduce:

Backend:
docker run --label "traefik.http.routers.testsvc.rule=Host(\`test.localhost\`)" nginx

Traefik v2.6:
docker run -v /var/run/docker.sock:/var/run/docker.sock -p 8000:80 traefik:v2.6 --log.level=DEBUG --providers.docker

Result
172.17.0.3 - - [05/Jul/2022:08:12:16 +0000] "GET /test?param=1;2;3 HTTP/1.1" 404 555 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" "172.17.0.1"

Traefik v2.7:
docker run -v /var/run/docker.sock:/var/run/docker.sock -p 8000:80 traefik:v2.7 --log.level=DEBUG --providers.docker

Result @ nginx
172.17.0.3 - - [05/Jul/2022:08:12:52 +0000] "GET /test?param=1&2&3 HTTP/1.1" 404 555 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" "172.17.0.1"

Result @ traefik:v2.7 debuglog

time="2022-07-05T08:20:23Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/test\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"param=1\\u00262\\u00263\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-NL,en;q=0.9,nl-NL;q=0.8,nl;q=0.7,en-GB;q=0.6,en-US;q=0.5\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"102\\\", \\\"Google Chrome\\\";v=\\\"102\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Linux\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36\"],\"X-Forwarded-Host\":[\"test.localhost:8000\"],\"X-Forwarded-Port\":[\"8000\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"519eac695821\"],\"X-Real-Ip\":[\"172.17.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"test.localhost:8000\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.17.0.1:53576\",\"RequestURI\":\"/test?param=1;2;3\",\"TLS\":null}"
time="2022-07-05T08:20:23Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://172.17.0.3:80" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/test\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"param=1\\u00262\\u00263\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-NL,en;q=0.9,nl-NL;q=0.8,nl;q=0.7,en-GB;q=0.6,en-US;q=0.5\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"102\\\", \\\"Google Chrome\\\";v=\\\"102\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Linux\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36\"],\"X-Forwarded-Host\":[\"test.localhost:8000\"],\"X-Forwarded-Port\":[\"8000\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"519eac695821\"],\"X-Real-Ip\":[\"172.17.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"test.localhost:8000\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.17.0.1:53576\",\"RequestURI\":\"/test?param=1;2;3\",\"TLS\":null}"
time="2022-07-05T08:20:23Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/test\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"param=1\\u00262\\u00263\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-NL,en;q=0.9,nl-NL;q=0.8,nl;q=0.7,en-GB;q=0.6,en-US;q=0.5\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"102\\\", \\\"Google Chrome\\\";v=\\\"102\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Linux\\\"\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36\"],\"X-Forwarded-Host\":[\"test.localhost:8000\"],\"X-Forwarded-Port\":[\"8000\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"519eac695821\"],\"X-Real-Ip\":[\"172.17.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"test.localhost:8000\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.17.0.1:53576\",\"RequestURI\":\"/test?param=1;2;3\",\"TLS\":null}"

[ddtmachado]

Just to clarify, this change was introduced in #9131 as a way to fix some odd issues with the Query matcher that would ignore semicolon arguments otherwise.

That means we are capable of interpreting the semicolon as an argument separator for backward compatibility but we also now normalize / replace it with & before forwarding to the backend as in the most recent spec recommendation it became illegal to use ; as a separator. This last move should not have any impact in backend applications as they should be capable of handling & as the separator anyway.

Let us know if we missed a real use case where the backend is unable to handle & as the separator.

[No13]

In my case it is domoticz which broke.. changing settings or updating values breaks when replacing the semicolon. I will check if there are updated versions available

Update: Domoticz will not play ball when replacing semicolon with ampersand, it expects a single string which it can split into several values on the semicolon...

I guess I understand why you would want to fix the query matcher, but is it a valid reason to pass the changed request to the backend? Why not use the replaced values for the matcher only, but send the unmodified string to the backend server?

[wollomatic]

In our case my company processes digital identification links for specific components. Those links can look like
https://example.com/?v=2;i=536;k=1NpBl
These URLs are individual on each component and are placed als a QR-code.

And so the URL https://example.com/?v=2&i=536&k=1NpBl would be some other component and would not match the scanned component.

[ldez]

FYI the encoded elements of an URL are not affected by the change, only non-encoded URLs are affected.
The RFC about URL asks to encode the query parameters.

[KeldorDE]

Same here, I downgraded Traefik to version 2.7.1, so it works for now.

[Olivier6767]

I had the same problem passing information to Domoticz through Traefik since release 2.7.2.
Example URL: https://domoticz.redacted.com:443/json.htm?type=command&param=udevice&idx=789&nvalue=0&svalue=13218988;5080219;0;0;0;0

I modified my scripts to ensure the content of the svalue parameter is urlencoded.
New URL: https://domoticz.redacted.com:443/json.htm?type=command&param=udevice&idx=789&nvalue=0&svalue=13218988%3B5080219%3B0%3B0%3B0%3B0

This solves the issue.

[No13]

This works for scripts within my own sphere of control. But try to change user permissions and notice that domoticz itself will not urlencode the permissions... Not that this issue should be resolved by traefik. But I still wonder why the request itself needs to be rewritten instead of doing the parsing and checking on a modified copy of the request while still preserving the original request when talking to the real backend servers