Host header include `%3a80`

[guilhem]

Hi all,

My context:

• kubernetes 1.2.2 (gke)
• latest docker image
• ingress like that

nothing too exotic :)

But when I was doing a curl, it always return a 301 (very frustrating).

After some debug, I find that træfik was using a bad Host header like that

Host:10.3.240.178%3a80
User-Agent:curl/7.26.0
[...]

And apache didn't like it (and he is right for one time...).

It seems that there is a problem on how is performed Host on backend 😞

[guilhem]

Very simple debug:

$ curl -IL http://127.0.0.1
HTTP/1.1 200 OK
Date: Wed, 27 Apr 2016 16:30:47 GMT
Server: Apache/2.4.10 (Debian) PHP/5.6.20
X-Powered-By: PHP/5.6.20
Link: <http://foo.bar.com/wp-json/>; rel="https://api.w.org/"
Content-Type: text/html; charset=UTF-8

$ curl -IL http://127.0.0.1 -H"host:foo.bar.com"
HTTP/1.1 200 OK
Date: Wed, 27 Apr 2016 16:31:10 GMT
Server: Apache/2.4.10 (Debian) PHP/5.6.20
X-Powered-By: PHP/5.6.20
Link: <http://foo.bar.com/wp-json/>; rel="https://api.w.org/"
Content-Type: text/html; charset=UTF-8

$ curl -IL http://127.0.0.1 -H"host:foo.bar.com:80"
HTTP/1.1 301 Moved Permanently
Date: Wed, 27 Apr 2016 16:31:17 GMT
Server: Apache/2.4.10 (Debian) PHP/5.6.20
X-Powered-By: PHP/5.6.20
Location: http://foo.bar.com/
Content-Type: text/html; charset=UTF-8

[guilhem]

seems to be related to golang/go#4172

[emilevauge]

Whoah, there is an URL encoding issue somewhere...
I can't reproduce it.
Could you give us DEBUG traefik entire logs?

[ekozan]

2016-04-27T19:17:06.734955487Z 2016/04/27 19:17:06 connection accepted 1
2016-04-27T19:17:06.735763593Z 2016/04/27 19:17:06 <->
2016-04-27T19:17:06.735768613Z 00000000  47 45 54 20 2f 20 48 54  54 50 2f 31 2e 31 0d 0a  |GET / HTTP/1.1..|
2016-04-27T19:17:06.735773192Z 00000010  48 6f 73 74 3a 20 31 30  2e 33 2e 31 38 35 2e 38  |Host: 10.3.185.8|
2016-04-27T19:17:06.735777693Z 00000020  30 3a 38 38 30 30 0d 0a  55 73 65 72 2d 41 67 65  |0:8800..User-Age|
2016-04-27T19:17:06.735782088Z 00000030  6e 74 3a 20 4d 6f 7a 69  6c 6c 61 2f 35 2e 30 20  |nt: Mozilla/5.0 |
2016-04-27T19:17:06.735786516Z 00000040  28 4d 61 63 69 6e 74 6f  73 68 3b 20 49 6e 74 65  |(Macintosh; Inte|
2016-04-27T19:17:06.735791004Z 00000050  6c 20 4d 61 63 20 4f 53  20 58 20 31 30 5f 31 31  |l Mac OS X 10_11|
2016-04-27T19:17:06.735795556Z 00000060  29 20 41 70 70 6c 65 57  65 62 4b 69 74 2f 36 30  |) AppleWebKit/60|
2016-04-27T19:17:06.735800036Z 00000070  31 2e 31 2e 35 36 20 28  4b 48 54 4d 4c 2c 20 6c  |1.1.56 (KHTML, l|
2016-04-27T19:17:06.735804847Z 00000080  69 6b 65 20 47 65 63 6b  6f 29 20 56 65 72 73 69  |ike Gecko) Versi|
2016-04-27T19:17:06.735809408Z 00000090  6f 6e 2f 39 2e 30 20 53  61 66 61 72 69 2f 36 30  |on/9.0 Safari/60|
2016-04-27T19:17:06.735813710Z 000000a0  31 2e 31 2e 35 36 0d 0a  41 63 63 65 70 74 3a 20  |1.1.56..Accept: |
2016-04-27T19:17:06.735818100Z 000000b0  74 65 78 74 2f 68 74 6d  6c 2c 61 70 70 6c 69 63  |text/html,applic|
2016-04-27T19:17:06.735822520Z 000000c0  61 74 69 6f 6e 2f 78 68  74 6d 6c 2b 78 6d 6c 2c  |ation/xhtml+xml,|
2016-04-27T19:17:06.735827030Z 000000d0  61 70 70 6c 69 63 61 74  69 6f 6e 2f 78 6d 6c 3b  |application/xml;|
2016-04-27T19:17:06.735831465Z 000000e0  71 3d 30 2e 39 2c 2a 2f  2a 3b 71 3d 30 2e 38 0d  |q=0.9,*/*;q=0.8.|
2016-04-27T19:17:06.735835818Z 000000f0  0a 41 63 63 65 70 74 2d  45 6e 63 6f 64 69 6e 67  |.Accept-Encoding|
2016-04-27T19:17:06.735840269Z 00000100  3a 20 67 7a 69 70 2c 20  64 65 66 6c 61 74 65 0d  |: gzip, deflate.|
2016-04-27T19:17:06.735844705Z 00000110  0a 41 63 63 65 70 74 2d  4c 61 6e 67 75 61 67 65  |.Accept-Language|
2016-04-27T19:17:06.735849258Z 00000120  3a 20 66 72 2d 66 72 0d  0a 58 2d 46 6f 72 77 61  |: fr-fr..X-Forwa|
2016-04-27T19:17:06.735853640Z 00000130  72 64 65 64 2d 46 6f 72  3a 20 37 38 2e 32 31 32  |rded-For: xx.xxx|
2016-04-27T19:17:06.735858029Z 00000140  xx xx xx xx xx xx xx xx  0a 58 2d 46 6f 72 77 61  |.xxx.xx..X-Forwa|
2016-04-27T19:17:06.735862378Z 00000150  72 64 65 64 2d 48 6f 73  74 3a 20 31 30 2e 33 2e  |rded-Host: 10.3.|
2016-04-27T19:17:06.735866751Z 00000160  31 38 35 2e 38 30 3a 38  38 30 30 0d 0a 58 2d 46  |185.80:8800..X-F|
2016-04-27T19:17:06.735871292Z 00000170  6f 72 77 61 72 64 65 64  2d 50 72 6f 74 6f 3a 20  |orwarded-Proto: |
2016-04-27T19:17:06.735875690Z 00000180  68 74 74 70 0d 0a 58 2d  46 6f 72 77 61 72 64 65  |http..X-Forwarde|
2016-04-27T19:17:06.735880697Z 00000190  64 2d 53 65 72 76 65 72  3a 20 74 72 61 65 66 69  |d-Server: traefi|
2016-04-27T19:17:06.735889176Z 000001a0  6b 2d 69 6e 67 72 65 73  73 2d 63 6f 6e 74 72 6f  |k-ingress-contro|
2016-04-27T19:17:06.735893852Z 000001b0  6c 6c 65 72 2d 37 33 6d  75 73 0d 0a 0d 0a        |ller-73mus....|
2016-04-27T19:17:06.759486575Z 2016/04/27 19:17:06 handle connection error, err=EOF

for me it's ok ( kubernetes 1.2.2 ( barrebone ) )

[emilevauge]

Me too, I suspect something is sending url encoded host header to traefik.

[guilhem]

more information for debug:

{
  "kubernetes": {
    "backends": {
      "foo.bar.com": {
        "servers": {
          "56c6cca4-0c8b-11e6-9bcb-42010af000e8": {
            "url": "http://10.3.240.178:80",
            "weight": 1
          }
        },
        "loadBalancer": {
          "method": "wrr"
        }
      }
    },
    "frontends": {
      "foo.bar.com": {
        "entryPoints": [
          "http"
        ],
        "backend": "foo.bar.com",
        "routes": {
          "foo.bar.com": {
            "rule": "Host:foo.bar.com"
          }
        }
      }
    }
  }
}

{
  "pid": 1,
  "uptime": "41m11.964036873s",
  "uptime_sec": 2471.964036873,
  "time": "2016-04-28 10:12:46.366580851 +0000 UTC",
  "unixtime": 1461838366,
  "status_code_count": {},
  "total_status_code_count": {
    "301": 28,
    "404": 3
  },
  "count": 0,
  "total_count": 31,
  "total_response_time": "1.41761211s",
  "total_response_time_sec": 1.41761211,
  "average_response_time": "45.729422ms",
  "average_response_time_sec": 0.045729422000000006
}

time="2016-04-28T10:06:22Z" level=debug msg="Received event from kubenetes map[type:ADDED object:map[kind:Ingress apiVersion:extensions/v1beta1 metadata:map[namespace:default selfLink:/apis/extensions/v1beta1/namespaces/default/ingresses/wordpress uid:dbc3e4ff-0d28-11e6-9bcb-42010af000e8 resourceVersion:199144 generation:1 creationTimestamp:2016-04-28T10:06:21Z name:wordpress] spec:map[rules:[map[host:foo.bar.com http:map[paths:[map[backend:map[serviceName:wpfrontend servicePort:80]]]]]]] status:map[loadBalancer:map[]]]]" 
time="2016-04-28T10:06:22Z" level=debug msg="Configuration received from provider kubernetes: {\"backends\":{\"foo.bar.com\":{\"servers\":{\"56c6cca4-0c8b-11e6-9bcb-42010af000e8\":{\"url\":\"http://10.3.240.178:80\",\"weight\":1}}}},\"frontends\":{\"foo.bar.com\":{\"backend\":\"foo.bar.com\",\"routes\":{\"foo.bar.com\":{\"rule\":\"Host:foo.bar.com\"}}}}}" 
time="2016-04-28T10:06:22Z" level=debug msg="Last kubernetes config received less than 2s, waiting..." 
time="2016-04-28T10:06:24Z" level=debug msg="Waited for kubernetes config, OK" 
time="2016-04-28T10:06:24Z" level=debug msg="Creating frontend foo.bar.com" 
time="2016-04-28T10:06:24Z" level=debug msg="Wiring frontend foo.bar.com to entryPoint http" 
time="2016-04-28T10:06:24Z" level=debug msg="Creating route foo.bar.com Host:foo.bar.com" 
time="2016-04-28T10:06:24Z" level=debug msg="Creating backend foo.bar.com" 
time="2016-04-28T10:06:24Z" level=debug msg="Creating load-balancer wrr" 
time="2016-04-28T10:06:24Z" level=debug msg="Creating server 56c6cca4-0c8b-11e6-9bcb-42010af000e8 at http://10.3.240.178:80 with weight 1" 
time="2016-04-28T10:06:24Z" level=info msg="Server configuration reloaded on :80" 
time="2016-04-28T10:06:25Z" level=debug msg="Round trip: http://10.3.240.178:80, code: 301, duration: 70.065096ms" 
time="2016-04-28T10:06:30Z" level=debug msg="Round trip: http://10.3.240.178:80, code: 301, duration: 47.242796ms" 
time="2016-04-28T10:06:35Z" level=debug msg="Round trip: http://10.3.240.178:80, code: 301, duration: 39.294382ms" 
time="2016-04-28T10:06:40Z" level=debug msg="Round trip: http://10.3.240.178:80, code: 301, duration: 45.900569ms"

[emilevauge]

@guilhem: traefik seems ok here, it seems that the strange header comes from your client. What client is making request to traefik?

[guilhem]

@emilevauge a simple curl: curl -i http://127.0.0.1:80/ -H"Host:foo.bar.com"

$ curl -v http://127.0.0.1:80/ -H"Host:foo.bar.com"
* About to connect() to 127.0.0.1 port 80 (#0)
*   Trying 127.0.0.1...
* connected
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.26.0
> Accept: */*
> Host:foo.bar.com
> 
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
< Date: Thu, 28 Apr 2016 10:20:51 GMT
< Location: http://10.3.240.178/
< Server: Apache/2.4.10 (Debian) PHP/5.6.20
< X-Powered-By: PHP/5.6.20
< 
* Connection #0 to host 127.0.0.1 left intact
* Closing connection #0

$ curl --version
curl 7.26.0 (x86_64-pc-linux-gnu) libcurl/7.26.0 OpenSSL/1.0.1e zlib/1.2.7 libidn/1.25 libssh2/1.4.2 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp 
Features: Debug GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

[guilhem]

on the apache side, here is complete log (with log_forensic module):

+439:5721e497:57|GET / HTTP/1.1|Host:10.3.240.178%3a80|User-Agent:curl/7.26.0|Accept:*/*|X-Forwarded-For:127.0.0.1|X-Forwarded-Host:10.3.240.178%3a80|X-Forwarded-Proto:http|X-Forwarded-Server:gke-test-clients-default-pool-b2614ade-boeo|Accept-Encoding:gzip

[guilhem]

@ekozan As I can see on your side there is same problem than me. Sended Host header is Host: 10.3.185.80:8800 so with host:port.
And apache seems to dislike it :)

It can be a problem on the apache2 side as træfik respect RFC

[guilhem]

sadly, I can't set passHostHeader with kubernetes
(maybe with annotations?)

[emilevauge]

@guilhem I need to know your architecture:

• what client: what makes request to traefik?
• what servers behind traefik?

host:port in Header is OK. The issue here is that : gets in %3a (url encoding) somewhere.

[guilhem]

@emilevauge as seen with logs from @ekozan behavior seems to be a "normal" one (træfik act the same way on both cases).

[emilevauge]

OK, then can you close this issue?

[guilhem]

I don't know. Do you want this behavior to happen? :)
Clearly it crash apache2 "as it".

(I'm currently testing with nginx, more to come)