Websocket: Unable to hijack the connection #452

ydemartino · 2016-06-11T05:55:56Z

Hi,

I have a hard time understanding this issue.
I use traefik 1.0.0-rc2 with consul discovery. Everything works fine except the websocket. When I try to connect to my websocket I see this log:
WARN Unable to hijack the connection:

I tried to monitor the connection with wireshark. What I can see is that traefik is opening the connection to the server but there is nothing more... Just SYN/ACK.

When I try to access another non-websocket URL from this backend, it works.

I tried to setup the same configuration using the file backend approach. With this configuration the websocket works normally.

Can you help me to figure out why this does not work with consul discovery?

Thank you!

samber · 2016-06-11T09:57:58Z

It's probably not a problem specific to consul backend.

What version of Traefik are you using ?

$ traefik version

We had a regression last month. It has been fixed in recent released.

emilevauge · 2016-06-11T10:41:22Z

It seems to be on rc2. Could you give us some DEBUG logs?

ydemartino · 2016-06-11T14:44:21Z

traefik version returns:

Version:      v1.0.0-rc2
Go version:   go1.6.2
Built:        2016-06-08_12:40:47PM
OS/Arch:      linux/arm

How can I get some DEBUG logs? I tried starting traefik with -d and -l DEBUG.
I have a list of DEBUG logs during the start up and the consul configuration, but nothing for the request except the warning:

time="2016-06-11T16:37:28+02:00" level=info msg="Traefik version v1.0.0-rc2 built on 2016-06-08_12:40:47PM"
time="2016-06-11T16:37:28+02:00" level=debug msg="Global configuration loaded {\"GraceTimeOut\":10,\"Debug\":true,\"AccessLogsFile\":\"\",\"TraefikLogsFile\":\"traefik.logs\",\"LogLevel\":\"DEBUG\",\"EntryPoints\":{\"http\":{\"Network\":\"\",\"Address\":\":80\",\"TLS\":null,\"Redirect\":null}},\"Constraints\":[],\"ACME\":null,\"DefaultEntryPoints\":[\"http\"],\"ProvidersThrottleDuration\":2000000000,\"MaxIdleConnsPerHost\":200,\"Retry\":null,\"Docker\":null,\"File\":null,\"Web\":{\"Address\":\":8080\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":false},\"Marathon\":null,\"Consul\":null,\"ConsulCatalog\":{\"Watch\":false,\"Filename\":\"traefik_consul.tmpl\",\"Constraints\":[],\"Endpoint\":\"127.0.0.1:8500\",\"Domain\":\"my-domain.com\",\"Prefix\":\"\"},\"Etcd\":null,\"Zookeeper\":null,\"Boltdb\":null,\"Kubernetes\":null}" 
time="2016-06-11T16:37:28+02:00" level=info msg="Preparing server http &{Network: Address::80 TLS:<nil> Redirect:<nil>}"
time="2016-06-11T16:37:28+02:00" level=info msg="Starting provider *main.WebProvider {\"Address\":\":8080\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":false}" 
time="2016-06-11T16:37:28+02:00" level=info msg="Starting server on :80"
time="2016-06-11T16:37:28+02:00" level=info msg="Starting provider *provider.ConsulCatalog {\"Watch\":false,\"Filename\":\"traefik_consul.tmpl\",\"Constraints\":[],\"Endpoint\":\"127.0.0.1:8500\",\"Domain\":\"my-domain.com\",\"Prefix\":\"\"}" 
time="2016-06-11T16:37:28+02:00" level=debug msg="List of services changed"
time="2016-06-11T16:37:28+02:00" level=debug msg="Fetching service" service=security
time="2016-06-11T16:37:28+02:00" level=debug msg="Fetching service" service=user
time="2016-06-11T16:37:28+02:00" level=debug msg="Fetching service" service=whisper
time="2016-06-11T16:37:28+02:00" level=debug msg="Fetching service" service=consul
time="2016-06-11T16:37:28+02:00" level=debug msg="Fetching service" service=location
time="2016-06-11T16:37:28+02:00" level=debug msg="Configuration received from provider consul_catalog: {\"backends\":{\"backend-location\":{\"servers\":{\"location--192-168-0-250--8082--traefik-backend-loadbalancer-drr--traefik-frontend-rule-PathPrefixStrip-location--4\":{\"url\":\"http://192.168.0.250:8082\",\"weight\":0}},\"loadBalancer\":{\"method\":\"drr\"}},\"backend-security\":{\"servers\":{\"security--192-168-0-250--8070--traefik-backend-loadbalancer-drr--traefik-frontend-rule-PathPrefixStrip-security--0\":{\"url\":\"http://192.168.0.250:8070\",\"weight\":0}},\"loadBalancer\":{\"method\":\"drr\"}},\"backend-user\":{\"servers\":{\"user--192-168-0-250--8081--traefik-backend-loadbalancer-drr--traefik-frontend-rule-PathPrefixStrip-user--1\":{\"url\":\"http://192.168.0.250:8081\",\"weight\":0}},\"loadBalancer\":{\"method\":\"drr\"}},\"backend-whisper\":{\"servers\":{\"whisper--192-168-0-250--8008--traefik-backend-loadbalancer-drr--traefik-frontend-rule-PathPrefixStrip-whisper--2\":{\"url\":\"http://192.168.0.250:8008\",\"weight\":0}},\"loadBalancer\":{\"method\":\"drr\"}}},\"frontends\":{\"frontend-location\":{\"backend\":\"backend-location\",\"routes\":{\"route-host-location\":{\"rule\":\"PathPrefixStrip:/location\"}},\"passHostHeader\":true,\"priority\":0},\"frontend-security\":{\"backend\":\"backend-security\",\"routes\":{\"route-host-security\":{\"rule\":\"PathPrefixStrip:/security\"}},\"passHostHeader\":true,\"priority\":0},\"frontend-user\":{\"backend\":\"backend-user\",\"routes\":{\"route-host-user\":{\"rule\":\"PathPrefixStrip:/user\"}},\"passHostHeader\":true,\"priority\":0},\"frontend-whisper\":{\"backend\":\"backend-whisper\",\"routes\":{\"route-host-whisper\":{\"rule\":\"PathPrefixStrip:/whisper\"}},\"passHostHeader\":true,\"priority\":0}}}" 
time="2016-06-11T16:37:28+02:00" level=debug msg="Last consul_catalog config received more than 2s, OK"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating frontend frontend-location"
time="2016-06-11T16:37:28+02:00" level=debug msg="Wiring frontend frontend-location to entryPoint http"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating route route-host-location PathPrefixStrip:/location"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating backend backend-location"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating load-balancer drr"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating server location--192-168-0-250--8082--traefik-backend-loadbalancer-drr--traefik-frontend-rule-PathPrefixStrip-location--4 at http://192.168.0.250:8082 with weight 0"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating frontend frontend-security"
time="2016-06-11T16:37:28+02:00" level=debug msg="Wiring frontend frontend-security to entryPoint http"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating route route-host-security PathPrefixStrip:/security"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating backend backend-security"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating load-balancer drr"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating server security--192-168-0-250--8070--traefik-backend-loadbalancer-drr--traefik-frontend-rule-PathPrefixStrip-security--0 at http://192.168.0.250:8070 with weight 0" 
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating frontend frontend-user"
time="2016-06-11T16:37:28+02:00" level=debug msg="Wiring frontend frontend-user to entryPoint http"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating route route-host-user PathPrefixStrip:/user"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating backend backend-user"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating load-balancer drr"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating server user--192-168-0-250--8081--traefik-backend-loadbalancer-drr--traefik-frontend-rule-PathPrefixStrip-user--1 at http://192.168.0.250:8081 with weight 0"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating frontend frontend-whisper"
time="2016-06-11T16:37:28+02:00" level=debug msg="Wiring frontend frontend-whisper to entryPoint http"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating route route-host-whisper PathPrefixStrip:/whisper"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating backend backend-whisper"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating load-balancer drr"
time="2016-06-11T16:37:28+02:00" level=debug msg="Creating server whisper--192-168-0-250--8008--traefik-backend-loadbalancer-drr--traefik-frontend-rule-PathPrefixStrip-whisper--2 at http://192.168.0.250:8008 with weight 0" 
time="2016-06-11T16:37:28+02:00" level=info msg="Server configuration reloaded on :80"
time="2016-06-11T16:37:39+02:00" level=warning msg="Unable to hijack the connection: <nil>"
time="2016-06-11T16:37:42+02:00" level=info msg="I have to go... interrupt"

emilevauge · 2016-06-11T14:51:52Z

Ok thanks. Last thing, what is your toml file (or command line)?

ydemartino · 2016-06-11T15:40:42Z

The command line is:
traefik --defaultentrypoints=http --consulcatalog.domain=my-domain.com --consulcatalog.filename=traefik_consul.tmpl --web -d -l DEBUG

traefik_consul.tmpl is the same as the original template except it skips consul:

[backends]
{{range $index, $node := .Nodes}}
  {{if ne (getAttribute "enable" $node.Service.Tags "true") "false"}}
  {{if ne $node.Service.Service "consul"}}
    [backends.backend-{{getBackend $node}}.servers.{{getBackendName $node $index}}]
      url = "{{getAttribute "protocol" $node.Service.Tags "http"}}://{{getBackendAddress $node}}:{{$node.Service.Port}}"
      {{$weight := getAttribute "backend.weight" $node.Service.Tags ""}}
      {{with $weight}}
        weight = {{$weight}}
      {{end}}
    {{end}}
   {{end}}
{{end}}

{{range .Services}}
  {{$service := .ServiceName}}
  {{$circuitBreaker := getAttribute "backend.circuitbreaker" .Attributes ""}}
  {{with $circuitBreaker}}
  [backends.backend-{{$service}}.circuitbreaker]
    expression = "{{$circuitBreaker}}"
  {{end}}

  {{$loadBalancer := getAttribute "backend.loadbalancer" .Attributes ""}}
  {{with $loadBalancer}}
  [backends.backend-{{$service}}.loadbalancer]
    method = "{{$loadBalancer}}"
  {{end}}
{{end}}

[frontends]
{{range .Services}}
  {{if ne .ServiceName "consul"}}
  [frontends.frontend-{{.ServiceName}}]
  backend = "backend-{{.ServiceName}}"
  passHostHeader = {{getAttribute "frontend.passHostHeader" .Attributes "true"}}
  {{$entryPoints := getAttribute "frontend.entrypoints" .Attributes ""}}
  {{with $entryPoints}}
    entrypoints = [{{range getEntryPoints $entryPoints}}
      "{{.}}",
    {{end}}]
  {{end}}
  [frontends.frontend-{{.ServiceName}}.routes.route-host-{{.ServiceName}}]
    rule = "{{getFrontendRule .}}"
  {{end}}
{{end}}

adamgoose · 2016-06-13T00:37:33Z

I'm getting the same error (without the crash) while using the file config driver. My back-end is Mattermost, fwiw.

ydemartino · 2016-06-13T07:57:09Z

To clarify, the last log time="2016-06-11T16:37:42+02:00" level=info msg="I have to go... interrupt" is me hitting CTRL+C
I ran another test using the default consul catalog template and I get the same error.

adamgoose · 2016-06-13T07:58:22Z

Oh, lol. Thanks for letting me know. :)

On Mon, Jun 13, 2016 at 2:57 AM ydemartino notifications@github.com wrote:

To clarify, the last log time="2016-06-11T16:37:42+02:00" level=info
msg="I have to go... interrupt" is me hitting CTRL+C
I ran another test using the default consul catalog template and I get the
same error.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#452 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAlS_EuoecRteFUW_7ftcWy7r0Gp4FlQks5qLQ1bgaJpZM4Izew8
.

emilevauge · 2016-06-15T09:26:59Z

It seems [retry] is not able to handle websockets...
Investigating on this.
The current workaround is to disable retry for now.

adamgoose · 2016-06-15T09:38:02Z

Here's my Traefik config. Running in a Docker container, with no specified CMD. Seems like I do not have [retry] enabled, but still getting the 500.

https://gist.github.com/adamgoose/45bccd014d9ca82c81ad36cb536c711d

emilevauge · 2016-06-15T10:13:38Z

After more investigations with @adamgoose, here are options that don't work with websockets:

retry
circuit breakers
ddr load balancer

These issues are in https://github.com/vulcand/oxy.
I will try to fix them in the fork https://github.com/containous/oxy.

adamgoose · 2016-06-15T10:14:34Z

Thank you! :D

ydemartino · 2016-06-15T13:16:25Z

I confirm, removing ddr load balancer from my config fixes the issue. Thank you!

emilevauge added the investigation-needed label Jun 12, 2016

emilevauge mentioned this issue Jun 15, 2016

Fix websocket connection Hijack #460

Merged

vdemeester closed this as completed in #460 Jun 18, 2016

yuriy-zhilovets2 mentioned this issue Mar 19, 2017

websocket: Unable to hijack the connection #1315

Closed

ldez removed the investigation-needed label May 12, 2017

ldez added the area/websocket label Feb 15, 2018

traefik locked and limited conversation to collaborators Sep 1, 2019

traefiker added the status/5-frozen-due-to-age label Sep 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Websocket: Unable to hijack the connection #452

Websocket: Unable to hijack the connection #452

ydemartino commented Jun 11, 2016 •

edited by emilevauge

samber commented Jun 11, 2016 •

edited

emilevauge commented Jun 11, 2016

ydemartino commented Jun 11, 2016

emilevauge commented Jun 11, 2016

ydemartino commented Jun 11, 2016

adamgoose commented Jun 13, 2016

ydemartino commented Jun 13, 2016

adamgoose commented Jun 13, 2016

emilevauge commented Jun 15, 2016

adamgoose commented Jun 15, 2016 •

edited

emilevauge commented Jun 15, 2016

adamgoose commented Jun 15, 2016

ydemartino commented Jun 15, 2016

Websocket: Unable to hijack the connection #452

Websocket: Unable to hijack the connection #452

Comments

ydemartino commented Jun 11, 2016 • edited by emilevauge

samber commented Jun 11, 2016 • edited

emilevauge commented Jun 11, 2016

ydemartino commented Jun 11, 2016

emilevauge commented Jun 11, 2016

ydemartino commented Jun 11, 2016

adamgoose commented Jun 13, 2016

ydemartino commented Jun 13, 2016

adamgoose commented Jun 13, 2016

emilevauge commented Jun 15, 2016

adamgoose commented Jun 15, 2016 • edited

emilevauge commented Jun 15, 2016

adamgoose commented Jun 15, 2016

ydemartino commented Jun 15, 2016

ydemartino commented Jun 11, 2016 •

edited by emilevauge

samber commented Jun 11, 2016 •

edited

adamgoose commented Jun 15, 2016 •

edited