Problem with Passenger Enterprise 4.0.14

[fokcep]

Just wondering if someone had issues running tubesock with the Passenger Enterprise Server. Could be nothing to do with tubesock at all. It supposed to run ok. But getting a long delay when calling the following bit of code from tubesock:

def self.hijack(env)
  .......    
  handshake.from_rack env
  .......
end

it takes around 50 seconds to execute this method and eventually it fails to get WS connection.

Dump of env['rack.hijack']

Proc:0x00000004505060@/opt/passenger/passenger-enterprise-server-4.0.14/lib/phusion_passenger/rack/thread_handler_extension.rb:53 (lambda)

any ideas?

Thanks

[ngauthier]

I haven't tried it on anything but Puma and Rails. Seems like they have supported it for a while:

http://blog.phusion.nl/2013/01/23/the-new-rack-socket-hijacking-api/

Can you run through that tutorial and make sure you have proper rack.hijack support going? After that, you're braving new territory! Let me know how it goes.

[fokcep]

hm, it's interesting
I went through this tutorial which I've seen before and replaced the problem line of code in your hijack method (https://github.com/ngauthier/tubesock/blob/master/lib/tubesock.rb#L25) with the piece of code from the tutorial.

i.e. now instead of

def self.hijack(env)
  ...
  handshake = WebSocket::Handshake::Server.new
  handshake.from_rack env
  ...
end

I have

def self.rack_env_key_to_http_header_name(key)
    name = key.downcase.gsub('_', '-')
    name[0] = name[0].upcase
    name.gsub!(/-(.)/) do |chr|
      chr.upcase
    end
    name
end

def self.hijack(env)
  ...
  handshake = WebSocket::Handshake::Server.new
  data = "#{env['REQUEST_METHOD']} #{env['REQUEST_URI']} #{env['SERVER_PROTOCOL']}\r\n"
        env.each_pair do |key, val|
          if key =~ /^HTTP_(.*)/
            name = rack_env_key_to_http_header_name($1)
            data << "#{name}: #{val}\r\n"
          end
        end
   data << "\r\n"
   handshake << data
  ...
end

and now Puma seems to work fine and Passenger works too. I will do more tests tomorrow and let you know. But can you explain why 'handshake.from_rack env' might not work for Passenger but it works for Puma?

Cheers

[ngauthier]

I added that specifically to websocket-ruby for tubesock's sake:

imanel/websocket-ruby#15

I think the best thing to do would be to take the env and make a test for the headers and post an issue at websocket-ruby. Then all projects would benefit from it.

Nick

[fokcep]

Thanks for that!

I've done few tests with Passenger and looks like it's opening connections just fine but then no requests coming through from the clients or from the server. I.e. pings are not coming through.
So need to find out what it is. It's definitely not my load balancer. Something else

[ngauthier]

Actually, it could be anything in front of your web server. A year ago when I was playing w/ websockets we couldn't use nginx, we had to use haproxy in a special mode to keep the websocket connections alive. Load balancers usually don't keep tcp connections like websockets open like that.

I believe nginx has websocket support now, but you may have to compile from scratch.

Try without the load balancer (directly connect) just to troubleshoot. If that still doesn't work at least we really can rule out the balancer.

What is your balancer by the way?

[fokcep]

No, it's definitely not load balancer. I've tried to connect directly from the same box. So it could be nginx or something else.

I've noticed the following code doesn't return anything, it's stuck there on IO.select, which is strange

def each_frame
    ....
      while IO.select([@socket])
    ...
end

i.e. https://github.com/ngauthier/tubesock/blob/master/lib/tubesock.rb#L92

Will do some more digging

[ngauthier]

Hmm. Just to check, are you on Ruby 2?

[fokcep]

yep, ofc. I always check that everything works with Puma after I've done all my changes.

[fokcep]

Obviously we've built nginx 1.4.2 and passenger enterprise 4.0 and they both have web socket support.

[fokcep]

By the way, it seems that 'websocket-rails' library (which in turn uses Faye WebSocket which seems to use rack hijack but based on event machine) works on Passenger and nginx, and this chat application - https://github.com/themgt/ws42-chat works fine. I will try to find out whether tubesock can be improved based on that info.

[ngauthier]

OK cool. Yeah I know the project, but at the time it didn't have
fine-grained enough control for me, hence tubesock. Also yeah, the EM
difference.

On Mon, Sep 30, 2013 at 12:34 PM, fokcep notifications@github.com wrote:

By the way, it seems that 'websocket-rails' library (which in turn uses
Faye WebSocket which seems to use rack hijack but based on event machine)
works on Passenger and nginx, and this chat application -
https://github.com/themgt/ws42-chat works fine. I will try to find out
whether tubesock can be improved based on that info.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-25379416
.

[fokcep]

Looks like EM should run in the different thread, and if it runs in the same thread it locks up.
So websocket-rails doesn't work with Passenger after all. So still no luck.
Any idea why IO.select([@socket]) could lock up when using tubesock?
I am gonna ask Passenger support and see if they can make it work.

[FooBarWidget]

Phusion here. We got a support request from an Enterprise customer a while ago. We got Tubesock working now, and this is what we've found.

There are Rack specification violations in both Tubesock and the Websocket gem. Let me first address the freezing problem that you guys have already encountered. The freeze is caused by the fact that WebSocket::Handshake::Server#handshake calls the following code snippet:

@leftovers = env['rack.input'].read

Obviously, the intention is to slurp all remaining data on the socket.

This code path is not supposed to work because env['rack.input'].read is supposed to be blocking, and at this point the client is waiting for the server to send a reply, so the code would never have done what it intended to. And this is actually the reason why the code freezes in #handshake when run on Phusion Passenger.

This code path happens to work on Puma because Puma replaces env['rack.input'] with a NullIO object right after performing hijacking. The NullIO object turns the env['rack.input'].read call into a no-op. So things work on Puma by accident, while on Passenger the code gets stuck on the read, forever trying to read data.

There is also a Rack specification violation in Tubesock itself. From tubesock.rb line 93:

data, addrinfo = @socket.recvfrom(2000)

According to the Rack specification, the hijacked socket object is not guaranteed to implement #recvfrom. An indeed, Phusion Passenger does not, while Puma does. Both Phusion Passenger and Puma happen to implement #readpartial so you can use that as a more portable (albeit still non-standard) substitute. Starting from version 4.0.20, Phusion Passenger will implement #recvfrom as well.

Finally, we confirm that Tubesock works fine even with the open source version of Phusion Passenger. Enterprise is not necessary.

[fokcep]

Interesting, thanks for that.

I don't seem to have a problem calling from_rack after fixing it in a different way, see above. I.e. opening connection now works but sending any subsequent WS message failed i.e. locks up in the LIne 92. Could it be something else in Passenger? A config option or something?

[FooBarWidget]

What do you mean by "locks up"? Isn't it normal that the select call there blocks until the client sends something over the websocket?

Replacing the @leftovers = ... line in the websocket gem with @leftovers = "", and replacing the recvfrom call with readpartial, makes the sock-chat demo work for us.