Load balancing on processes without concurrency (Meteor) #1322

Closed
charlesvallieres opened this Issue Dec 13, 2014 · 4 comments

Comments

Projects
None yet
2 participants
@charlesvallieres

Hi,

I am pretty sure there is an issue with load balancing clients on processes when concurrency is 0.

First, let me give you some context :

  1. We're using Meteor (node.js)
  2. We're using sticky sessions (for websockets)
  3. Node.js has no threads (concurrency == 0)
  4. We're using multiple processes

Even if we're using sticky sessions users should be load-balanced across processes on their first request. Surely the load balancing can't be 100% perfect because we're using sticky sessions, but I think it should be reasonable. When inspecting processes with sudo passenger-status I constantly see some processes with 300+ sessions while some others have only 1 session and it slows down our app.

I installed the vagrant box and reproduced the problem, here are my steps :

  1. Install the dev. vagrant box & ssh in
  2. Install meteor
  3. Create a new meteor app (meteor create)
  4. Bundle & build the app (http://docs.meteor.com/#/full/deploying)
  5. nginx.conf : https://gist.github.com/charlesvallieres/a0f610b0ce1699549b52
  6. Start nginx
  7. Load the app with your browser to make sure all processes are up and running (so it's not #1317)
  8. Send ~ 200 websocket (DDP) connections to app : https://gist.github.com/charlesvallieres/0785c76009440cf5098e

I get results like this one :

  * PID: 24526   Sessions: 84      Processed: 1       Uptime: 9s
  * PID: 24545   Sessions: 1       Processed: 0       Uptime: 9s
  * PID: 24571   Sessions: 83      Processed: 0       Uptime: 8s
  * PID: 24596   Sessions: 1       Processed: 0       Uptime: 8s

The bug is resolved by changing this condition :
https://github.com/phusion/passenger/blob/stable-4.0/ext/common/ApplicationPool2/Process.h#L551

If I change return 1; by return sessions; it fixes everything and clients are load-balanced across processes :

  * PID: 27056   Sessions: 38      Processed: 6       Uptime: 8s
  * PID: 27075   Sessions: 38      Processed: 0       Uptime: 8s
  * PID: 27103   Sessions: 38      Processed: 0       Uptime: 7s
  * PID: 27137   Sessions: 37      Processed: 0       Uptime: 6s

I obviously don't know the whole codebase, making this change may or may not have some huge consequences. To my knowledge, busyness() seems mainly used to order processes. (https://github.com/phusion/passenger/blob/stable-4.0/ext/common/ApplicationPool2/Group.h#L452)

Would changing the condition to a simple return sessions; (maxed out at INT_MAX or something) be a viable solution?

Thank you,

Charles

@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
@FooBarWidget

FooBarWidget Dec 18, 2014

Member

You might be right. Let me investigate this further.

Member

FooBarWidget commented Dec 18, 2014

You might be right. Let me investigate this further.

@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
@FooBarWidget

FooBarWidget Dec 18, 2014

Member

If you are right, then it would obsolete issue #1317.

Member

FooBarWidget commented Dec 18, 2014

If you are right, then it would obsolete issue #1317.

@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
@FooBarWidget

FooBarWidget Dec 18, 2014

Member

This change is ok. I will port this to both the 4.0 and 5.0 branches. Thanks for analyzing this!

Member

FooBarWidget commented Dec 18, 2014

This change is ok. I will port this to both the 4.0 and 5.0 branches. Thanks for analyzing this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment