This repository has been archived by the owner. It is now read-only.

Cluster workers are confused? #4091

Closed
hemanth opened this Issue Oct 7, 2012 · 17 comments

Comments

Projects
None yet
6 participants
@hemanth
Member

hemanth commented Oct 7, 2012

Did some code with cluster, that looks like this on a quad core system only 2 workers are responding for each request from the browser and for cli curl calls only one of them is responding...Am I missing something? 💫

@bnoordhuis

This comment has been minimized.

Show comment Hide comment
@bnoordhuis

bnoordhuis Oct 7, 2012

Member

Unless you can type really fast, curl is not a great benchmarking tool. What do the numbers look like when you use ab or siege?

Member

bnoordhuis commented Oct 7, 2012

Unless you can type really fast, curl is not a great benchmarking tool. What do the numbers look like when you use ab or siege?

@hemanth

This comment has been minimized.

Show comment Hide comment
@hemanth

hemanth Oct 8, 2012

Member

Well, tried ab -n 1000 -c 5 http://127.0.0.1:8000/ it servered 1002 requests. It should have been 1004 is in't?

Member

hemanth commented Oct 8, 2012

Well, tried ab -n 1000 -c 5 http://127.0.0.1:8000/ it servered 1002 requests. It should have been 1004 is in't?

@langpavel

This comment has been minimized.

Show comment Hide comment
@langpavel

langpavel Oct 8, 2012

@hemanth Look at gist https://gist.github.com/3850507, run ab (ApacheBench)
Oh, sorry, you already do it, but try to increase concurrency

@hemanth Look at gist https://gist.github.com/3850507, run ab (ApacheBench)
Oh, sorry, you already do it, but try to increase concurrency

@hemanth

This comment has been minimized.

Show comment Hide comment
@hemanth

hemanth Oct 8, 2012

Member

@langpavel The question is on a quad core machine, when 4 workers are running only 2 are severing the requests, why so?

Member

hemanth commented Oct 8, 2012

@langpavel The question is on a quad core machine, when 4 workers are running only 2 are severing the requests, why so?

@langpavel

This comment has been minimized.

Show comment Hide comment
@langpavel

langpavel Oct 8, 2012

@hemanth Ok, I modified code you referenced at issue report to spawn 4 workers on my 2 core CPU and I see every worker dispatch something. https://gist.github.com/3850827
both versions works for me, from master and v0.8.11

What is your HW, bare metal or virtual, check process affinity, I really don't know....

@hemanth Ok, I modified code you referenced at issue report to spawn 4 workers on my 2 core CPU and I see every worker dispatch something. https://gist.github.com/3850827
both versions works for me, from master and v0.8.11

What is your HW, bare metal or virtual, check process affinity, I really don't know....

@bnoordhuis

This comment has been minimized.

Show comment Hide comment
@bnoordhuis

bnoordhuis Oct 8, 2012

Member

on a quad core machine, when 4 workers are running only 2 are severing the requests

Could be one of several reasons:

  • other processes (like ab) are tying up the other two cores
  • ab is not stressing the server enough, try ab -c 200 -n 50000
  • the accept() back-off mechanism in v0.8 is not very effective, you may get more balanced numbers with master

If you're running your benchmarks on OS X, try a real operating system.

Member

bnoordhuis commented Oct 8, 2012

on a quad core machine, when 4 workers are running only 2 are severing the requests

Could be one of several reasons:

  • other processes (like ab) are tying up the other two cores
  • ab is not stressing the server enough, try ab -c 200 -n 50000
  • the accept() back-off mechanism in v0.8 is not very effective, you may get more balanced numbers with master

If you're running your benchmarks on OS X, try a real operating system.

@hemanth

This comment has been minimized.

Show comment Hide comment
@hemanth

hemanth Oct 8, 2012

Member

@bnoordhuis heh heh liked your last point!

  • With curl/browser it's still the same
  • ab is not eating much of the cores.
  • Hmm what better than accept() ?

@langpavel ^ 😄

Member

hemanth commented Oct 8, 2012

@bnoordhuis heh heh liked your last point!

  • With curl/browser it's still the same
  • ab is not eating much of the cores.
  • Hmm what better than accept() ?

@langpavel ^ 😄

@bnoordhuis

This comment has been minimized.

Show comment Hide comment
@bnoordhuis

bnoordhuis Oct 8, 2012

Member

@hemanth Try the master branch, it uses a revised algorithm. Run it with export UV_TCP_SINGLE_ACCEPT=1 and export UV_TCP_SINGLE_ACCEPT=0.

Member

bnoordhuis commented Oct 8, 2012

@hemanth Try the master branch, it uses a revised algorithm. Run it with export UV_TCP_SINGLE_ACCEPT=1 and export UV_TCP_SINGLE_ACCEPT=0.

@hemanth

This comment has been minimized.

Show comment Hide comment
@hemanth

hemanth Oct 8, 2012

Member

ok 👍

Member

hemanth commented Oct 8, 2012

ok 👍

@christophsturm

This comment has been minimized.

Show comment Hide comment
@christophsturm

christophsturm Oct 23, 2012

@bnoordhuis:

"the accept() back-off mechanism in v0.8 is not very effective, you may get more balanced numbers with master"

is there a way to get this into 0.8?

@bnoordhuis:

"the accept() back-off mechanism in v0.8 is not very effective, you may get more balanced numbers with master"

is there a way to get this into 0.8?

@bnoordhuis

This comment has been minimized.

Show comment Hide comment
@bnoordhuis

bnoordhuis Oct 23, 2012

Member

@christophsturm In an official v0.8 release? No, because it changes the C/C++ ABI. You're free to back-port it though, it's pretty straightforward. I heard your company employs one or two people who know node's internals somewhat, maybe they can help you. ;-)

Member

bnoordhuis commented Oct 23, 2012

@christophsturm In an official v0.8 release? No, because it changes the C/C++ ABI. You're free to back-port it though, it's pretty straightforward. I heard your company employs one or two people who know node's internals somewhat, maybe they can help you. ;-)

@christophsturm

This comment has been minimized.

Show comment Hide comment
@christophsturm

christophsturm Oct 23, 2012

lol, what company are you talking about? I always thought I'm the resident node guru here :)

lol, what company are you talking about? I always thought I'm the resident node guru here :)

@bnoordhuis

This comment has been minimized.

Show comment Hide comment
@bnoordhuis

bnoordhuis Oct 23, 2012

Member

Ah, sorry - I thought you were our (that is, Cloud9's) Christoph. :-)

Member

bnoordhuis commented Oct 23, 2012

Ah, sorry - I thought you were our (that is, Cloud9's) Christoph. :-)

@brettkiefer

This comment has been minimized.

Show comment Hide comment
@brettkiefer

brettkiefer Dec 10, 2012

@christophsturm Did that do the trick for you? We seem to be seeing really serious imbalances with Cluster and node 0.9.3 with the 3.2.0 linux kernel (and not with the 2.6.32 kernel), but our experiment may be flawed (joyent#3241 (comment)).

@christophsturm Did that do the trick for you? We seem to be seeing really serious imbalances with Cluster and node 0.9.3 with the 3.2.0 linux kernel (and not with the 2.6.32 kernel), but our experiment may be flawed (joyent#3241 (comment)).

@brettkiefer

This comment has been minimized.

Show comment Hide comment
@brettkiefer

brettkiefer Dec 11, 2012

We tested the environment variables here and a build from master, just to make sure we weren't missing anything, and with the Linux kernel 3.2.0 we're still seeing one process hogging most of the connections. joyent#3241 (comment)

We tested the environment variables here and a build from master, just to make sure we weren't missing anything, and with the Linux kernel 3.2.0 we're still seeing one process hogging most of the connections. joyent#3241 (comment)

@brettkiefer

This comment has been minimized.

Show comment Hide comment
@brettkiefer

brettkiefer Dec 12, 2012

@christophsturm If it happens that you got this fixed, and you happen to be feeling charitable, could you run the test I posted here with your setup and tell me if it balances any better? joyent#3241 (comment)

@christophsturm If it happens that you got this fixed, and you happen to be feeling charitable, could you run the test I posted here with your setup and tell me if it balances any better? joyent#3241 (comment)

@jasnell

This comment has been minimized.

Show comment Hide comment
@jasnell

jasnell May 18, 2015

Member

... if this is still an issue, please reopen with additional information.

Member

jasnell commented May 18, 2015

... if this is still an issue, please reopen with additional information.

@jasnell jasnell closed this May 18, 2015

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.