Issues - ezmobius/nanite

ezmobius / nanite

688

Description:

self assembling fabric of ruby daemons

Sort by: Priority | Votes | Last Updated

Loading…

3.

1 vote

Mapper unable to send requests from Merb (mongrel adapter)
2 comments Created 4 months ago by highandwild

Title

Body
I have started the mapper in config/init.rb but Nanite.request() simply returns false without any error message. The agent does not receive the request, either. But it works fine if I test it with nanite-mapper ! Here's the code: http://gist.github.com/147641 ruby - 1.8.7 mongrel - 1.1.5 nanite - 0.4.1 (edge) amqp - 0.6.0 rabbitmq - 1.5.5 Arch - x64 OS - Cent OS 5.1 (virtual box)
or cancel
I have started the mapper in config/init.rb but Nanite.request() simply returns false without any error message. The agent does not receive the request, either.

But it works fine if I test it with nanite-mapper !

Here's the code: http://gist.github.com/147641

ruby - 1.8.7
mongrel - 1.1.5
nanite - 0.4.1 (edge)
amqp - 0.6.0
rabbitmq - 1.5.5
Arch - x64
OS - Cent OS 5.1 (virtual box)

Comments
highandwild Thu Jul 16 01:46:32 -0700 2009 | link
Update: I have no idea how (perhaps because I downgraded to ruby 1.8.6) but it seems to finally queue the job. Now the problem is that it just queues it and it never reaches the agent ! (I'm running it inside a merb console).

Nanite.mapper.job_warden.inspect shows this:

#<Nanite::JobWarden:0x2aaaabef7790 @serializer=# <Nanite::Serializer:0x2aaaabef8208 @serializers=[Marshal, JSON, YAML]>, @jobs={"b16084315b1e87436c37493e656f6f60"=>#<Nanite::Job:0x2aaaae71b370 @results={}, @intermediate_handler=nil, @request=#<Nanite::Request:0x2aaaae71b668 @persistent=false, @reply_to="mapper-d201c7294713d6f8d4ef1daa1cf793a6", @type="/title/search_by_title", @target=nil, @token="b16084315b1e87436c37493e656f6f60", @tags=[], @selector=:least_loaded, @payload={:term=>"Harry", :id=>82523}, @from="mapper-d201c7294713d6f8d4ef1daa1cf793a6">, @intermediate_state={}, @token="b16084315b1e87436c37493e656f6f60", @targets=["nanite-67a7b8cf51fe5131cc3096a6b951331a"], @completed=nil, @pending_keys=[]>}>

And yes, I am getting a hearbeat from the agents. Also, Nanite.mapper.cluster.nanites lists the nanites !

What am I doing wrong ?
Comment
Update: I have no idea how (perhaps because I downgraded to ruby 1.8.6) but it seems to finally queue the job. Now the problem is that it just queues it and it never reaches the agent ! (I'm running it inside a merb console). Nanite.mapper.job_warden.inspect shows this: #<Nanite::JobWarden:0x2aaaabef7790 @serializer=# <Nanite::Serializer:0x2aaaabef8208 @serializers=[Marshal, JSON, YAML]>, @jobs={"b16084315b1e87436c37493e656f6f60"=>#<Nanite::Job:0x2aaaae71b370 @results={}, @intermediate_handler=nil, @request=#<Nanite::Request:0x2aaaae71b668 @persistent=false, @reply_to="mapper-d201c7294713d6f8d4ef1daa1cf793a6", @type="/title/search_by_title", @target=nil, @token="b16084315b1e87436c37493e656f6f60", @tags=[], @selector=:least_loaded, @payload={:term=>"Harry", :id=>82523}, @from="mapper-d201c7294713d6f8d4ef1daa1cf793a6">, @intermediate_state={}, @token="b16084315b1e87436c37493e656f6f60", @targets=["nanite-67a7b8cf51fe5131cc3096a6b951331a"], @completed=nil, @pending_keys=[]>}> And yes, I am getting a hearbeat from the agents. Also, Nanite.mapper.cluster.nanites lists the nanites ! What am I doing wrong ?
or cancel
highandwild Mon Jul 20 04:03:02 -0700 2009 | link

Tested it with sinatra (mongrel), and it works perfectly. Also upgraded to amqp edge. Still Zero luck with merb :( I also ran merb in debug mode but can't find anything amiss...

Comment
Tested it with sinatra (mongrel), and it works perfectly. Also upgraded to amqp edge. Still Zero luck with merb :( I also ran merb in debug mode but can't find anything amiss...
or cancel

Parsed with GitHub Flavored Markdown
6.

0 votes

load average / status function not updated with heartbeat messages when using Redis
1 comment Created 3 months ago by joshwilsdon

Title

Body
When using redis the nanite-<identity> key gets set with the load_average when the node registers but it is not updated after that by heartbeat messages. It seems that the intention is that the heartbeat/ping sends the load average so that the load average will be put in Redis. This is not happening because the code in cluster.rb (handle_ping) is doing: if nanite = nanites[ping.identity] nanite[:status] = ping.status but nanites[ping.identity] returns an anonymous Hash, so updating it here does nothing. As such this value is never sent to Redis. I have confirmed that hacking in a update_status function to the Nanite::State class (which just updates the nanite-<identity> key in redis) and then calling it in the handle_ping as: nanites.update_status(ping.identity, ping.status) causes the value in Redis to be updated at every heartbeat. Was it the intended behavior for this function (handle_ping) to update Redis? The comment seems to indicate that is the case.
or cancel
When using redis the nanite- key gets set with the load_average when the node registers but it is not updated after that by heartbeat messages. It seems that the intention is that the heartbeat/ping sends the load average so that the load average will be put in Redis. This is not happening because the code in cluster.rb (handle_ping) is doing:
```
if nanite = nanites[ping.identity]
    nanite[:status] = ping.status
```
but nanites[ping.identity] returns an anonymous Hash, so updating it here does nothing. As such this value is never sent to Redis. I have confirmed that hacking in a update_status function to the Nanite::State class (which just updates the nanite- key in redis) and then calling it in the handle_ping as:
```
nanites.update_status(ping.identity, ping.status)
```
causes the value in Redis to be updated at every heartbeat. Was it the intended behavior for this function (handle_ping) to update Redis? The comment seems to indicate that is the case.
Comments

mattmatt Wed Sep 09 07:47:32 -0700 2009 | link

I can confirm this issue, and I've taken your idea to update the Redis store accordingly.

Comment
I can confirm this issue, and I've taken your idea to update the Redis store accordingly.
or cancel

Parsed with GitHub Flavored Markdown
7.

0 votes

Client-agent requests gets multiplicated
10 comments Created 2 months ago by nofxx

Title

Body
Supposing cli-mapper that sends a push to a client-agent: push('/client_agent/foo', "hi") And this client-agent sends a push (or request) to a main-agent: push('/main_agent/foo', ["bar"] ) Main agent receives 2 requests instead of one. If I have for instance, 8 thins running, main agent receives 10 exaclty equal requests. Looks like the requests are getting multiplicated by each mapper online on that time.
or cancel
Supposing cli-mapper that sends a push to a client-agent:
```
push('/client_agent/foo', "hi")
```
And this client-agent sends a push (or request) to a main-agent:
```
push('/main_agent/foo', ["bar"] )
```
Main agent receives 2 requests instead of one. If I have for instance, 8 thins running, main agent receives 10 exaclty equal requests.
Looks like the requests are getting multiplicated by each mapper online on that time.
Comments

mattmatt Fri Sep 11 06:36:09 -0700 2009 | link

I'm guessing you're not using Redis, correct? If you don't all mappers will get the agent's request, and therefore all of them will forward it to an agent of their own. When using Redis only one mapper will get the request and forward it. It's probably a matter of discussion if this is a bug or a feature. I guess it wouldn't hurt to enable exclusiveness on the request queue when not using Redis as well, but there may be situations where not all mappers know the appropriate agents yet to handle that request. That again wouldn't hurt too much when using an offline queue, since that'd eventually lead to the right agent getting the message.

Comment
I'm guessing you're not using Redis, correct? If you don't all mappers will get the agent's request, and therefore all of them will forward it to an agent of their own. When using Redis only one mapper will get the request and forward it. It's probably a matter of discussion if this is a bug or a feature. I guess it wouldn't hurt to enable exclusiveness on the request queue when not using Redis as well, but there may be situations where not all mappers know the appropriate agents yet to handle that request. That again wouldn't hurt too much when using an offline queue, since that'd eventually lead to the right agent getting the message.
or cancel

nofxx Fri Sep 11 06:55:28 -0700 2009 | link

Hi Matt, thanks for the quick reply.
Yup, I'm not using redis. Hm, lots of options:
Use Redis, because this will really be a bug here this, heh...
Try to play with a tokyo adapter (already got it running on the server), redis is key/value store, right?
Or, if it's not too much to ask, could you give me some directions on how to disable this "feature" ? and rely on offline queue.

Comment
Hi Matt, thanks for the quick reply. Yup, I'm not using redis. Hm, lots of options: Use Redis, because this will really be a bug here this, heh... Try to play with a tokyo adapter (already got it running on the server), redis is key/value store, right? Or, if it's not too much to ask, could you give me some directions on how to disable this "feature" ? and rely on offline queue.
or cancel

mattmatt Fri Sep 11 07:00:39 -0700 2009 | link

Redis is key/value, that's correct. I don't have a solution for it per se. A quickfix would be to look into cluster.rb. In setup_request_queue it's using an exclusive queue when using Redis. You could try always using that, i.e. remove the shared_state? check. The offline queue is a simple command line switch (--offline-failsafe) for the nanite-agent and a parameter for the mapper class (:offline_failsafe => true). It's just an additional sanity check which you should do anyway when you're relying on your messages being delivered.

The worst case scenario should be avoidable when using the offline queue and only having one mapper pick up the message from the request queue. Let me know if that works. Maybe it's worth considering to get that fix in.

Comment
Redis is key/value, that's correct. I don't have a solution for it per se. A quickfix would be to look into cluster.rb. In setup_request_queue it's using an exclusive queue when using Redis. You could try always using that, i.e. remove the shared_state? check. The offline queue is a simple command line switch (--offline-failsafe) for the nanite-agent and a parameter for the mapper class (:offline_failsafe => true). It's just an additional sanity check which you should do anyway when you're relying on your messages being delivered. The worst case scenario should be avoidable when using the offline queue and only having one mapper pick up the message from the request queue. Let me know if that works. Maybe it's worth considering to get that fix in.
or cancel

nofxx Fri Sep 11 08:12:57 -0700 2009 | link

Ops, now I realize I was pvt messaging matt. Sorry man.

Same issue with redis enabled. =/

UPDATE: I've tryed editing setup_request_queue in all ways I could imagine, same result.
Btw, got 2 specs failiing too, something about the ProxyMapper instance was not being erased...

Comment
Ops, now I realize I was pvt messaging matt. Sorry man. Same issue with redis enabled. =/ UPDATE: I've tryed editing setup_request_queue in all ways I could imagine, same result. Btw, got 2 specs failiing too, something about the ProxyMapper instance was not being erased...
or cancel

mattmatt Fri Sep 11 08:26:43 -0700 2009 | link

That's a bit odd, even though we had to fix some issues with Redis and internal timeouts, that solved it for us, since then only one mapper gets the request and forwards it. I'd need some more log output from the mapper logs to get a better look at what's going on.

Comment
That's a bit odd, even though we had to fix some issues with Redis and internal timeouts, that solved it for us, since then only one mapper gets the request and forwards it. I'd need some more log output from the mapper logs to get a better look at what's going on.
or cancel

nofxx Fri Sep 11 08:43:22 -0700 2009 | link

Gosh, I'm embarassed now, There was a mapper running I didn't see (w/o redis).
It's working fine with Redis. Really sorry, need to sleep, nanite is givin me some insomnia... (and it feels great).
Thank you Matt, I owe you a (or some) beer. Just let me know when you came to Brazil.

Comment
Gosh, I'm embarassed now, There was a mapper running I didn't see (w/o redis). It's working fine with Redis. Really sorry, need to sleep, nanite is givin me some insomnia... (and it feels great). Thank you Matt, I owe you a (or some) beer. Just let me know when you came to Brazil.
or cancel

nofxx Fri Sep 11 19:48:48 -0700 2009 | link

Just to confirm, removing the -#{identity} and the exclusive option of the amq.queue request, it works. Only one request and without Redis.
I'll be happy to work in a patch to make this an "option", if anyone is interested, or no better solution came to light. In the while, will keep a fork to make it easy to install it on my servers.

Thanks matt, thanks all you nanite devs, you guys rock!

Comment
Just to confirm, removing the -#{identity} and the exclusive option of the amq.queue request, it works. Only one request and without Redis. I'll be happy to work in a patch to make this an "option", if anyone is interested, or no better solution came to light. In the while, will keep a fork to make it easy to install it on my servers. Thanks matt, thanks all you nanite devs, you guys rock!
or cancel

nofxx Fri Sep 11 21:27:06 -0700 2009 | link

I've added an option to the mapper init, well, it works, will try in production this weekend.
http://github.com/nofxx/nanite/commit/7804058cf297088f063cf5d1d2695c8b15ab71a0

Gonna write/fix the specs soon.. heh, sorry about the emacs whitespace cleanup too.

Comment
I've added an option to the mapper init, well, it works, will try in production this weekend. http://github.com/nofxx/nanite/commit/7804058cf297088f063cf5d1d2695c8b15ab71a0 Gonna write/fix the specs soon.. heh, sorry about the emacs whitespace cleanup too.
or cancel

nofxx Mon Sep 14 23:41:41 -0700 2009 | link

Wow, finally, looks like it's working now! ;)
It's only calling it once, offline_failsafe ensure that some agent will find the request. All good.

Was having a weird problem with some actors that use ActiveRecord, they just stop advertising their methods. The problem was I didn't knew about single-threaded... working fine now.

I've added those stones I've found on my path to the wiki. Again, thanks!
Good to be on nanite hehe..

Comment
Wow, finally, looks like it's working now! ;) It's only calling it once, offline_failsafe ensure that some agent will find the request. All good. Was having a weird problem with some actors that use ActiveRecord, they just stop advertising their methods. The problem was I didn't knew about single-threaded... working fine now. I've added those stones I've found on my path to the wiki. Again, thanks! Good to be on nanite hehe..
or cancel

nofxx Tue Sep 29 05:26:20 -0700 2009 | link

Just an update: Albeit working flawless for weeks, on the deploy something strange happens (sometimes), the rails mappers, if I'm not wrong:

heartbeat-19018d26cdb64d27e25c55d007e73ebb 8149
heartbeat-25941f2a7e262e05e05c2349f08ff468 8153
....

Heartbeats start to accumulate, until god restart RabbitMQ, than everything gets back to normal... heh weird.
But heartbeat is about to be gonne, right? heh

Comment
Just an update: Albeit working flawless for weeks, on the deploy something strange happens (sometimes), the rails mappers, if I'm not wrong: heartbeat-19018d26cdb64d27e25c55d007e73ebb 8149 heartbeat-25941f2a7e262e05e05c2349f08ff468 8153 .... Heartbeats start to accumulate, until god restart RabbitMQ, than everything gets back to normal... heh weird. But heartbeat is about to be gonne, right? heh
or cancel

Parsed with GitHub Flavored Markdown
9.

0 votes

What environment are you guys using?
1 comment Created 15 days ago by allenwei

Title

Body
I met some promblem when using nanite. I think it may caused by gem environment. Can you tell me the gem version information which you guys running. One problem I met is. After I start up agent and mapper using the simple-agent example which is in the source code, the agent side log stopped at : [Fri, 06 Nov 2009 17:07:01 +0800] INFO: SEND [register] d72993a0de1aed09f42dd291e5046d3d, services: /simple/echo, /simple/time, /simple/gems, /simple/yielding, /simple/delayed, tags: And the mapper side the log stopped at: [Fri, 06 Nov 2009 17:07:11 +0800] INFO: [setup] starting mapper after a while the mapper down, cause this error /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:811:in `connect_server': no connection (RuntimeError) from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:811:in `reconnect' from /opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.0/lib/amqp/client.rb:172:in `reconnect' from /opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.0/lib/amqp/client.rb:85:in `call' from /opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.0/lib/amqp/client.rb:85:in `unbind' from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:in `call' from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:in `run_deferred_callbacks' from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:in `times' from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:in `run_deferred_callbacks' from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:242:in `run_machine' from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:242:in `run' from simpleagent/cli.rb:21 My environment is EventMachine 0.12.8 and amqp 0.6.0, rabbitmq 1.7.1 I just think whether it is rabbitmq permission,I try to re-install rabbitmq and run the "rabbitconf.rb" , then I the problem still. here is the log when I run "rabbitconf.rb" Setting permissions for user "mapper" in vhost "/nanite" ... ...done. Setting permissions for user "nanite" in vhost "/nanite" ... ...done. Listing users ... guest mapper nanite ...done. Listing vhosts ... / /nanite ...done. Listing permissions in vhost "/nanite" ... mapper . . . nanite . . . ...done.
or cancel
I met some promblem when using nanite. I think it may caused by gem environment.
Can you tell me the gem version information which you guys running.

One problem I met is.
After I start up agent and mapper using the simple-agent example which is in the source code, the agent side log stopped at :
[Fri, 06 Nov 2009 17:07:01 +0800] INFO: SEND [register] d72993a0de1aed09f42dd291e5046d3d, services: /simple/echo, /simple/time, /simple/gems, /simple/yielding, /simple/delayed, tags:

And the mapper side the log stopped at:
[Fri, 06 Nov 2009 17:07:11 +0800] INFO: [setup] starting mapper

after a while the mapper down, cause this error
/opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:811:in `connect_server': no connection (RuntimeError)
```
from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:811:in `reconnect'
from /opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.0/lib/amqp/client.rb:172:in `reconnect'
from /opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.0/lib/amqp/client.rb:85:in `call'
from /opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.0/lib/amqp/client.rb:85:in `unbind'
from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:in `call'
from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:in `run_deferred_callbacks'
from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:in `times'
from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:995:in `run_deferred_callbacks'
from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:242:in `run_machine'
from /opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:242:in `run'
from simpleagent/cli.rb:21
```
My environment is EventMachine 0.12.8 and amqp 0.6.0, rabbitmq 1.7.1

I just think whether it is rabbitmq permission,I try to re-install rabbitmq and run the "rabbitconf.rb" , then I the problem still.
here is the log when I run "rabbitconf.rb"

Setting permissions for user "mapper" in vhost "/nanite" ...
...done. Setting permissions for user "nanite" in vhost "/nanite" ... ...done. Listing users ... guest
mapper
nanite
...done. Listing vhosts ... / /nanite ...done. Listing permissions in vhost "/nanite" ... mapper . . .
nanite . . .
...done.
Comments

mattmatt Thu Nov 05 23:56:18 -0800 2009 | link

Can you be a bit more specific? Do you mean the installed dependencies of Nanite? Do you get a specific error message at some point while trying it out?

If so, you need either EventMachine 0.12.8 and the amqp gem < 0.6.5 or EventMachine 0.12.10 and the amqp gem = 0.6.5.

Comment
Can you be a bit more specific? Do you mean the installed dependencies of Nanite? Do you get a specific error message at some point while trying it out? If so, you need either EventMachine 0.12.8 and the amqp gem < 0.6.5 or EventMachine 0.12.10 and the amqp gem = 0.6.5.
or cancel

Parsed with GitHub Flavored Markdown
14.

0 votes

Agent stalls with 100% cpu utilization
3 comments Created 4 days ago by kingcu

Title

Body
I have been struggling for several days with a problem in my agents. Randomly, they will stall and use 100% of the CPU. strace reveals the agents are just context switching and doing nothing: --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 40001616 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 0 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 0 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 1 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 1 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 0 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 101 --- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 0 I have tried everything: modified agents to use epoll rather than select, tried ruby enterprise edition and ruby1.9 (they remove the syscalls in strace, but agents still lock). I cannot discern a pattern or reason the agents lock specifically, meaning the job they lock on isn't consistent ASIDE from happening during a job that utilizes net/http to pull down some images and stitch them together. I thought it might be an issue with calling sleep() inside the agents, but that didn't solve anything. I really have no idea where to go from here. Pastie to my agent code: http://pastie.org/702881 Pastie to image fetch/stitch code: http://pastie.org/702895 On the plus side, I'll be able to give you a quick modification to nanite that causes it to use epoll, which dropped my CPU utilization a hair while performing a large amount of jobs! Any ideas on where to start even looking from here would be appreciated, otherwise I am going to just start commenting out code until something changes (the worst way to debug!).
or cancel

I have been struggling for several days with a problem in my agents. Randomly, they will stall and use 100% of the CPU. strace reveals the agents are just context switching and doing nothing:

--- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 40001616
--- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 0
--- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 0
--- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 1
--- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 1
--- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 0
--- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 101
--- SIGVTALRM (Virtual timer expired) @ 0 (0) --- rt_sigreturn(0) = 0

I have tried everything: modified agents to use epoll rather than select, tried ruby enterprise edition and ruby1.9 (they remove the syscalls in strace, but agents still lock). I cannot discern a pattern or reason the agents lock specifically, meaning the job they lock on isn't consistent ASIDE from happening during a job that utilizes net/http to pull down some images and stitch them together.

I thought it might be an issue with calling sleep() inside the agents, but that didn't solve anything. I really have no idea where to go from here.

Pastie to my agent code: http://pastie.org/702881
Pastie to image fetch/stitch code: http://pastie.org/702895

On the plus side, I'll be able to give you a quick modification to nanite that causes it to use epoll, which dropped my CPU utilization a hair while performing a large amount of jobs! Any ideas on where to start even looking from here would be appreciated, otherwise I am going to just start commenting out code until something changes (the worst way to debug!).

Comments

mattmatt Wed Nov 18 05:07:43 -0800 2009 | link

I wish I had any idea where to start. What's the number of messages your seeing? Please also run the agents with debug log mode so you can at least see what the last of their activities is. I'd like to get Nanite a lot more bullet-proof in that regard.

Also, what EventMachine and AMQP version are you using?

Comment
I wish I had any idea where to start. What's the number of messages your seeing? Please also run the agents with debug log mode so you can at least see what the last of their activities is. I'd like to get Nanite a lot more bullet-proof in that regard. Also, what EventMachine and AMQP version are you using?
or cancel

kingcu Wed Nov 18 10:38:10 -0800 2009 | link

Whoops, forgot to mention the particulars:

AMQP 0.6.5
EM 0.12.10 (same happens with 0.12.8)

Happens when I push through a group of 200 or so jobs, with prefetch set at 1, so only one job is on the agent at a time. It doesn't seem to stall on any particular piece of code that I can discern. Additionally, strace shows absolutely 0 activity outside of the SIGVTALRM syscalls, even though the CPU is pegged, which is beyond me. From my understanding, this means all threads have finished. This is why I think there is something odd going on with Nanite/AMQP/EM, because it's inconsistent and they think work is done before it is done.

I have been reading up on ltrace and more detailed strace use, so I'll have a bit more something to go off here soon I think.

Comment
Whoops, forgot to mention the particulars: AMQP 0.6.5 EM 0.12.10 (same happens with 0.12.8) Happens when I push through a group of 200 or so jobs, with prefetch set at 1, so only one job is on the agent at a time. It doesn't seem to stall on any particular piece of code that I can discern. Additionally, strace shows absolutely 0 activity outside of the SIGVTALRM syscalls, even though the CPU is pegged, which is beyond me. From my understanding, this means all threads have finished. This is why I think there is something odd going on with Nanite/AMQP/EM, because it's inconsistent and they think work is done before it is done. I have been reading up on ltrace and more detailed strace use, so I'll have a bit more something to go off here soon I think.
or cancel

mattmatt Sat Nov 21 04:13:54 -0800 2009 | link

I'll try to set up something to fire similar jobs. Maybe I can reproduce it.

Comment
I'll try to set up something to fire similar jobs. Maybe I can reproduce it.
or cancel

Parsed with GitHub Flavored Markdown
15.

0 votes

How to initiate a sync request?
1 comment Created 3 days ago by leomayleomay

Title

Body
I'm working on a test for our Nanite agent, but the request initiated by Nanite.request are all async, which means, i can get nothing back from my agent before the test finishes. Can I initiate a sync request? Thank you
or cancel

I'm working on a test for our Nanite agent, but the request initiated by Nanite.request are all async, which means, i can get nothing back from my agent before the test finishes.

Can I initiate a sync request? Thank you

Comments

mattmatt Wed Nov 18 03:19:53 -0800 2009 | link

There is no synchronous request in Nanite due to the asynchronous nature of the broker in between. Your best bet might be to try moqueue which patches the amqp gem to use a basic in-memory queue instead of a full-blown broker like RabbitMQ.

Comment
There is no synchronous request in Nanite due to the asynchronous nature of the broker in between. Your best bet might be to try [moqueue](http://github.com/danielsdeleo/moqueue) which patches the amqp gem to use a basic in-memory queue instead of a full-blown broker like RabbitMQ.
or cancel

Parsed with GitHub Flavored Markdown
16.

0 votes

Gracefully stopping agents & Timeouts
3 comments Created 3 days ago by taazza

Title

Body
Two questions 1. When a nanite agent timeouts - Is there a way to detect it with code? Also sometimes the agents come back alive after a long running task. We want to strike a balance between number of agents vs timeouts. Any suggestions/ideas? 2. Also is there an inbuilt way to gracefully stop agents? Something like nanite-agent --token "test" stop --force-after 60 seconds I did read an email from mattmatt a while ago. Thoughts?
or cancel
Two questions
1. When a nanite agent timeouts - Is there a way to detect it with code? Also sometimes the agents come back alive after a long running task. We want to strike a balance between number of agents vs timeouts. Any suggestions/ideas?
2. Also is there an inbuilt way to gracefully stop agents? Something like nanite-agent --token "test" stop --force-after 60 seconds
I did read an email from mattmatt a while ago. Thoughts?
Comments
mattmatt Wed Nov 18 05:05:02 -0800 2009 | link
There's callbacks available for timeout, register and unregister in the mapper, the RDoc for Mapper#start has all the info available.

Planned feature for me, since we need something similar. I'd like to get some Unix-y stuff like Unicorn has into Nanite so that you can gracefully shutdown and bring up new agents in the meantime.
Comment
1. There's callbacks available for timeout, register and unregister in the mapper, the RDoc for Mapper#start has all the info available. 2. Planned feature for me, since we need something similar. I'd like to get some Unix-y stuff like Unicorn has into Nanite so that you can gracefully shutdown and bring up new agents in the meantime.
or cancel
taazza Wed Nov 18 05:46:04 -0800 2009 | link

Regarding the first issue is this functionality available outside of the mapper? That would be ideal. How does the nanite-admin get a hold of it?

Just curious how do you deal with creating new agents to scale? Time-outs impacts that decision for us big time.

Comment
Regarding the first issue is this functionality available outside of the mapper? That would be ideal. How does the nanite-admin get a hold of it? Just curious how do you deal with creating new agents to scale? Time-outs impacts that decision for us big time.
or cancel

mattmatt Wed Nov 18 06:06:35 -0800 2009 | link

That functionality is only available in the mapper. If you need to react on it with some other means I'd say the best way is to fire off a new request to a management agent.

Comment
That functionality is only available in the mapper. If you need to react on it with some other means I'd say the best way is to fire off a new request to a management agent.
or cancel

Parsed with GitHub Flavored Markdown
17.

0 votes

nanite does not startup in ruby1.9
3 comments Created 3 days ago by pmamediagroup

Title

Body
nanite wont startup in ruby1.9 without some code changes. nanite uses FileUtils which does not seem to be available by default in 1.9. I have sent a pull request with a fix to the issue but never got a response (probably with good reason). in any case, with the change nanite runs flawlessly on 1.9. we have it running on several servers without any issues once that change is made.
or cancel

nanite wont startup in ruby1.9 without some code changes. nanite uses FileUtils which does not seem to be available by default in 1.9. I have sent a pull request with a fix to the issue but never got a response (probably with good reason). in any case, with the change nanite runs flawlessly on 1.9. we have it running on several servers without any issues once that change is made.

Comments

mattmatt Wed Nov 18 08:42:25 -0800 2009 | link

I'll check it out. I thought I had caught most of the Ruby 1.9 issues, but I'll look into your fork.

Comment
I'll check it out. I thought I had caught most of the Ruby 1.9 issues, but I'll look into your fork.
or cancel

kingcu Wed Nov 18 10:39:56 -0800 2009 | link

mattmatt: i fixed this on mine, it's 1.9 getting confused on where FileUtils is. It thinks it's a Nanite class. Putting in a "require 'fileutils'" inside nanite.rb solved the issue for me. There is probably a better place for the require, but I didn't feel like digging through and finding all recurrences of FileUtils inside Nanite, and confining the require to those.

Comment
mattmatt: i fixed this on mine, it's 1.9 getting confused on where FileUtils is. It thinks it's a Nanite class. Putting in a "require 'fileutils'" inside nanite.rb solved the issue for me. There is probably a better place for the require, but I didn't feel like digging through and finding all recurrences of FileUtils inside Nanite, and confining the require to those.
or cancel

pmamediagroup Wed Nov 18 12:59:32 -0800 2009 | link

i think its global to 1.9? irb doesn't recognize fileutils either until it's required.. at least on our system. we put ours in nanite.rb too

Comment
i think its global to 1.9? irb doesn't recognize fileutils either until it's required.. at least on our system. we put ours in nanite.rb too
or cancel

Parsed with GitHub Flavored Markdown

Labels

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments