ezmobius / nanite
- Source
- Commits
- Network (64)
- Issues (8)
- Downloads (0)
- Wiki (8)
- Graphs
-
Branch:
master
-
0 votes
comparison of Array with Array failed (ArgumentError)
13 comments Created about 1 month ago by taazzaNot sure if this is an amqp error or a nanite error, I have posted it on amqp as well.
/vendor/gems/gems/amqp-0.6.5/lib/amqp/buffer.rb:252:in `min': comparison of Array with Array failed (ArgumentError)
from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/cluster.rb:137:in `each' from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/cluster.rb:137:in `min' from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/cluster.rb:137:in `least_loaded' from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/cluster.rb:23:in `__send__' from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/cluster.rb:23:in `targets_for' from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/mapper.rb:198:in `send_request' from /home/test/v_0.1/vendor/gems/gems/nanite-0.4.1.10/lib/nanite/mapper.rb:191:in `request' from base_prog.rb:58:in `start' from /home/test/v_0.1/vendor/gems/gems/eventmachine-0.12.10/lib/em/timers.rb:51:in `call' from /home/test/v_0.1/vendor/gems/gems/eventmachine-0.12.10/lib/em/timers.rb:51:in `fire' from /home/test/v_0.1/vendor/gems/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `call' from /home/test/v_0.1/vendor/gems/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine' from /home/test/v_0.1/vendor/gems/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run' from base_prog.rb:41:in `start' from base_prog.rb:70This happens a lot. And when it happens it continues to happen repeatedly every couple of minutes till a restart is done. Wondering if this has to do with rabbitmq/amqp or the state of the nanite.
Any thoughts would be greatly appreciated.
Thanks!Comments
-
Nanite mapper seems to log all requests to agents as INFO:
[Sun, 15 Nov 2009 01:34:35 -0500] INFO: SEND [result] to TEST_I [Sun, 15 Nov 2009 01:34:35 -0500] INFO: RECV [result] from TEST_II
Is there any way to control this? The log files grow like crazy when every request is logged. Not sure if this is a logging issue or the way we are using it.
Comments
It is indeed a logging issue. There was a merge that changed the logging in a way that everything would go super-noisy even in log level info. I might change that to only use debug for dumping every request, since it's bothering me as well. It is pretty useful to follow the trail of requests, but I agree that it's not great.
Agreed, it would make sense to move it to DEBUG. Any chance of a quick fix? We are looking to move to prod in the next week. Thanks ;)
-
2 votes
prefetch option only available for mapper, not for agent
4 comments Created about 1 month ago by kingcuThe prefetch option that is a savior when you have high volumes of intense tasks is only available on mappers. The mapper doesn't really benefit from having a prefetch limit, as the mapper receives mostly pings and registrations, which can be processed really fast. It is the agent that is going to predominantly be affected, and should also have the option.
I went ahead and forked nanite, made the changes and am using it in production. This finally fixed my issue with overloading my agents to the point they were unresponsive and had to be killed. The fork is available on my github page if you are interested in bringing the change upstream.
In the fork, I simply added a configuration option for the agent, including adding the command line options for the nanite-agent script. I also documented the new config option in the init method, so aside from some review, it should be ready to go.
Comments
Cool, I'll merge it later today.
Any reason why you're checking if the prefetch method is there on the AMQP connection?
that was actually brought over from the mapper.rb method. I left it as I am assuming some versions of the AMQP gem don't have that method?
-
0 votes
eventmachine not initialized: evma_send_data_to_connection (RuntimeError)
10 comments Created 2 months ago by leomayleomaySystem configurations:
ruby 1.8.7 (2008-08-11 patchlevel 72) [universal-darwin10.0]
amqp (0.6.0)
eventmachine (0.12.10)
rabbitmq 1.7.0
erlang 5.6.5I follow the steps given by the "Test Nanite (finally)",
First, I created a new agent called bob, http://gist.github.com/223056
Second, I created a new mapper in another shell, http://gist.github.com/223058
Comments
Could you please install EventMachine 0.12.8 and check if that works? I'd be curious if the problem is the just recently released 0.12.10 which we haven't tried with Nanite and the AMQP library yet.
leomayleomay
Sat Oct 31 06:22:47 -0700 2009
| link
thank you, it works
Pretty odd that it'd break with a patch release of EventMachine. Thanks for testing, I'll look into it.
The solution is to use the latest AMQP (version 0.6.5) gem, it's up on gemcutter and removes a custom patch for EventMachine. I'll update the documentation accordingly.
leomayleomay
Mon Nov 02 16:45:38 -0800 2009
| link
great info, thank you, mattmatt
leomayleomay
Mon Nov 23 23:00:12 -0800 2009
| link
mattmatt, the problem is still there, environment:
amqp 0.6.5
eventmachine 0.12.10
nanite 0.4.1.13
leomayleomay
Mon Nov 23 23:07:42 -0800 2009
| link
mattmatt, since I switch my mod_rails to conservative spawning mode, it works quite well right now, but you know, I still want it work under smart mode if it's possible
Interesting, though in any way I usually recommend running mappers as separate processes anyway. Reduces that kind of pain quite considerably.
leomayleomay
Tue Nov 24 19:53:28 -0800 2009
| link
mattmatt, for now, I've extracted Nanite related code out of Rails stack, and running it on a DRB server (basically, a separate process), it's working now. Thanks for the tips.
-
0 votes
Bug in mapper.rb where options are passed as wrong parameter (size)
2 comments Created 4 months ago by joshwilsdonIn mapper.rb the request function looks like:
request(type, payload = '', opts = {}, &blk)and then passes the first 3 arguments in the same order to:
build_deliverable(deliverable_type, type, payload, opts)with deliverable_type == Request. Then build_deliverable passes these arguments directly to:
deliverable_type.new(type, payload, opts)but the problem is that Request has an initialize function that looks like:
initialize(type, payload, size=nil, opts={})so the options that were passed in to the original request get passed in as the 'size' parameter, which obviously doesn't work. This causes any :selector or :target to be ignored for example.
In our environment, this was causing problems when we passed the :target option because it was being ignored. Changing the build_deliverable function to make the call:
deliverable_type.new(type, payload, nil, opts)fixed the issue.
Comments
-
Getting a weird SSL error with ruby 1.9r129 on osx - any help / ideas would be appreciated!
$ rake spec /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:31: [BUG] Bus Error ruby 1.9.1p129 (2009-05-12 revision 23412) [i386-darwin9.7.0] -- control frame ---------- c:0029 p:---- s:0080 b:0080 l:000079 d:000079 CFUNC :initialize c:0028 p:---- s:0078 b:0078 l:000077 d:000077 CFUNC :new c:0027 p:0063 s:0075 b:0075 l:000074 d:000074 CLASS /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:31 c:0026 p:0011 s:0073 b:0073 l:000072 d:000072 CLASS /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:23 c:0025 p:0011 s:0071 b:0071 l:000070 d:000070 CLASS /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:22 c:0024 p:0045 s:0069 b:0069 l:000068 d:000068 TOP /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:21 c:0023 p:---- s:0067 b:0067 l:000066 d:000066 FINISH c:0022 p:---- s:0065 b:0065 l:000064 d:000064 CFUNC :require c:0021 p:0059 s:0061 b:0061 l:000060 d:000060 TOP /usr/local/lib/ruby19/site_ruby/1.9.1/openssl.rb:22 c:0020 p:---- s:0059 b:0059 l:000058 d:000058 FINISH c:0019 p:---- s:0057 b:0057 l:000056 d:000056 CFUNC :require c:0018 p:0083 s:0053 b:0053 l:000052 d:000052 TOP /Users/ippy04/Code/Samples/nanite/lib/nanite.rb:7 c:0017 p:---- s:0051 b:0051 l:000050 d:000050 FINISH c:0016 p:---- s:0049 b:0049 l:000048 d:000048 CFUNC :require c:0015 p:0084 s:0045 b:0045 l:000044 d:000044 TOP /Users/ippy04/Code/Samples/nanite/spec/spec_helper.rb:6 c:0014 p:---- s:0043 b:0043 l:000042 d:000042 FINISH c:0013 p:---- s:0041 b:0041 l:000040 d:000040 CFUNC :require c:0012 p:0039 s:0037 b:0037 l:000036 d:000036 TOP /Users/ippy04/Code/Samples/nanite/spec/actor_registry_spec.rb:1 c:0011 p:---- s:0035 b:0035 l:000034 d:000034 FINISH c:0010 p:---- s:0033 b:0033 l:000032 d:000032 CFUNC :load c:0009 p:0012 s:0029 b:0029 l:000020 d:000028 BLOCK /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:15 c:0008 p:---- s:0026 b:0026 l:000025 d:000025 FINISH c:0007 p:---- s:0024 b:0024 l:000023 d:000023 CFUNC :each c:0006 p:0036 s:0021 b:0021 l:000020 d:000020 METHOD /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:14 c:0005 p:0097 s:0017 b:0017 l:000016 d:000016 METHOD /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/options.rb:107 c:0004 p:0068 s:0012 b:0012 l:000011 d:000011 METHOD /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/command_line.rb:9 c:0003 p:0077 s:0007 b:0006 l:002614 d:002494 EVAL /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/bin/spec:4 c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH c:0001 p:0000 s:0002 b:0002 l:002614 d:002614 TOP :43044 --------------------------- -- Ruby level backtrace information----------------------------------------- /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:31:in `initialize' /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:31:in `new' /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:31:in `' /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:23:in `' /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:22:in `' /usr/local/lib/ruby19/site_ruby/1.9.1/openssl/ssl.rb:21:in `' /usr/local/lib/ruby19/site_ruby/1.9.1/openssl.rb:22:in `require' /usr/local/lib/ruby19/site_ruby/1.9.1/openssl.rb:22:in `' /Users/ippy04/Code/Samples/nanite/lib/nanite.rb:7:in `require' /Users/ippy04/Code/Samples/nanite/lib/nanite.rb:7:in `' /Users/ippy04/Code/Samples/nanite/spec/spec_helper.rb:6:in `require' /Users/ippy04/Code/Samples/nanite/spec/spec_helper.rb:6:in `' /Users/ippy04/Code/Samples/nanite/spec/actor_registry_spec.rb:1:in `require' /Users/ippy04/Code/Samples/nanite/spec/actor_registry_spec.rb:1:in `' /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:15:in `load' /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:15:in `block in load_files' /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:14:in `each' /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/example_group_runner.rb:14:in `load_files' /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/options.rb:107:in `run_examples' /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib/spec/runner/command_line.rb:9:in `run' /usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/bin/spec:4:in `' -- C level backtrace information ------------------------------------------- 0x117042 0 ruby19 0x00117042 rb_vm_bugreport + 82 0x2c21c 1 ruby19 0x0002c21c rb_warning + 444 0x2c27b 2 ruby19 0x0002c27b rb_bug + 43 0xbd37b 3 ruby19 0x000bd37b rb_enable_interrupt + 75 0x91e0b2bb 4 libSystem.B.dylib 0x91e0b2bb _sigtramp + 43 0xffffffff 5 ??? 0xffffffff 0x0 + 4294967295 [NOTE] You may encounter a bug of Ruby interpreter. Bug reports are welcome. For details: http://www.ruby-lang.org/bugreport.html rake aborted! Command /usr/local/bin/ruby19 -I"/usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/lib" "/usr/local/lib/ruby19/gems/1.9.1/gems/rspec-1.2.7/bin/spec" "spec/actor_registry_spec.rb" "spec/actor_spec.rb" "spec/agent_spec.rb" "spec/cached_certificate_store_proxy_spec.rb" "spec/certificate_cache_spec.rb" "spec/certificate_spec.rb" "spec/cluster_spec.rb" "spec/dispatcher_spec.rb" "spec/distinguished_name_spec.rb" "spec/encrypted_document_spec.rb" "spec/job_spec.rb" "spec/local_state_spec.rb" "spec/log_spec.rb" "spec/mapper_spec.rb" "spec/packet_spec.rb" "spec/rsa_key_pair_spec.rb" "spec/secure_serializer_spec.rb" "spec/serializer_spec.rb" "spec/signature_spec.rb" "spec/static_certificate_store_spec.rb" "spec/util_spec.rb" --format specdoc --colour failed
Comments
pmamediagroup
Tue Jul 28 16:02:23 -0700 2009
| link
same problem with 1.8.7
Is this still an issue? If not I'd like to close it, since it doesn't seem to be related to Nanite?
pmamediagroup
Wed Nov 18 08:27:29 -0800 2009
| link
no, its an issue with mac
-
3 votes
Problem with example section "Test Nanite (finally)"
2 comments Created 8 months ago by masonleeLooks like the file "nanite/examples/cli.rb" got moved to "nanite/examples/simpleagent/cli.rb". This causes two problems:
First, it breaks a path in cli.rb:
"require File.dirname(FILE) + '/../lib/nanite'" should become "require File.dirname(FILE) + '/../../lib/nanite'"
Second, the tutorial code on the main webpage in the section "Test Nanite (finally)" needs to change from "cd examples; ./cli.rb;" to "cd examples/simpleagent; ./cli.rb"
Comments
kennethkalmer
Tue Apr 28 14:21:51 -0700 2009
| link
Updated in my branch and applied by ezmobius. Could be closed now
-
5 votes
Example setup is out of date for recent RabbitMQ versions
3 comments Created 8 months ago by prbWith RabbitMQ 1.5.4, some of the commands (permissioning) in the
examples/rabbitconf.rbfail. These failures don't affect the functioning of the examples, but they don't exactly inspire user confidence, either...There are also some places in the examples where hardcoded paths are still in place, e.g.,
examples/crew.rb.Comments
summary of similarities and differnces
sudo rabbitmqctl add_vhost /nanite
sudo rabbitmqctl add_user mapper testing
sudo rabbitmqctl add_user nanite testingVersion 1.5.x
sudo rabbitmqctl map_user_vhost mapper /nanite
sudo rabbitmqctl map_user_vhost nanite /naniteVersion 1.6.x
sudo rabbitmqctl set_permissions -p /nanite mapper '.*' '.*' '.*'
sudo rabbitmqctl set_permissions -p /nanite nanite '.*' '.*' '.*'Yeah this is unfortunate that rabbit has changed between versions. The scripts right now will run both of these settings which will look like it failed for some commands but will in the end leave you with a working rabbitmq server setup as it will run the proper command for whichever version of rabbit you have.
I'm going to close this ticket with a caveat that you can safely ignore the errors and still end up with a working system.
When did you update/install your Nanite gem? The current version on gemcutter.org is 0.4.12, and I've never seen that happen.
On a second note, I'll see how I go with the AMQP 0.6.5 gem today, but still, I'd encourage you to update your Nanite installation.
We use gem bundler and the current version of nanite on gemcutter is 4.1.10 http://gemcutter.org/gems/nanite Where are seeing 0.4.12? Am I missing something here?
The 0.4.1.2 version is right there in the list. Version 0.4.1.10 is not the official Nanite gem. I'm afraid it's the RightScale fork and it's full of custom patches for the RightScale product and not properly tested from my point of view. Please install 0.4.1.2, and I'll talk to Ezra how that version ended up on Gemcutter.
Aah... 0.4.1.2! I was looking for 0.4.1 [12] as you had mentioned earlier.
When someone installs nanite 0.4.1.[10] gets selected by default. No worries I will give this a shot and hopefully the problem disappears!
I'll try to push an updated gem later today.
Thanks! Pls try and get the logging issue in as well ;) You help and prompt responses have been very helpful! Thanks a bunch! Pls close both issues once you are done with the build & push.
I assuming the updated Gem will be posted on gemcutter. Thanks again!
The gem on gemcutter has been updated. Let me know if there are any problems.
No such luck. Tested it out with nanite-0.4.1.13 and after running for a few hours it runs into the same problem. Exception attached below
/home/test/v_0.1/vendor/gems/gems/amqp-0.6.5/lib/amqp/buffer.rb:252:in `min': comparison of Array with Array failed (ArgumentError)
Had to reboot the machine.
As for logging .. The mapper is all set, INFO issue has disappeared. But the agent still logs the request as INFO
[Sat, 21 Nov 2009 03:35:44 -0500] INFO: SEND [result] [Sat, 21 Nov 2009 03:35:44 -0500] INFO: RECV [result]
leading big log files. Pls reopen this issue. Thx
Are you using Redis as state storage?
Somehow the status of an agent comes out as an array from the state storage. It would help me to find out what's going on if you could patch the cluster.rb at line 132 to output a[1] and b[1]. Otherwise it'd get hard for me to debug. I'll have a hard look at the data coming into the state store, but it'd be easier to figure out.
I'll look into the agent logging as well, I thought I got them all.
Nope, not using Redis. Let me patch and rebuild the gem and test it out.
I will send you the logs soon. I dont understand why I have to restart the machine for the problem to disappear. Anyways, thanks for taking a look at the issue, we are out of bandwidth to contribute at the moment.
We will pitch in soon. Thanks for all your effort/help. Cheers!
I printed the candidates variable
When you start the mapper and every thing is fine Here is what gets printed.
INFO: [ARGUMENT_ERROR_PATCH] candidates ->
nanite-SMEBARUTHI
timestamp1258956831
tags
status0.0
services/masala/process/thadka/process/lao/process/test/execute/vayudooth/process/khale/process/thadayam/process
nanite-ROJA
timestamp1258956827
tags
status0.0
services/masala/process/thadka/process/lao/process/test/execute/vayudooth/process/khale
/process/thadayam/process
And when things go wrong and array compare failed error pops up this is what gets printed
INFO: [ARGUMENT_ERROR_PATCH] candidates ->
nanite-SMEBARUTHI
timestamp1259006907
tags
statusno status [THIS SEEMS TO BE THE ISSUE - no value instead [no status] gets printed]
services/masala/process/thadka/process/lao/process/test/execute/vayudooth/process/khale/process/thadayam/process
nanite-ROJA
timestamp1259006915
tags
status0.46
services/masala/process/thadka/process/lao/process/test/execute/vayudooth/process/khale/process/thadayam/process
Hope this helps.
Thanks, that does help. I'll look into it.