…uch more sense now. - Broke Skynet::Message into 3 classes, TaskMessage, ResultMessage and ErrorMessage (which inherits from ResultMessage) - MessageQueue adapters now assume if you pass :any as the task type you want, they should randomly pick from the master or task queues. - Added new SingleProcess message queue adapter to be the reference implementation and mock what a real message queue would work like. - Add new QuiQ queue, though I'm not using it yet. - Rafactor TupleSpace message queue adapter to use new superclass - Begin trying to move what was in Skynet::Task into Skynet::TaskMessage - Skynet::TaskMessage now runs the task. - Create a universal adapter tester. Just plug in your adapter and run the tests.
Some ruby installs Queue class does not have enq and deq, so I switched it to push/pop Fixed calculation of untaken_future_tasks in mysql message queue adaptor
… goes down
…r. Move start_worker_queue_thread to new method
Last commit missed the following message: Skynet Workers now communicate with skynet manager by grabbing the manager over DRB and passing hashes directly to an internal Queue object held by the manager. The manager has a thread that watches the queue and updates the @worker_queue when new items are in the queue. The worker/manager no longer use tuplespace to communicate Removed unecessary logging in a few places.
…ch just finds the ts connected to the manager on that machine
…ving it in the individually installed launchers. Much cleaner. - Make sure skynet_launcher passes ALL config options when it starts the tuple_space_server. - Remove starting of tuplespace server from installed skynet launcer - Make sure we verify if we have a valid tuplespace even if we're not using a ringserver. - Allow you to pass use_ringserver as a command line flag to skynet_tuplespace_server
…or some reason
…uplespace server as a worker queue, which it kills when it dies. Workers communicate directly with that queue instead of the main message queue. Also, skynet shuts down more quickly. - SkynetMessageQueue now optionally takes a hash of options instead of a proxy object. - Renamed Skynet::Server as Skynet::TuplespaceServer - Config changes - renamed TS_DRBURIS to TS_DRBUSI which is now a string instead of an array - added TS_WQUEUE_URI - Skynet::Worker has a short circuit exit if its not processing a task at the moment. It notifies_worker_stop when it does. - Skynet::WorkerQueue now assumes it's using a separate queue than the MessageQueue. It passes the TS_WQUEUE_URI to the Worker Queue proxy class. - skynet runner that gets installed from skynet_install now passes some piddir and logdir flags to skynet_tuplespace_server start. - Skynet::Manager - the manager now forks off a tuplespace worker queue for its workers to communicate with it. It kills that process once the manager is killed. - Fix bug in mark_worker_as_sopped - Signal workers now takes :idle as a tasktype where it sends the signal to workers that aren't reported as busy. It also takes an array of tasktypes. - shutdown() passe an array of tasktypes including :idle to send_signal
…hods to SkynetDebugger
- Dramatically improve shutdown time. If a worker isn't doing anything it just dies right away regardless of where it is in the polling process - Skynet runner does not need 'boot.rb' in rails
…t wouldn't handle empty results well. - Fixed Skynet::Partitioners::ArrayDataSplitByFirstEntry to handle strings as keys better
…et/branches/ts_manager - Skynet manager now supports being daemonized by running ./script/skynet start and ./script/skynet stop! - Change mysql text fields to longtext in migration and schema files - Include some extras including our init.d script, our nagios monitoring script, rails controller and view for monitoring - Created a new skynet rails initializer which gets installed from skynet_install --rails - Modified the skynet_install skynet runner to take skynet initializer into account - Skynet::Manager - Work on the shutdown proceedure. Much more graceful now. - Changed some of the debug levels in Skynet::Worker and Skynet::Manager - Skynet::ActiveRecordExtensions Fixed a bug where it would fail if your table had fewer than 1000 rows. It performs a count first now to make sure there are enough rows. - Fix a comment in Skynet::MapreduceHelper. self.reduce should ahve been self.reduce_each - Fix some bugs in skynet_config where it would choke on Skynet::Config.logfile_location if the SKYNET_LOG_FILE was set to a filehandle like STDOUT. - Fix a skynet manager bug where it would get weird empty BigDecimals from the worker statuses - Added Skynet::Job.results_by_job_id to retrieve results from asyncronous jobs - Moved all worker stats into the manager class which now does stats no matter what worker_queue_adapter your using. The stats are more detailed now as well. - Removed all stats code from worker queue classes - Added a stats_for_hosts method to Skynet::Manager which aggregates stats accross many managers. - Move worker methods out of the mysql message queue adapter into a new mysql worker queue adapter - Added a Skynet::Manager#stats method - Skynet::Manager now saves its stats to a file and reloads them when it starts. - Skynet::Manager.get returns the current local manager drb object - Workers now just add their statuses to the worker queue (without trying to take their old one off) - Managers just keep polling the queue, taking worker statuses off, and holding an internal hash of worker statuses. - The Manager doesn't pass around the worker_pids or worker_queue anymore, just updates when needed and uses the cached one. - Changed how the skynet pid file and log file are specified. You can now specify the directory and file separately. - Lowered the WORKER_CHECK_DELAY - Made some API changes to Skynet::Config so you can call class or instance methods for all the CONFIG options. - Add new WORKER_QUEUE_ADAPTER config setting - Create Skynet::MessageQueue.start_or_connect - More work on Skynet::WorkerQueue class
… it puts the log and pid file in the right place.
… regardless of the log level. Updated docs