Permalink
Commits on May 28, 2008
  1. - Rewriting message_queue_adapter interface (and superclass). Makes m…

    …uch more sense now.
    
    - Broke Skynet::Message into 3 classes, TaskMessage, ResultMessage and ErrorMessage (which inherits from ResultMessage)
    
    - MessageQueue adapters now assume if you pass :any as the task type you want, they should randomly pick from the master or task queues.
    
    - Added new SingleProcess message queue adapter to be the reference implementation and mock what a real message queue would work like.
    
    - Add new QuiQ queue, though I'm not using it yet.
    
    - Rafactor TupleSpace message queue adapter to use new superclass
    
    - Begin trying to move what was in Skynet::Task into Skynet::TaskMessage
    - Skynet::TaskMessage now runs the task.
    
    - Create a universal adapter tester. Just plug in your adapter and run the tests.
    phlapjack committed May 28, 2008
Commits on May 25, 2008
  1. Create a new type of Q class called qui_q

    Create a new message q adapter for the qui_q MQ.   This will be the reference implementation for all q's
    phlapjack committed May 25, 2008
  2. Trying some different message queue architectures

    phlapjack committed May 25, 2008
Commits on May 23, 2008
  1. update history

    phlapjack committed May 23, 2008
  2. Make sure we don't die if we can't connect to a manager for stats

    Some ruby installs Queue class does not have enq and deq, so I switched it to push/pop
    Fixed calculation of untaken_future_tasks in mysql message queue adaptor
    phlapjack committed May 23, 2008
  3. Fix to future tasks in stats

    phlapjack committed May 23, 2008
  4. Add future tasks to stats

    phlapjack committed May 23, 2008
Commits on May 22, 2008
  1. Make sure manager drb object accepts connections from any host (not j…

    …ust localhost)
    phlapjack committed May 22, 2008
  2. Add rescue blocks to worker/manager communication in case the manager…

    … goes down
    phlapjack committed May 22, 2008
  3. Make sure to save the worker_queue_file every minute in Skynet Manage…

    …r. Move start_worker_queue_thread to new method
    phlapjack committed May 22, 2008
  4. Remove worker queue adapters and skynet_worker_queue

    Last commit missed the following message:
    
    Skynet Workers now communicate with skynet manager by grabbing the manager over DRB and passing hashes directly to an internal
    Queue object held by the manager.
    The manager has a thread that watches the queue and updates the @worker_queue when new items are in the queue.
    The worker/manager no longer use tuplespace to communicate
    
    Removed unecessary logging in a few places.
    phlapjack committed May 22, 2008
  5. give WorkerQueueAdapter::TupleSpace its own get_tuplespace method whi…

    …ch just finds the ts connected to the manager on that machine
    phlapjack committed May 22, 2008
Commits on May 20, 2008
  1. - Add Launching of TupleSpace server to skynet_launcher instead of ha…

    …ving it in the individually installed launchers. Much cleaner.
    
    - Make sure skynet_launcher passes ALL config options when it starts the tuple_space_server.
    - Remove starting of tuplespace server from installed skynet launcer
    - Make sure we verify if we have a valid tuplespace even if we're not using a ringserver.
    - Allow you to pass use_ringserver as a command line flag to skynet_tuplespace_server
    phlapjack committed May 20, 2008
  2. Fix manifest

    phlapjack committed May 20, 2008
  3. remove docs from svn

    phlapjack committed May 20, 2008
  4. add docs to svn

    phlapjack committed May 20, 2008
  5. include information about examples in the readme

    phlapjack committed May 20, 2008
Commits on May 18, 2008
  1. Clean up manifest and skynet directory

    phlapjack committed May 18, 2008
Commits on May 17, 2008
  1. Have Skynet::Manager handle what happens when the worker queue dies f…

    …or some reason
    phlapjack committed May 17, 2008
  2. Clean up debugging

    Try and handle double kills better
    phlapjack committed May 17, 2008
  3. The major change is that now every skynet manager forks off its own t…

    …uplespace server as a worker queue, which it kills when it dies. Workers communicate directly with that queue instead of the main message queue.
    
    Also, skynet shuts down more quickly.
    
    - SkynetMessageQueue now optionally takes a hash of options instead of a proxy object.
    - Renamed Skynet::Server as Skynet::TuplespaceServer
    - Config changes
      - renamed TS_DRBURIS to TS_DRBUSI which is now a string instead of an array
      - added TS_WQUEUE_URI
    - Skynet::Worker has a short circuit exit if its not processing a task at the moment.  It notifies_worker_stop when it does.
    - Skynet::WorkerQueue now assumes it's using a separate queue than the MessageQueue.  It passes the TS_WQUEUE_URI to the Worker Queue proxy class.
     - skynet runner that gets installed from skynet_install now passes some piddir and logdir flags to skynet_tuplespace_server start.
    - Skynet::Manager
      - the manager now forks off a tuplespace worker queue for its workers to communicate with it.   It kills that process once the manager is killed.
      - Fix bug in mark_worker_as_sopped
      - Signal workers now takes :idle as a tasktype where it sends the signal to workers that aren't reported as busy.  It also takes an array of tasktypes.
      - shutdown() passe an array of tasktypes including :idle to send_signal
    phlapjack committed May 17, 2008
Commits on May 9, 2008
  1. Added some debugging to Skynet::Manager. Added some new debugging met…

    …hods to SkynetDebugger
    phlapjack committed May 9, 2008
Commits on May 4, 2008
  1. - Gracefully handles trying to start skynet more than once

    - Dramatically improve shutdown time.   If a worker isn't doing anything it just dies right away regardless of where it is in the
      polling process
    - Skynet runner does not need 'boot.rb' in rails
    phlapjack committed May 4, 2008
Commits on May 3, 2008
  1. - Fix serious bug in Skynet::Partitioners::RecombineAndSplit where i…

    …t wouldn't handle empty results well.
    
     - Fixed Skynet::Partitioners::ArrayDataSplitByFirstEntry to handle strings as keys better
    phlapjack committed May 3, 2008
  2. Update history

    phlapjack committed May 3, 2008
  3. Update History

    phlapjack committed May 3, 2008
  4. svn merge -r 1845:1860 svn+ssh://phlapjack@rubyforge.org/var/svn/skyn…

    …et/branches/ts_manager
    
    - Skynet manager now supports being daemonized by running ./script/skynet start and ./script/skynet stop!
    - Change mysql text fields to longtext in migration and schema files
    - Include some extras including our init.d script, our nagios monitoring script, rails controller and view for monitoring
    - Created a new skynet rails initializer which gets installed from skynet_install --rails
    - Modified the skynet_install skynet runner to take skynet initializer into account
    - Skynet::Manager - Work on the shutdown proceedure. Much more graceful now.
    - Changed some of the debug levels in Skynet::Worker and Skynet::Manager
    - Skynet::ActiveRecordExtensions Fixed a bug where it would fail if your table had fewer than 1000 rows.  It performs a count
     first now to make sure there are enough rows.
    - Fix a comment in Skynet::MapreduceHelper.  self.reduce should ahve been self.reduce_each
    - Fix some bugs in skynet_config where it would choke on Skynet::Config.logfile_location if the SKYNET_LOG_FILE was set to a filehandle like STDOUT.
    - Fix a skynet manager bug where it would get weird empty BigDecimals from the worker statuses
    - Added Skynet::Job.results_by_job_id to retrieve results from asyncronous jobs
    - Moved all worker stats into the manager class which now does stats no matter what worker_queue_adapter your using.  The stats are more detailed now as well.
    - Removed all stats code from worker queue classes
    - Added a stats_for_hosts method to Skynet::Manager which aggregates stats accross many managers.
    - Move worker methods out of the mysql message queue adapter into a new mysql worker queue adapter
    - Added a Skynet::Manager#stats method
    - Skynet::Manager now saves its stats to a file and reloads them when it starts.
    - Skynet::Manager.get returns the current local manager drb object
    - Workers now just add their statuses to the worker queue (without trying to take their old one off)
    - Managers just keep polling the queue, taking worker statuses off, and holding an internal hash of worker statuses.
    - The Manager doesn't pass around the worker_pids or worker_queue anymore, just updates when needed and uses the cached one.
    - Changed how the skynet pid file and log file are specified.  You can now specify the directory and file separately.
    - Lowered the WORKER_CHECK_DELAY
    - Made some API changes to Skynet::Config so you can call class or instance methods for all the CONFIG options.
    - Add new WORKER_QUEUE_ADAPTER config setting
    - Create Skynet::MessageQueue.start_or_connect
    - More work on Skynet::WorkerQueue class
    phlapjack committed May 3, 2008
Commits on Apr 11, 2008
  1. Rename skynet_schema.sql to skynet_mysql_schema.sql

    phlapjack committed Apr 11, 2008
  2. Fix up the skynet script that installs with skynet_install. Make sure…

    … it puts the log and pid file in the right place.
    phlapjack committed Apr 11, 2008
  3. update webiste

    phlapjack committed Apr 11, 2008
  4. Add new SkynetDebugger method printlog which always prints to the log…

    … regardless of the log level. Updated docs
    phlapjack committed Apr 11, 2008