Worker Sharing

When you deploy a new version of Emcee, the new Emcee version will spawn its own set of workers, which in turn will spawn a new set of their own simulators. This can easily overload your worker machines: simulators and/or tests may become unresponsive, machines may run out of free disk space quickly.

To make upgrade process hassle-free, Emcee queues has an ability to share workers between each other. Worker sharing means the whole machine is dedicated to a single queue, while other queues won't use that machine to perform jobs. EmceeWorker processes will still run on all worker machines, but they will be no-op until they are allowed to execute jobs.

In order to take advantage of the worker sharing feature, all queue hosts must be listed in the queue configuration.

Each queue will search for other queues, and select a single one as a master queue. Then master queue will dedicate a subset of workers for each queue. When any queue dies, the workers will be re-balanced again. When only all old queues die, the most fresh queue will gain back all the workers.