Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
A brief list of things it would be nice to see contributed to CloudCrowd:
- A Job Creation UI for the Operations Center … pick your action, add inputs and options, and you’re off to the races.
- More unit and acceptance tests.
- Additional storage backends for the AssetStore, besides s3 and filesystem. Database and memcache and webdav backends seem like they would be interesting options.
- Some performance tests and benchmarks, to see what the expected throughput looks like at different levels.
- It would be nice to take a look at Grand Central Dispatch for ideas on how to efficiently manage workers and job queues.
- A UI for changing the
max_workerssetting for each Node, via the Operations Center.
It would be great to add more conditions (in addition to
max_workers), to limit the number of Workers that can be run on a node.
max_loadseem particularly promising. The node would be considered busy if it hit any of those limits.
— (Version 0.2.1)
min_free_memoryare now implemented.
max_cpucan be added with the same mechanism as soon as someone figures out a good, accurate, cross-UNIX way to measure it.
Add/Remove Workers via the server UI – drnic
— (Version 0.2) Workers are created and removed on-the-fly, up to
max_workers, so there’s no longer a need to manually add and remove them.
And how about running a single daemon process per box, that fetches work and forks off workers. You could configure a “percentage of memory to leave free”, and the daemon would continue to fetch work and fork workers up to that limit. Would be a nice way to scale to the confines of a box while avoiding memory leaks.
— (Version 0.2) Now there’s the notion of Nodes, representing a physical machine that runs a cluster of ephemeral workers. Scales to the confines of the machine without persistent processes that could leak memory.
We should also at least consider a queue-per-worker / work-stealing setup.
— Work-stealing would only make sense for CloudCrowd if a particular node had a backed-up queue. That’s currently not the case. Any work units that a node receives begin execution immediately.
Clone this wiki locally
Press h to open a hovercard with more details.