Worker Locality

[edwardcapriolo]

Currently the Worker scans ZK looking for new Plans on an interval and if it has available slots attempts to start a worker for that plan.

We should have support for locality, meaning a Worker should be able to determine how "close" it is to the partition of a Feed. If it is "far" it might wait a few scans before taking the job, or potentially never take it at all.

I see this feature adding some information to the Plan object tot turn off/on. Some information to the feed object, and changing the logic in the worker.

[edwardcapriolo]

The current more immediate problem is getting a more even splay of the tasks across nodes. Depending on node start up times a node could easily have all the partitions of a particlular plan.

A plan will have a new optional property: maxWorkersPerNode (int). This setting will ensure in the trivial case that tasks get splayed out to different nodes as currently the worker polling only results in an optimistic splay.

Feed do not have a known number of partitions it is calculated by the instance of the Feed at runtime. However each Plan does support maxWorkers.

If maxWorkersPerNode * number of live nodes < maxWorkers some partitions will not be consumed. So potentially in that case we can avoid the rule

[edwardcapriolo]

Solved for now in edwardcapriolo/teknek-core#4