WIP: cluster: make scheduler configurable

[cjihrig]

This is a WIP for making the cluster scheduler configurable by users. This still needs docs, tests, etc.

In the current implementation, the cluster module exports a Scheduler class. The class provides the following methods/hooks:

• constructor() - Used to configure data structures for managing connections (handles) and cluster workers.
• addConnection() - Called when a new connection is received. This is where the user would store the incoming connection handle to be distributed later.
• addWorker() - Called when a new cluster worker is added. This is where the user would add the worker into the pool of workers.
• removeWorker() - Called when a cluster worker is being removed from the pool.
• shutdown() - Called when the scheduler is finished working. This would be used to clean up any lingering resources such as handles.
• schedule() - This is the scheduling algorithm. The output of the algorithm should be a connection handle to distribute, and the worker to distribute it to.

I tried to minimize the exposure of the cluster's inner workings to incoming connection handles. Worker objects are already part of the public API.

Remaining questions:

• Should we enforce that handles from incoming connections are stored in a queue (array)? This would eliminate the need for the current addConnection() hook.
• Should we enforce that workers are stored in a "worker id to worker object" map, like they currently are? This would eliminate the need for the addWorker() and removeWorker() hooks.
• If we can get rid of the connections and workers hooks, then the shutdown hook can probably go. Then we could drop the Scheduler class, and just provide a schedule() function that takes the available workers and handles as input, and schedules accordingly. Ideally, I'd like to go this simpler route, but want to check with some people who are actually doing this in practice to make sure it would be flexible enough. cc: @Yemanu and @redonkulus from RFC cluster: make scheduler pluggable #10880. Would this approach work for your needs?

Closes #10880

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

Affected core subsystem(s)

cluster

[Fishrock123]

not really sure I'll ever get used to that one

[mscdex]

The variable name aliasing for this style is even harder to remember/get used to.

[Fishrock123]

possibly neater/more readable as

var handle;
while (handle = this.handles.shift()) {
  handle.close();
}

linter might complain less too?

[bnoordhuis]

This class-based approach seems way overkill. Why is plugging in a custom schedule function not sufficient?

[cjihrig]

I agree, and I think that it should be sufficient (see "Remaining questions" in the PR description). It really just comes down to whether or not people who do this need to define their own data structures for managing the workers and handles, or if we can standardize on the current data structures.

[cjihrig]

Ping @Yemanu and @redonkulus. I'd like your feedback before dropping the class based approach.

[yemanett]

@cjihrig, we still want to use all functionality provided by the RR scheduler, but we want to customize the schedule(callback) method. It looks like RR is no longer visible to the public API, right?

[cjihrig]

@yemanett all of the functionality will still be available, but you'll be responsible for scheduling connections to the worker yourself. For RR, that's pretty simple: you grab the first worker out of an array, then put that worker in the back of the array.

My question is whether or not you need special hooks for managing the data structures for the connection queue and workers. My guess is no.

[yemanett]

@cjihrig sorry I missed line https://github.com/cjihrig/node-1/blob/c47fe62a1d53959a12c741731608b7efaf5856e1/lib/internal/cluster/master.js#L300 which lead me to my previous incorrect comment.

Either way should be fine, but the Class based approach feels more intuitive. In that case we need just addWorker(), removeWorker and schedule.

[cjihrig]

@yemanett may I ask how you plan to use the addWorker() and removeWorker() hooks? Note that these are called when a worker is added or removed. They are not used to add and remove workers.

If I recall correctly, your original use case was to take workers offline for some amount of time. In that scenario, I would recommend that the child process sends an IPC message to the cluster master saying that it needs to go offline. The master can then take that information into account when calling schedule().

[yemanett]

@cjihrig Basically, we are planning to use it with our application runner module which manages the node processes including re-spawning a new worker when a child process dies. The application runner requires cluster. The runner loads the scheduler script in the master process. The scheduler takes a process out of rotation based on its algorithm by calling the removeWorker(w) method which removes w from the ‘this.free’ list and notify the worker via IPC message. The worker can then perform some tasks and notify master via PCI to put it back in rotation, in which case master calls addWorker(w) to add w into this.free list.

[cjihrig]

The scheduler takes a process out of rotation based on its algorithm by calling the removeWorker(w) method which removes w from the ‘this.free’ list and notify the worker via IPC message.

That's not how removeWorker() works. It isn't used to take a worker offline. It's a function that is called once the worker is already offline. In your case, the scheduler would take the worker out of rotation based on its own algorithm, or some message received from the worker. With the worker then marked as offline, the schedule() function just wouldn't route any requests to it.

The worker can then perform some tasks and notify master via PCI to put it back in rotation, in which case master calls addWorker(w) to add w into this.free list. In your case, the worker would send an IPC message to the master, and the master would mark the worker as online again. Then schedule() would see that worker as a valid receiver of requests.

Based on this, it still seems like schedule() is all that's needed.

Also not how addWorker() works. It would be invoked when a cluster worker comes online.

[yemanett]

@cjihrig yes schedule is what we need, but currently it takes worker and handle, how do we get the handle?

[bnoordhuis]

What do you need the handle for? schedule()'s job is to pick the next worker. Ideally, you don't need to look at the handle for that.

That said, schedule() should be flexible enough to support a use case like IP-based load balancing. That's difficult though because the cluster module operates on internal handle objects, not on net.Socket objects.

[yemanett]

@bnoordhuis

What do you need the handle for? schedule()'s job is to pick the next worker. Ideally, you don't need to look at the handle for that.
Because the above proposal says "Then we could drop the Scheduler class, and just provide a schedule() function that takes the available workers and handles as input, and schedules accordingly."

That said, schedule() should be flexible enough to support a use case like IP-based load balancing. That's difficult though because the cluster module operates on internal handle objects, not on net.Socket objects.

Our use case simple, it is just to take a worker OOR(out of rotation) so that master won't distribute new incoming requests to the worker while the worker is OOR . We still want to continue to use RR scheduler. But we just need to add a functionality to take a worker offline. The sequence diagram look something like this

[cjihrig]

That said, schedule() should be flexible enough to support a use case like IP-based load balancing. That's difficult though because the cluster module operates on internal handle objects, not on net.Socket objects.

@bnoordhuis do you have any preference between:

1. Dropping the handles from schedule(). IP-based load balancing wouldn't really be possible.
2. Passing the raw handles to schedule(). This would expose internals.
3. Wrapping the handles in a net.Socket. There is probably some performance overhead.

[bnoordhuis]

I'd say option 3, probably with a way to opt in so people with no need for the handle don't have to pay the overhead.

[cjihrig]

@bnoordhuis I've gotten rid of the scheduler class. You would just have to pass in a scheduler function now. You can request that the socket be passed in by adding a flag to the scheduler function. Before I go about writing docs and tests, what do you think?

[bnoordhuis]

Looks like an alright approach to me.

[bnoordhuis]

readable = writable = false. Is that intentional?

[cjihrig]

I figured that the socket should only be used to determine where to pass the handle, and not actually read or written.

[yemanett]

@cjihrig, the scheduler works nicely for me, The changes looks good to me.

Thanks,

[cjihrig]

@bnoordhuis updated.

• Removed the cluster.SCHED_CUSTOM constant.
• Added stronger language to the docs about altering workers

I went with scheduler over schedule since the function is the thing that determines the schedule.

[bnoordhuis]

LGTM although the wording in the documentation about not modifying the array could still be stronger.

[cjihrig]

@bnoordhuis do you have any specific wording in mind? I'm happy to try to scare users.

[bnoordhuis]

"The array should under no condition be mutated"?

[cjihrig]

OK, I'll update to that, although we do mutate the array in the round robin scheduler.

[bnoordhuis]

Yes, but we are allowed to break our own rules.

[cjihrig]

Updated to the suggested wording.

CI: https://ci.nodejs.org/job/node-test-pull-request/7265/. Of course Windows wouldn't work.

[yemanett]

The following test is failing: test/windows-fanned

[yemanett]

@cjihrig - do you know why test test/windows-fanned is failing? the 'Detail' link returns status 502

[cjihrig]

@yemanett the CI results are only kept around temporarily. It looks like those ones are no longer available.

[yemanett]

@cjihrig - when will this be merged ?

[cjihrig]

@yemanett It needs a rebase on the documentation, and some work to make the CI pass on Windows.

[redonkulus]

@cjihrig are you still actively working on this PR? If so, when do you think it can be completed?

[cjihrig]

@redonkulus still working on it, but if you wanted to debug the test on Windows, I wouldn't try to stop you.

[yemanett]

@cjihrig do you think the PR issue will be resolved and merged soon?

[Trott]

Labeling stalled. Feel free to remove the label if that's wrong.

[cjihrig]

Someone else can take this over if they want.