Feature request and one question

[tosbaha]

Feature Request

I loved the simplicity and elegance of this library. I am planning to abandon fivebeans and switch to this one. I will appreciate if you can increase the documentation add some information about keeping connection, re-connecting etc.

Question

Is it possible to reserve multiple jobs with the worker pattern? Instead of forking lots of processes with while (true) {}, is it possible to do some parallel processing? Because another fivebeans based library named bsw has a feature that allows processing multiple jobs.

[divmgl]

Thanks for your input. I'll look into whether reserving multiple jobs is possible but I don't see why it wouldn't if you use cluster. Thanks.

[tosbaha]

Thanks for the reply. Bsw is doing something like this. Since nodejs is single threaded, it reserves jobs and increases job_counter and after each job, it decreases job counter. Example code is here

I really didn't know how cluster can be used in here. Isn't cluster designed for network processes? I don't know how this issue can be tackled with clustering. Those processes don't need any communication. Isn't is same as just forking many workers as we want? If we fork lots of processes, we will increase memory load. However if we reserve many jobs inside the same process, we will basically do the same thing without the memory overhead. Please let me know if I am wrong. Best regards.

[divmgl]

Here's the documentation for cluster:

A single instance of Node.js runs in a single thread. To take advantage of multi-core systems, the user will sometimes want to launch a cluster of Node.js processes to handle the load.

In the current README.md there's a small section that talks about the worker pattern where, essentially, you use a while loop to infinitely reserve jobs and process them. You can achieve the level of parallelism that you want by spinning up multiple workers inside of a cluster.

For instance, if you have eight logical CPUs, you can spin up eight workers so that each of them processes one job at a time, etc.

Please keep in mind that clustering and multi-threading is outside of the scope of this package and the documentation for it. Hope this helps.

[divmgl]

To give a concise answer to your question: we will not be implementing multiple job reservation in this library because it is outside of the scope of this library. If you are reserving multiple jobs at the same time within the same Node.js process, you're probably doing something wrong. Spin up many Node.js processes or start a cluster of workers within one process.

[tosbaha]

Thanks for detailed answer. I still think that there are use cases for multiple job reservations in a single process. If let's say you're connecting to multiple web sites and scraping data. Each site needs different extraction logic. If you use single job only, you will request one web site at a time. However with multiple job reservations you can fire 10 requests to websites at a time. Spinning multiple workers just increases memory. I totally understand your reasoning though. Thanks once again for detailed answers. Best regards.

[divmgl]

Thanks for detailed answer. I still think that there are use cases for multiple job reservations in a single process.

Sorry for necroing this thread, but this should be answered for anyone thinking about multithreading in the future.

Based on beanstalkd's implementation, it's simply not possible to issue concurrent commands a single connection. beanstalkd processes all incoming commands serially for each open socket.

Because of this limitation, you probably should not share one jackd instance across multiple threads, regardless of what language you're using. Instead, open a socket per thread. This will avoid all problems related to out-of-order messages and multi-threading.

However, if you really want to use one jackd instance with many different concurrent executing contexts, you technically can. jackd now keeps a queue of incoming messages and outgoing commands and will flush each of them serially. This use-case is not supported however, so you know, here be dragons or whatever.