Advice on proper use of Agenda

[raffaelescala97]

Hello everyone,
I just decided to adopt agenda for my project. I would like some advice on the correct usage.
I have a table with about 100k rows (Mongo), each with a unique id. I would like to set up a job every second that does an operation on each row.
Also I would like to parallelize the jobs on multiple machines, how can I make sure they don't iterate the same ids?
What is the correct approach in agenda?
Thanks in advance.

[koresar]

Typically, agenda creates a mongodb doc per each job. In your case it sounds like creating 100k job records - a job per each of the IDs.
There can be as many as you want agenda worker machines.

Not sure what else to add. Please be more specific with your questions.

[raffaelescala97]

Thank you for your reply. That's my point, I wanted to avoid creating a table with 100k jobs, also because in the future these 100k ids will grow and may become 900k, so could it not be redundant?
Could I have agenda work on the same table "Products" in my case and add the fields necessary for it to work?
The operation performed on each id is an api call, and a refresh of a field in the row related to the id.
I was thinking of creating a job that would fetch a different row at each iteration, keeping track of it with a stored boolean variable.
The problem also arises in the management of concurrency, how could I make sure that by increasing the number of machines the update is not done on the same id?

[harisvsulaiman]

Thank you for your reply. That's my point, I wanted to avoid creating a table with 100k jobs, also because in the future these 100k ids will grow and may become 900k, so could it not be redundant?
Could I have agenda work on the same table "Products" in my case and add the fields necessary for it to work?
The operation performed on each id is an api call, and a refresh of a field in the row related to the id.
I was thinking of creating a job that would fetch a different row at each iteration, keeping track of it with a stored boolean variable.
The problem also arises in the management of concurrency, how could I make sure that by increasing the number of machines the update is not done on the same id?

Atomic ops is coming to agenda. I'm working on , but haven't gotten into it.

If you are inclined, you can make it work.

[koresar]

The update won't be done on the same id if you create 100k job docs. It's what Agenda is about. It's core feature. (BTW, MongoDB does not have tables but collections.) Not wanting to create 100k docs - sounds like premature optimisation. 100k records is 1MB or 10MB of disk space. Computers were born to do exactly that, to simplify human's life. If you don't want to create 100k jobs - create 100, each for 1000 records. But then you don't need Agenda, it loses sense to use Agenda.

…

On Sun, 7 Feb 2021, 23:05 Raffaele Scala, ***@***.***> wrote: Thank you for your reply. That's my point, I wanted to avoid creating a table with 100k jobs, also because in the future these 100k ids will grow and may become 900k, so could it not be redundant? Could I have agenda work on the same table "Products" in my case and add the fields necessary for it to work? The operation performed on each id is an api call, and a refresh of a field in the row related to the id. I was thinking of creating a job that would fetch a different row at each iteration, keeping track of it with a stored boolean variable. The problem also arises in the management of concurrency, how could I make sure that by increasing the number of machines the update is not done on the same id? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1237 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMMEL2NCV6PM4VIFPIM3PDS5Z6W3ANCNFSM4XEQBTLQ> .

[koresar]

1 call per src means you don't need multiple workers. One is enough.

To simplify your life I'd recommend simply to add 1 sec delay just before the remote API call. That's all you need to process those 100k.

Alternatively, you coy your could query Agenda jobs, find the most recent one, calculate the minimal delay from it, then do the API call.

[raffaelescala97]

Sorry for the redundant questions but I am new to Agenda. So there is the possibility of defining 100k jobs, and a worker that will process them sequentially, so it picks up one job at a time and executes it? If yes, could you post me an example?
Because until now I thought there was only the possibility to create jobs, and each job had its own execution cron.

[koresar]

Sorry mate. I don't quite have time writing code for you.

Would you like me to find you a consultant you can pay so that he could write any code you like?

[raffaelescala97]

No worries thanks, I had only asked for advice on the correct way to proceed with Agenda. I can write my own code, I just asked about the difference between worker and job and if there were any examples in the documentation or online. If you don't have time you don't have to answer, thanks anyway.

[koresar]

It's all in the README.

Job is a doc in MongoDB. https://github.com/agenda/agenda#creating-jobs
Worker is a node.js process. https://github.com/agenda/agenda#spawning--forking-processes