[Question] Scalability

[abnerjacobsen]

I am interested in using ndscheduler in a solution that I am developing but I have the following requirement:

1. My app will use the ndscheduler API to generate about 30,000 schedules per day within 2 hours time frame.

2. These schedules will run only once and their purpose will be to send an attendance confirmation email to a customer.

3. Schedules will have their execution start date and time distributed over a time period of 6 hours during the day.

4. After the schedule is successfully executed it can be deleted from the ndscheduler.

Does the ndscheduler support this workload?

Thank you.

[kartick]

Why don't you use a distributed task queue such as Huey, RQ, or Celery for this? Or you can use a combo of a task queue and NDS (though to me, it seems like an overkill)

[sabw8217]

So I think you could do this, but if you have jobs that will only run once, you probably want a distributed task queue for that. There's not really any reason generating 30K schedules a day wouldn't work, but we certainly haven't tested it.

If you want things to run at certain times, I'd recommend instead having a job that starts up periodically during the day and checks some other distributed queue (e.g. SQS, RabbitMQ) to see what work needs to be done.