scheduler can NOT enqueue job correctly sometimes

[ceshihao]

I use resque-scheduler v4.0.0

Recently, my resque scheduler can NOT enqueue jobs correctly sometimes.
There are 3 phenomenons, but i can NOT find root cause so far.

1. Sometimes, all scheduled jobs can NOT be enqueued automatically, but they can be triggered from resque web.
2. Sometimes, some of jobs can be enqueued automatically, but some are not.
3. Sometimes, all jobs work correctly.

I have some hints but not sure.

• dynamic schedule is enabled.
• I have 2 scheduler for HA.

[carsonreinke]

@ceshihao Well that is concerning.

In the resque-web, do you see all the scheduled jobs?

[ceshihao]

I found it was because of an incompatible problem between static and dynamic schedules.

• Previously, I use static scheduler and load the schedules from a yaml. It works and No schdule lost.

• Recently, I try to use dynamic one. However, dynamic scheduler thinks the schedules are temporary ones if NOT set persist = true. Therefore, when one scheduler is killed for some reason, original static ones are cleared (v4.0.0 behavior).

[carsonreinke]

@ceshihao It seems like changing the default to persist: true would make the most sense and prevent the issue you had in the future?

[ceshihao]

Yes, I think so.

BTW
I find resque-scheduler consuming memory increases a lot according to the number of schedules.

In my case, I have ~30 schedules after migrated to dynamic scheduler (the number may increase a little because dynamic), and the scheduler consumes ~500 MB memory.

I did NOT suppose it consumes so many memory for just cron schedulers.

@carsonreinke
Do you have any experience on scheduler memory consuming?

[carsonreinke]

@ceshihao I think that is more or less Ruby, but that is virtual, not real memory. Tweaking the Ruby VM garbage collection settings might help.