Real time trigger processing #442
gilberto-p-matos-alb
started this conversation in
Ideas
Replies: 1 comment 3 replies
-
Hi @gilberto-p-matos , although this was originally planned for JobRunrPro (exact scheduling) I'm also open for a PR for exact scheduling. With the implementation you suggest, I see some problems when having multiple If I would add exact job scheduling in the Pro version, would it warrant the license for you? |
Beta Was this translation helpful? Give feedback.
3 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi, first of all, congrats on developing Jobrunr.
We've been using quartz for several years as our base for distributed scheduling and one of the main pains we always struggled with was its triggering per second rate, which was much lower than expected because the process of getting the triggers and firing them is very time demanding
Because of that we recently started testing other alternatives and JobRunr has provided great results, providing a high scheduling trigger rate.
Also, the visibility given by the provided dashboard and its native support for MongoDB / NoSql is a plus.
One "caveat" we found, is that the query executed to fetch the scheduled jobs takes into consideration the polling interval time, meaning that scheduled tasks may be fired a few seconds ahead of time (the same happens in quartz and other frameworks). We understand the rationale behind this approach, and for long-run tasks, this isn't an issue.
Unfortunately, this is undesirable for our scenario because not only do we trigger a lot of tasks simultaneously, but we also have some aggressive timeouts (5 to 30 seconds), which we don't want to fire ahead of time.
We took a look into Jobrunr code, mainly the
JobZooKeeper
andBackgroundJobServer
classes.While reading through
BackgroundJobServer
, we noticed that the only implementation ofJobRunrExecutor
isScheduledThreadPoolJobRunrExecutor
. Because it is an extension of java'sScheduledThreadPoolExecutor
, it can schedule tasks.So we did a little hack, where we tried the following change to
BackgroundJobServer
This is a very raw implementation, but our initial tests gave good results, with no triggers being fired ahead of time: with 2 nodes, we achieved a throughput of 350 tasks being triggered per second.
We were wondering if a similar approach was initially intended since the Executor implementation already is a ScheduledThreadPoolExecutor.
If not, would consider having this feature (for example, configuration of "fireAheadTimeMs", which by default was equal to pollIntervalInSec), or, as an alternative, provide a way of customizing this behaviour?
Beta Was this translation helpful? Give feedback.
All reactions