Allow specific jobs or group execution per thread count/limit

[hirreddy]

Same as https://jira.terracotta.org/jira/browse/QTZ-258. Just porting it to GitHub for tracking.

Use-case:
We use quartz scheduler (2.0.2) in clustered mode and have a below use case .
There are about total 30 jobs and we have org.quartz.threadPool.threadCount=25. This limit gets applied to all the jobs. However we have 3-4 jobs that can be sometimes very long running and resource intensive (not always). So we want to put a limit on the number of these jobs running parallel on a node and cluster.

This would be really helpful.

[hirreddy]

This surely will help us. I did go through the code and it seems we have to set the job group limits to the scheduler using QuartzScheduler.setJobGroupExecutionLimit(String jobGroup, Integer limit).

Can these be externalized to quartz.properties? If it is already done, Please do mention how to set the limits in property files. May be something like below.
org.quartz.threadPool.threadCount.jobGroupName=5

[mederly]

Hello. Excuse me, I missed your comment (haven't set up notifications from this thread).

In last 2 days I reworked the limitations support to be independent of job keys, because of our own requirements (regarding midPoint product in which we use Quartz Scheduler). We need to be able to set up the execution groups for triggers in a very flexible way. So, there is an additional attribute of each trigger called executionGroup, and it drives the firing of such trigger on individual nodes. I am just finishing mini-documentation for it.

Having seen your message I will try to externalize the setting to quartz.properties. Will commit hopefully soon.

I hope the change from static job groups to dynamic execution groups will not cause you much trouble. It should be quite easy to use.

[mederly]

Hopefully finished. You can use quartz.properties to set the limits now.

The code is available here (version 2.3.0.e2).
The (short) documentation is here.

I have written a number of tests for this feature and everything looks ok. Feel free to try it in your environment.

[kadirtn]

XMLSchedulingDataProcessorPlugin is not updated for execution-group definitions.

Also if i define 2 triggers which belong to different execution groups for a StatefulJob, both of them can run simultaneously. But they should not since job is marked as @DisallowConcurrentExecution

[mederly]

I have to write some tests for concurrent execution. Hopefully will do that next week. (As for the XMLSchedulingDataProcessorPlugin, cannot promise anything. But I will at least have a look.)

[mederly]

At first sight the xml plugin seems doable.

[kadirtn]

concurrent execution problem still exists, i rechecked

[mederly]

:) No problem. I will try to change the plugin (next week).

[charlesviegas]

Hello how are you? I was very interested in this feature that limits the execution of the nodes by group. Do you have any deadlines to enter to be released?

[zemian]

A similar issue #344 is open and relate to this.

[jbarotin]

Hi @mederly and @zemian, this feature seems very nice.
Why it is not merged into the master of quartz ?

[mederly]

I am ready to prepare a pull request, if there's a reasonable chance it will be accepted :) What do you think, @zemian ?

Maybe not quite these days, as we are in a hurry releasing our product (that uses Quartz and this feature in particular).

But I have to admit that things mentioned above (xml plugin, concurrent execution) that I have promised to fix were not fixed because of the time restrictions. Maybe someone else could fix them, after merging this feature into master...

[praveendk]

Hi @mederly, I like this feature, looking for this one for a while. We are currently using 2.3.0 in our production, how do i take your changes and apply into our build? or is it good to take jar from this branch? I understood db change, it is easy. only change is how to take code changes form your branch to quartz branch?

[mederly]

Hello @praveendk. One possibility is to simply take https://github.com/Evolveum/quartz/tree/quartz-2.3.0.e2 and build it. This is what I would recommend for now. Or you can take compiled JAR from https://nexus.evolveum.com/nexus/repository/releases/org/quartz-scheduler/quartz/2.3.0.e2/quartz-2.3.0.e2.jar. This is what we use in our product midPoint (for 2 years now). Or you can wait if there would be a pull request to merge this to master Quartz branch (this could require a bit of merging), or try to create such a pull request yourself.

There's one additional inconvenience besides the two issues mentioned above: https://jira.evolveum.com/browse/MID-4558. It occurs from time to time but seems to be harmless. I or someone other should find a little time to look at it.

[s-jepsen]

Any progress on this issue?

[mederly]

I am not sure if there is any chance that a pull request would be accepted by the Quartz maintainers. So this is basically a question towards them... I can do almost nothing. (Except that e.g. in the meanwhile I have created 2.3.2e1 version, based on Quartz 2.3.2. See https://github.com/Evolveum/quartz.)

[s-jepsen]

@zemian, @chrisdennis : Any possibility of accepting/reviewing a PR for this issue, as proposed by @mederly ?
Appears there is some interest in this feature.

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[elgabbouch]

why this feature is closed? it seems to be very useful