[BUG] Not all the recurring jobs are getting scheduled

[meghVi]

Are you running the latest version of JobRunr?
Yes

Describe the bug
Not all the recurring jobs are are getting scheduled and processed at the trigger time. All these jobs have the same trigger time.
Below are the scenarios I tried. Please find attached the Dashboard screen shots for scenario with 100 jobs created:
Dashboard_screenshots.docx

workers 50
- recurring jobs created 35 - All 35 jobs scheduled and succeeded
- recurring jobs created 100 - 56 scheduled - 55 processed and succeeded. Remaining not triggered. No jobs Failed

workers 100 - recurring jobs created 100 - 3 scheduled - 2 succeeded. Remaining not triggered. No jobs failed. (Screenshots in the attached doc)

Also, tried removing the worker-count property so that the Jobrunr would initiate based on available resources.
workers initiated 64 by jobrunr
- recurring jobs created 55 - All 55 processed and succeeded.
- recurring jobs created 100 - 66 scheduled and 65 succeeded. No Jobs failed (Screenshots in the attached doc)

Environment
I'm using JobRunr version: 5.0.1
I'm running on JRE / JDK: java 11
I'm using the following StorageProvider : MongoDB
jobRunr properties configured in my app:
org.jobrunr.background-job-server.enabled= true
org.jobrunr.background-job-server.dashboard.enabled= true
org.jobrunr.background-job-server.dashboard.worker-count (Tried with 50,100 and default initiation by Jobrunr)

How to Reproduce
Using jobrunr to schedule background recurring jobs.

      BackgroundJob.scheduleRecurrently(
              scheduleId,
              cronDetails.getCronExpression(),
              ZoneId.of(TIMEZONE_UTC),
              () -> schedulerMqJobRequestProducerService.execute(schedulerJobTask));

Here schedulerMqJobRequestProducerService is a service that pushes the scheduleJobTask on to MQ.

Expected behavior
All the recurring jobs should get scheduled and processed. Can you please let me know if there is any required configuration missing on my end that is leading to this problem.

Additional context
Later, I also tried adding the org.jobrunr.background-job-server.poll-interval= 60. But in that case, the Background Job Server details kept on showing the default polling interval as 15.

[rdehuyss]

I'm afraid that I cannot reproduce it. See all the details below. Feel free to reopen if you can create a Github repo that reproduces this.

I created 100 recurring jobs as follows:

    @GetMapping("/recurring-jobs")
    public String recurringJobs() {
        for(int i = 0; i < 100; i++) {
            int finalI = i;
            BackgroundJob.scheduleRecurrently(
                    "scheduleId-" + i,
                    Cron.every30seconds(),
                    ZoneOffset.UTC,
                    () -> sampleService.executeSampleJob("schedulerJobTask" + finalI));
        }
        return "100 recurring jobs enqueued.";
    }

The job:

@Service
public class SampleJobService {

    public void executeSampleJob(String input) throws InterruptedException {
        Thread.sleep(8000); // simulate a long running job
        System.out.println("Executing sample job with input: " + input);
    }
}

Overview of my background job server (default settings on Mac M1)

Overview of the 100 recurring jobs

After 1 minute (using Cron.every30Seconds()):

After 3 minutes:

[meghVi]

Hi @rdehuyss,
I am on Windows OS. Also, can you please try once with a daily cron to run a job just once per day. Meanwhile, I will setup a Github repo to reproduce this issue.

Thanks.