Sudden loss of all Schedulers / Jobs

[Vandersteen]

Describe the bug

Once in a while (give or take once a month) all of our jobs disappear from the scheduler. We have not found a root cause for it yet. We create / update jobs 'dynamically' at runtime. When looking for the jobkeys, they are suddenly lost. Without rescheduling these jobs 'manually' they are no longer triggered.

Version used

3.0.5

To Reproduce

await context.Scheduler.ScheduleJob(
    new JobDetailImpl() { Key = new JobKey("someName", "someGroup"), ... }, 
    new CronTriggerImpl("someCronName", "someGroup"){ ... }
);

At a regular interval we do something like:

private async Task<bool> IsJobAlreadyScheduledAsync(string jobName, IJobExecutionContext context)
{
    var jobKeys = await context.Scheduler.GetJobKeys(GroupMatcher<JobKey>.AnyGroup());
    return jobKeys.Any(k => k.Name == jobName);
}

At some point, this returns false, even if we know that the job was scheduled (it was running for a month before suddenly being gone). Even more, all the jobs are gone and/or no longer triggering.

We do not see any exceptions / issue in our logs. The application does not suffer from memory / cpu pressure at the point of failure. Only that the scheduled jobs no longer fire

Expected behavior

Scheduled jobs not to disappear

Additional context
We are using the RAM store

[lahma]

Sounds very curious. First I'd enable logging and see if there's something odd happening at the time of disappearance. Secondly if you have such dynamic behavior why don't you use a persistent store? In-memory store is best for applications that know beforehand their tasks and can safely restart on server reboots etc.

[Vandersteen]

I do understand there is not much for you guys to go on.

However, we do have logging enabled and there is nothing special going on.

It just happens sporadically from time to time.

We have added additional logging just to be safe, I will report them here once it happens again.

[lahma]

Another note. If there's not triggers left the job will be removed from scheduler if it doesn't have Durable set to true. So if there's a moment in time in your system where no trigger refers to a job, it will be cleaned.

[Vandersteen]

Interesting, what could cause cronjobtriggers to disappear ?

[lahma]

Well if they don't have any firings left 🙂 That would be a cron trigger with fixed year or an end time.

[Vandersteen]

I don't believe we do have an end time specified. But I will look into that path sometime next week.
Thanks for your insights

[lahma]

Converting this to a discussion, we can open an issue if we find something actionable.