Severe JobRunr Exception

[longtv02]

SevereJobRunrException occurred in BackgroundJobServer 137d7112-41c7-4009-9e55-388eeb13d4a5: Could not resolve ConcurrentJobModificationException

Runtime information

• Timestamp: 2021-05-12T10:04:27.958Z
• Location: BackgroundJobServer 137d7112-41c7-4009-9e55-388eeb13d4a5
• JobRunr Version: Unable to determine version
• StorageProvider: org.jobrunr.storage.nosql.elasticsearch.ElasticSearchStorageProvider
• Java Version: 1.8.0_282
• Is running from nested jar: false

Background Job Servers

• BackgroundJobServer id: 137d7112-41c7-4009-9e55-388eeb13d4a5
  (workerPoolSize: 64, pollIntervalInSeconds: 15, firstHeartbeat: 2021-05-12T10:01:14.788Z, lastHeartbeat: 2021-05-12T10:04:27.804Z)

Diagnostics from exception

Concurrent modified jobs:

Job id: 71cb54b3-dc5a-486a-b792-57f5ba7812d8
Local version: 26; Storage version: 26
Local state: SCHEDULED; Storage state: SUCCEEDED
Local last updated: 2021-05-12T10:04:27.771Z; Storage last updated: 2021-05-12T10:04:27.723Z

Exception

org.jobrunr.SevereJobRunrException: Could not resolve ConcurrentJobModificationException
	at org.jobrunr.server.JobZooKeeper.processJobList(JobZooKeeper.java:199)
	at org.jobrunr.server.JobZooKeeper.processJobList(JobZooKeeper.java:183)
	at org.jobrunr.server.JobZooKeeper.checkForOrphanedJobs(JobZooKeeper.java:121)
	at org.jobrunr.server.JobZooKeeper.runMasterTasks(JobZooKeeper.java:96)
	at org.jobrunr.server.JobZooKeeper.run(JobZooKeeper.java:72)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:308)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[rdehuyss]

Thanks for creating this issue.

I investigated it a bit and it seems that some tasks take quite a lot of time whereas I expect them to finish within 15 seconds (the poll interval in seconds).

I'm thinking of adding extra logging and releasing a new version so we can see which call(s) takes so long . Is this an option for you?