Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Quartz Scheduler stops working after some time #121

Closed
mazkozi opened this issue Apr 5, 2017 · 3 comments
Closed

Quartz Scheduler stops working after some time #121

mazkozi opened this issue Apr 5, 2017 · 3 comments

Comments

@mazkozi
Copy link

mazkozi commented Apr 5, 2017

We are using Quartz scheduler (with the spring) to trigger few tasks. This set up was working for so many years without any issues. Recently, the scheduler stopped working all of a sudden. After server restart it works fine, however stops working again after some time. Can some one help me resolve this?

Below are the version details for your reference:

Windows Server: 2003
Quartz: 1.5.1
Spring: 3.2.2

I generated thread dump to check if every thing is ok. It looks fine. Below is the part of thread dump pertaining to Quartz threads.

"QuartzScheduler_QuartzScheduler-NON_CLUSTERED_MisfireHandler" daemon prio=6 tid=0x57c26400 nid=0x270 runnable [0x58ecf000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at oracle.net.ns.Packet.receive(Unknown Source)
    at oracle.net.ns.DataPacket.receive(Unknown Source)
    at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
    at oracle.net.ns.NetInputStream.read(Unknown Source)
    at oracle.net.ns.NetInputStream.read(Unknown Source)
    at oracle.net.ns.NetInputStream.read(Unknown Source)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:971)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:941)
    at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:432)
    at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:181)
    at oracle.jdbc.driver.T4CPreparedStatement.execute_for_describe(T4CPreparedStatement.java:420)
    at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:896)
    at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describe(T4CPreparedStatement.java:452)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:986)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:2888)
    at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:2929)
    - locked <0x10a0f020> (a oracle.jdbc.driver.T4CPreparedStatement)
    - locked <0x10f39060> (a oracle.jdbc.driver.T4CConnection)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
    at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.obtainLock(StdRowLockSemaphore.java:137)
    at org.quartz.impl.jdbcjobstore.JobStoreCMT.doRecoverMisfires(JobStoreCMT.java:1298)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:2409)
    at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:2428)

"QuartzScheduler_QuartzSchedulerThread" prio=6 tid=0x57bfc400 nid=0x1334 runnable [0x58e1f000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at oracle.net.ns.Packet.receive(Unknown Source)
    at oracle.net.ns.DataPacket.receive(Unknown Source)
    at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
    at oracle.net.ns.NetInputStream.read(Unknown Source)
    at oracle.net.ns.NetInputStream.read(Unknown Source)
    at oracle.net.ns.NetInputStream.read(Unknown Source)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:971)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:941)
    at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:432)
    at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:181)
    at oracle.jdbc.driver.T4CPreparedStatement.execute_for_describe(T4CPreparedStatement.java:420)
    at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:896)
    at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describe(T4CPreparedStatement.java:452)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:986)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:2888)
    at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:2929)
    - locked <0x10a0f358> (a oracle.jdbc.driver.T4CPreparedStatement)
    - locked <0x10f391c8> (a oracle.jdbc.driver.T4CConnection)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
    at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.obtainLock(StdRowLockSemaphore.java:137)
    at org.quartz.impl.jdbcjobstore.JobStoreCMT.acquireNextTrigger(JobStoreCMT.java:1134)
    at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:233)

"QuartzScheduler_Worker-9" prio=6 tid=0x57c75400 nid=0x794 in Object.wait() [0x58d6f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
    - locked <0x0ec94410> (a java.lang.Object)
    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

"QuartzScheduler_Worker-8" prio=6 tid=0x57c73c00 nid=0x1aa4 in Object.wait() [0x58cbf000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
    - locked <0x0ec94410> (a java.lang.Object)
    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

"QuartzScheduler_Worker-7" prio=6 tid=0x57c79400 nid=0xfec in Object.wait() [0x58c0f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
    - locked <0x0ec94410> (a java.lang.Object)
    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

"QuartzScheduler_Worker-6" prio=6 tid=0x57c77c00 nid=0x650 in Object.wait() [0x58b5f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
    - locked <0x0ec94410> (a java.lang.Object)
    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

"QuartzScheduler_Worker-5" prio=6 tid=0x57c69400 nid=0x8cc in Object.wait() [0x58aaf000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
    - locked <0x0ec94410> (a java.lang.Object)
    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

"QuartzScheduler_Worker-4" prio=6 tid=0x57c68000 nid=0x338 in Object.wait() [0x589ff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
    - locked <0x0ec94410> (a java.lang.Object)
    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

"QuartzScheduler_Worker-3" prio=6 tid=0x57c62800 nid=0x178c in Object.wait() [0x5894f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
    - locked <0x0ec94410> (a java.lang.Object)
    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

"QuartzScheduler_Worker-2" prio=6 tid=0x57c72400 nid=0x1e8c in Object.wait() [0x5889f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
    - locked <0x0ec94410> (a java.lang.Object)
    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

"QuartzScheduler_Worker-1" prio=6 tid=0x57c56400 nid=0x11bc in Object.wait() [0x587ef000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
    - locked <0x0ec94410> (a java.lang.Object)
    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

"QuartzScheduler_Worker-0" prio=6 tid=0x57c36000 nid=0x1770 in Object.wait() [0x5873f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
    - locked <0x0ec94410> (a java.lang.Object)
    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
@jhouserizer
Copy link
Contributor

Wow - 1.5.1 is many, MANY years old - like a decade. It's really impractical for us to provide any help with that. Even 1.8.x is ancient now.

Can you upgrade to at least 1.8.6 and see if the problem persists?

@gawande09
Copy link

We are using Quartz: 2.2.1 and we are facing the same issue all of a sudden, it was working fine till now but the triggers are in the waiting state and are not able to update the next fire time in quartz metadata table.

Below is the thread dump:

UDPQuartzScheduler_QuartzSchedulerThread" #50 prio=5 os_prio=0 tid=0x00007f903b526000 nid=0x7b3a in Object.wait() [0x00007f9094e64000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:410)
        - locked <0x00000006ddb1b350> (a java.lang.Object)

"UDPQuartzScheduler_Worker-8" #49 prio=4 os_prio=0 tid=0x00007f903b4ca800 nid=0x7b39 in Object.wait() [0x00007f9094f65000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
        - locked <0x00000006ddb1b368> (a java.lang.Object) 

"UDPQuartzScheduler_Worker-7" #48 prio=4 os_prio=0 tid=0x00007f903b4c8800 nid=0x7b38 in Object.wait() [0x00007f9095066000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
        - locked <0x00000006ddb1b380> (a java.lang.Object)

"UDPQuartzScheduler_Worker-6" #47 prio=4 os_prio=0 tid=0x00007f903b4c6800 nid=0x7b37 in Object.wait() [0x00007f9095167000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
        - locked <0x00000006ddb1b398> (a java.lang.Object)

"UDPQuartzScheduler_Worker-5" #46 prio=4 os_prio=0 tid=0x00007f903b4c5000 nid=0x7b36 in Object.wait() [0x00007f9095268000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
        - locked <0x00000006ddb1b3b0> (a java.lang.Object)

"UDPQuartzScheduler_Worker-4" #45 prio=4 os_prio=0 tid=0x00007f903b4c3000 nid=0x7b35 in Object.wait() [0x00007f9095369000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
        - locked <0x00000006ddb1b3f8> (a java.lang.Object)
"UDPQuartzScheduler_Worker-3" #44 prio=4 os_prio=0 tid=0x00007f903b4c0800 nid=0x7b34 in Object.wait() [0x00007f909546a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
        - locked <0x00000006ddb1caf0> (a java.lang.Object)

"UDPQuartzScheduler_Worker-2" #43 prio=4 os_prio=0 tid=0x00007f903b4bf000 nid=0x7b33 in Object.wait() [0x00007f909556b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
        - locked <0x00000006ddb1c9d0> (a java.lang.Object)

"UDPQuartzScheduler_Worker-1" #42 prio=4 os_prio=0 tid=0x00007f903b3e5800 nid=0x7b32 in Object.wait() [0x00007f909566c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
        - locked <0x00000006ddb1cb50> (a java.lang.Object)```

@zemian
Copy link
Contributor

zemian commented Feb 15, 2019

Closing outdated, unresolved issues. Please, reopen if still relevant.

@zemian zemian closed this as completed Feb 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jhouserizer @gawande09 @mazkozi @zemian