Skip to content

[JENKINS-24763] Long running OldDataMonitor.doDiscard() results in thread starvation #23457

Closed
Closed
[JENKINS-24763] Long running OldDataMonitor.doDiscard() results in thread starvation#23457
Labels
component:coreenhancementimported-jira-issueperformancepriority:minorresolution:fixed

Description

@jenkins-infra-bot
jenkins-infra-bot
opened on Sep 18, 2014

A very large set of OldData can take a long time to discard. While this is running, most Jenkins operations which save any data are blocked.

https://gist.github.com/recampbell/9336d3a32270e75a9333

doDiscard is synchronized and hudson.diagnosis.OldDataMonitor#remove also wants to lock OldDataMonitor. Most threads eventually lock when saving something, at least until the doDiscard eventually completes.

Originally reported by recampbell, imported from: Long running OldDataMonitor.doDiscard() results in thread starvation
  • status: Resolved
  • priority: Minor
  • component(s): core
  • label(s): 1.580.3-rejected, lock, performance, threads
  • resolution: Fixed
  • resolved: 2014-10-10T19:31:34+00:00
  • votes: 1
  • watchers: 6
  • imported: 2025-11-24
Raw content of original issue 
A very large set of OldData can take a long time to discard. While this is running, most Jenkins operations which save any data are blocked.



https://gist.github.com/recampbell/9336d3a32270e75a9333



doDiscard is synchronized and hudson.diagnosis.OldDataMonitor#remove also wants to lock OldDataMonitor. Most threads eventually lock when saving something, at least until the doDiscard eventually completes.

Activity

  1. jenkins-infra-bot commented on Sep 18, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Sep 18, 2014
    Author
    danielbeck:
    • Original comment link
    • Raw content of original comment: 
      Appears to be a rather minor issue (with the workaround being to wait for ODM to finish).
      


      In what way is this a livelock?

    Appears to be a rather minor issue (with the workaround being to wait for ODM to finish).

    In what way is this a livelock?

  2. jenkins-infra-bot commented on Sep 18, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Sep 18, 2014
    Author
    recampbell:
    • Original comment link
    • Raw content of original comment: 
      In the meantime, no build can finish, and no slaves can come online. In the incident observed, the discard was taking more than an hour and Jenkins could not complete builds or do basically any other operation.
      


      It's unacceptable for Jenkins to be inoperable for minutes at a time, hence a higher severity.
      


      I suppose this is really just "starvation' instead of a livelock per se. Will update the description.

    In the meantime, no build can finish, and no slaves can come online. In the incident observed, the discard was taking more than an hour and Jenkins could not complete builds or do basically any other operation.

    It's unacceptable for Jenkins to be inoperable for minutes at a time, hence a higher severity.

    I suppose this is really just "starvation' instead of a livelock per se. Will update the description.

  3. jenkins-infra-bot commented on Sep 18, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Sep 18, 2014
    Author
    recampbell:

    Pull request: #1399

  4. jenkins-infra-bot commented on Sep 18, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Sep 18, 2014
    Author
    jglick:
    • Original comment link
    • Raw content of original comment: 
      What makes you think this is really a problem in OldDataMonitor? I look at the thread dump and I see JobConfigHistorySaveableListener being likely at fault.

    What makes you think this is really a problem in OldDataMonitor? I look at the thread dump and I see JobConfigHistorySaveableListener being likely at fault.

  5. jenkins-infra-bot commented on Sep 19, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Sep 19, 2014
    Author
    danielbeck:
    • Original comment link
    • Raw content of original comment: 
      FWIW JobConfigHistorySaveableListener takes forever in my experience if you enable the deduplication option ("Do not save duplicate history"). I've long disabled it for that reason.

    FWIW JobConfigHistorySaveableListener takes forever in my experience if you enable the deduplication option ("Do not save duplicate history"). I've long disabled it for that reason.

  6. jenkins-infra-bot commented on Sep 20, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Sep 20, 2014
    Author
    recampbell:
    • Original comment link
    • Raw content of original comment: 
      I see Jesse's point about JobConfigHistorySaveableListener – it's certainly the root cause.
      


      Nevertheless, OldDataMonitor could be more robust by reducing the scope of it's lock – to better handle other poorly performing SaveableListeners. So I'll change this to an Improvement if that suits you.

    I see Jesse's point about JobConfigHistorySaveableListener – it's certainly the root cause.

    Nevertheless, OldDataMonitor could be more robust by reducing the scope of it's lock – to better handle other poorly performing SaveableListeners. So I'll change this to an Improvement if that suits you.

  7. jenkins-infra-bot commented on Sep 20, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Sep 20, 2014
    Author
    jglick:
    • Original comment link
    • Raw content of original comment: 
      Makes sense.

    Makes sense.

  8. jenkins-infra-bot commented on Sep 24, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Sep 24, 2014
    Author
    recampbell:

    Latest pull request: #1402

  9. jenkins-infra-bot commented on Oct 10, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Oct 10, 2014
    Author
    scm_issue_link:

    Code changed in jenkins
    User: Ryan Campbell
    Path:
    core/src/main/java/hudson/diagnosis/OldDataMonitor.java
    test/src/test/java/hudson/diagnosis/OldDataMonitorTest.java
    http://jenkins-ci.org/commit/jenkins/423b51a43a3c2f469d12695c5fddcda52e97159e
    Log:
    JENKINS-24763 Prevent thread starvation in OldDataMonitor by reducing scope of synchronization

  10. jenkins-infra-bot commented on Oct 10, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Oct 10, 2014
    Author
    scm_issue_link:

    Code changed in jenkins
    User: Jesse Glick
    Path:
    core/src/main/java/hudson/diagnosis/OldDataMonitor.java
    test/src/test/java/hudson/diagnosis/OldDataMonitorTest.java
    http://jenkins-ci.org/commit/jenkins/ef6c04244744670650fb36a4482e74daf2094f26
    Log:
    Merge branch 'JENKINS-24763' of github.com:recampbell/jenkins

  11. jenkins-infra-bot commented on Oct 10, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Oct 10, 2014
    Author
    scm_issue_link:

    Code changed in jenkins
    User: Jesse Glick
    Path:
    changelog.html
    http://jenkins-ci.org/commit/jenkins/a8262d503e6af11feebac6005360d4db7639aff4
    Log:
    [FIXED JENKINS-24763] Noting merge of #1402.

    Compare: 3addbab...a8262d5

  12. jenkins-infra-bot
    closed this as completedon Oct 10, 2014

  13. jenkins-infra-bot commented on Oct 10, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Oct 10, 2014
    Author
    dogfood:

    Integrated in jenkins_main_trunk #3738
    JENKINS-24763 Prevent thread starvation in OldDataMonitor by reducing scope of synchronization (Revision 423b51a)
    [FIXED JENKINS-24763] Noting merge of #1402. (Revision a8262d5)

    Result = SUCCESS
    Ryan Campbell : 423b51a43a3c2f469d12695c5fddcda52e97159e
    Files :

    • test/src/test/java/hudson/diagnosis/OldDataMonitorTest.java
    • core/src/main/java/hudson/diagnosis/OldDataMonitor.java

    Jesse Glick : a8262d503e6af11feebac6005360d4db7639aff4
    Files :

    • changelog.html

  14. jenkins-infra-bot commented on Nov 21, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on Nov 21, 2014
    Author
    imakowski:
    • Original comment link
    • Raw content of original comment: 
      Could you please add this also to lts?

    Could you please add this also to lts?

  15. jenkins-infra-bot commented on Feb 5, 2015

    @jenkins-infra-bot
    jenkins-infra-bot
    on Feb 5, 2015
    Author
    danielbeck:
    • Original comment link
    • Raw content of original comment: 
      The 1.580.x line is done with the RC to 1.580.3 posted, and it will be in 1.596.x already.

    The 1.580.x line is done with the RC to 1.580.3 posted, and it will be in 1.596.x already.

  16. jenkins-infra-bot
    added
    performance
    imported-jira-issue
    component:core
    priority:minor
    enhancement
    on Nov 24, 2025

  17. jenkins-infra-bot commented on Nov 24, 2025

    @jenkins-infra-bot
    jenkins-infra-bot
    on Nov 24, 2025
    Author

    [Original is_duplicated_by from Jira: JENKINS-24303]

  18. jenkins-infra-bot
    added
    resolution:fixed
    on Nov 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    component:coreenhancementimported-jira-issueperformancepriority:minorresolution:fixed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions