Skip to content

[JENKINS-34213] Under high throughput the Unexporter can fail to clean up fast enough #1158

Closed
Bug
Closed
[JENKINS-34213] Under high throughput the Unexporter can fail to clean up fast enough#1158
Bug
Labels
component:remotingimported-jira-issuepriority:minorresolution:fixed

Description

@jenkins-infra-bot
jenkins-infra-bot
opened on Apr 13, 2016

The Unexporter does not try to drain the queue, rather it unexports at most one object every 200ms... which can create GC pressure when the remoting layer is under stress.

Originally reported by stephenconnolly, imported from: Under high throughput the Unexporter can fail to clean up fast enough
  • assignee: stephenconnolly
  • status: Closed
  • priority: Minor
  • component(s): remoting
  • label(s): 1.651.2-rejected, 1.651.3-fixed
  • resolution: Fixed
  • resolved: 2016-04-13T20:53:17+00:00
  • votes: 0
  • watchers: 4
  • imported: 2025-11-25
Raw content of original issue 
The Unexporter does not try to drain the queue, rather it unexports at most one object every 200ms... which can create GC pressure when the remoting layer is under stress.

Activity

  1. jenkins-infra-bot commented on Apr 13, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on Apr 13, 2016
    Author
    scm_issue_link:

    Code changed in jenkins
    User: Stephen Connolly
    Path:
    src/main/java/hudson/remoting/RemoteInvocationHandler.java
    http://jenkins-ci.org/commit/remoting/aee5a6f1756d6bcfba0123567e0195c4d584f8a2
    Log:
    [FIXED JENKINS-34213] Ensure that the unexporter cleans up whatever it can each sweep

  2. jenkins-infra-bot
    closed this as completedon Apr 13, 2016

  3. jenkins-infra-bot commented on Apr 13, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on Apr 13, 2016
    Author
    aheritier:

    stephenconnolly why is is it resolved ? The fix isn't yet merged / released ??

    aee5a6f

  4. jenkins-infra-bot commented on Apr 14, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on Apr 14, 2016
    Author
    scm_issue_link:
    • Original comment link
    • Raw content of original comment: 
      Code changed in jenkins
User: Stephen Connolly
Path:
 src/main/java/hudson/remoting/RemoteInvocationHandler.java
http://jenkins-ci.org/commit/remoting/98a7e3a837853ab1e8bf8b008862ed5738378bfc
Log:
  JENKINS-34213 Collect and report meaningful stats, batch the reference collection for better stability
      


        
	
      • We want to report on these things only if they are an issue. Logging of the actual stats should be below the radar of the users using a default logger level of `INFO` provided that the Unexporter is not doing much
        • 
	
      • When the Unexporter is busy (i.e. the m1 rate is > 100/sec) then we should start reporting at `INFO`
        • 
	
      • In the event that there is sustained high levels of work, we should alert the user and recommend turning off the stack traces to reduce GC pressure
        • 

      



        
	
      • My stress testing revealed that under very heavy load it is better to batch the removal and then batch the clean-up even if this batching means that the sweeps are not as frequent.
        •

    Code changed in jenkins
    User: Stephen Connolly
    Path:
    src/main/java/hudson/remoting/RemoteInvocationHandler.java
    http://jenkins-ci.org/commit/remoting/98a7e3a837853ab1e8bf8b008862ed5738378bfc
    Log:
    JENKINS-34213 Collect and report meaningful stats, batch the reference collection for better stability

    • We want to report on these things only if they are an issue. Logging of the actual stats should be below the radar of the users using a default logger level of `INFO` provided that the Unexporter is not doing much
    • When the Unexporter is busy (i.e. the m1 rate is > 100/sec) then we should start reporting at `INFO`
    • In the event that there is sustained high levels of work, we should alert the user and recommend turning off the stack traces to reduce GC pressure
    • My stress testing revealed that under very heavy load it is better to batch the removal and then batch the clean-up even if this batching means that the sweeps are not as frequent.

  5. jenkins-infra-bot commented on Apr 14, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on Apr 14, 2016
    Author
    scm_issue_link:

    Code changed in jenkins
    User: Stephen Connolly
    Path:
    src/main/java/hudson/remoting/RemoteInvocationHandler.java
    http://jenkins-ci.org/commit/remoting/d9654829cade2bbedae787a1b5db5c2a66005b03
    Log:
    JENKINS-34213 Oops typo in the stats measurement

  6. jenkins-infra-bot commented on Apr 14, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on Apr 14, 2016
    Author
    scm_issue_link:

    Code changed in jenkins
    User: Stephen Connolly
    Path:
    src/main/java/hudson/remoting/RemoteInvocationHandler.java
    http://jenkins-ci.org/commit/remoting/4dbf0ea480defd9bf98fca6d8a50f499df4ff08d
    Log:
    JENKINS-34213 Copy and paste error

    The three most common programming errors are:

    • Off by one errors
    • Copy and paste errors

  7. jenkins-infra-bot commented on Apr 15, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on Apr 15, 2016
    Author
    scm_issue_link:

    Code changed in jenkins
    User: Stephen Connolly
    Path:
    src/main/java/hudson/remoting/RemoteInvocationHandler.java
    http://jenkins-ci.org/commit/remoting/c9c077b7508fd715315a59a0423b7faec0b1a143
    Log:
    JENKINS-34213 Just when you were least expecting it, findbugs finds an actual real bug

  8. jenkins-infra-bot commented on Apr 20, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on Apr 20, 2016
    Author
    scm_issue_link:
    • Original comment link
    • Raw content of original comment: 
      Code changed in jenkins
User: Stephen Connolly
Path:
 src/main/java/hudson/remoting/RemoteInvocationHandler.java
http://jenkins-ci.org/commit/remoting/f4d287652e09da627f51779bec1c3ca7f1d46e70
Log:
  [FIXED JENKINS-34213] Ensure that the unexporter cleans up whatever it can each sweep (#81)
      


        
	
      • [FIXED JENKINS-34213] Ensure that the unexporter cleans up whatever it can each sweep
        • 

      



        
	
      • JENKINS-34213 Collect and report meaningful stats, batch the reference collection for better stability
        • 

      



        
	
      • We want to report on these things only if they are an issue. Logging of the actual stats should be below the radar of the users using a default logger level of `INFO` provided that the Unexporter is not doing much
        • 
	
      • When the Unexporter is busy (i.e. the m1 rate is > 100/sec) then we should start reporting at `INFO`
        • 
	
      • In the event that there is sustained high levels of work, we should alert the user and recommend turning off the stack traces to reduce GC pressure
        • 

      



        
	
      • My stress testing revealed that under very heavy load it is better to batch the removal and then batch the clean-up even if this batching means that the sweeps are not as frequent.
        • 

      






        
	
      • Ignore when the channel is already closed due to a race between testing for close and close
        • 

      



      Spotted in https://jenkins.ci.cloudbees.com/job/libraries/job/remoting/177/console
      





      The three most common programming errors are:
      


        
	
      • Off by one errors
        • 
	
      • Copy and paste errors
        • 

      



        
	
      • JENKINS-34213 Just when you were least expecting it, findbugs finds an actual real bug
        •

    Code changed in jenkins
    User: Stephen Connolly
    Path:
    src/main/java/hudson/remoting/RemoteInvocationHandler.java
    http://jenkins-ci.org/commit/remoting/f4d287652e09da627f51779bec1c3ca7f1d46e70
    Log:
    [FIXED JENKINS-34213] Ensure that the unexporter cleans up whatever it can each sweep (#81)

    • [FIXED JENKINS-34213] Ensure that the unexporter cleans up whatever it can each sweep
    • JENKINS-34213 Collect and report meaningful stats, batch the reference collection for better stability
    • We want to report on these things only if they are an issue. Logging of the actual stats should be below the radar of the users using a default logger level of `INFO` provided that the Unexporter is not doing much
    • When the Unexporter is busy (i.e. the m1 rate is > 100/sec) then we should start reporting at `INFO`
    • In the event that there is sustained high levels of work, we should alert the user and recommend turning off the stack traces to reduce GC pressure
    • My stress testing revealed that under very heavy load it is better to batch the removal and then batch the clean-up even if this batching means that the sweeps are not as frequent.
    • Ignore when the channel is already closed due to a race between testing for close and close

    Spotted in https://jenkins.ci.cloudbees.com/job/libraries/job/remoting/177/console

    The three most common programming errors are:

    • Off by one errors
    • Copy and paste errors
    • JENKINS-34213 Just when you were least expecting it, findbugs finds an actual real bug

  9. jenkins-infra-bot commented on May 2, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 2, 2016
    Author
    aheritier:
    • Original comment link
    • Raw content of original comment: 
      What do we do stephenconnolly ? Do we try to provide a stable version of the remoting to have this included in 1.651.3 ?

    What do we do stephenconnolly ? Do we try to provide a stable version of the remoting to have this included in 1.651.3 ?

  10. jenkins-infra-bot commented on May 2, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 2, 2016
    Author
    stephenconnolly:
    • Original comment link
    • Raw content of original comment: 
      We need ci_jenkinsci_org to actually cut a new release of remoting... when that has been out long enough then the fix will be eligible for the LTS release line

    We need ci_jenkinsci_org to actually cut a new release of remoting... when that has been out long enough then the fix will be eligible for the LTS release line

  11. jenkins-infra-bot commented on May 14, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 14, 2016
    Author
    scm_issue_link:

    Code changed in jenkins
    User: Oleg Nenashev
    Path:
    pom.xml
    http://jenkins-ci.org/commit/jenkins/409438f36dc80f20964fb16f8d88041e11ba4ed4
    Log:
    [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344)

    Changes:

  12. jenkins-infra-bot commented on May 17, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 17, 2016
    Author
    oleg_nenashev:
    • Original comment link
    • Raw content of original comment: 
      It has been released as remoting-2.58 and jenkins-2.4

    It has been released as remoting-2.58 and jenkins-2.4

  13. jenkins-infra-bot commented on May 25, 2016

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 25, 2016
    Author
    scm_issue_link:

    Code changed in jenkins
    User: Oleg Nenashev
    Path:
    pom.xml
    http://jenkins-ci.org/commit/jenkins/12e79963cca5122351943ee107f65c3ad91a2e25
    Log:
    [JENKINS-19445, JENKINS-34213, JENKINS-34808, JENKINS-34121] Bump remoting to 2.59. (#2344)

    Changes:

    (cherry picked from commit 409438f36dc80f20964fb16f8d88041e11ba4ed4)

  14. jenkins-infra-bot
    added
    bugFor changelog: Fixes a bug.
    on Nov 25, 2025

  15. jenkins-infra-bot commented on Nov 25, 2025

    @jenkins-infra-bot
    jenkins-infra-bot
    on Nov 25, 2025
    Author

    [Original is_duplicated_by from Jira: JENKINS-31467]

  16. jenkins-infra-bot commented on Nov 25, 2025

    @jenkins-infra-bot
    jenkins-infra-bot
    on Nov 25, 2025
    Author

    [Original relates_to from Jira: JENKINS-26855]

  17. jenkins-infra-bot
    added
    imported-jira-issue
    component:remoting
    resolution:fixed
    priority:minor
    on Nov 25, 2025
  18. jenkins-infra-bot
    added the
    Bug
    issue type on Nov 26, 2025
  19. jenkins-infra-bot
    removed
    bugFor changelog: Fixes a bug.
    on Nov 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    component:remotingimported-jira-issuepriority:minorresolution:fixed

    Type

    Bug

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions