Repository navigation
[JENKINS-52165] Use push rather than pull for durable task logging #615
Description
Retroactively filing an issue originally treated as part of JENKINS-38381">JENKINS-38381, since it is independently mergeable and has some independent benefits even when using existing (per-step, master-based) log storage.
Historically, when running durable tasks (sh and kin), running output was handled by having the master send a callable to the agent at intervals, initially short (¼s) but growing exponentially up to some maximum (15s) if the process seems to be idle, and resetting to short again if and when fresh output is detected.
Not only is there up to a 15s delay in displaying new output, this is wasteful of master and network resources when the process is idle for a long time; and sending a UserRequest and corresponding response involves a fair amount of Java serialization.
By contrast, non-durable processes (such as those created by Launcher in a freestyle build) use RemoteOutputStream to send content from the agent to master as soon as it is available, minimizing network traffic; Remoting is also able to optimize this kind of traffic by sending low-overhead Chunk packets of tailored sizes.
Durable tasks should switch from the policy of pulling log output to having the agent push log output. The content cannot be detected immediately, since we are effectively tailing a log file, but it can be detected quickly after the log file is updated with minimal overhead.
Originally reported by 
jglick, imported from: Use push rather than pull for durable task logging
- assignee: jglick
- status: Reopened
- priority: Major
- component(s): durable-task-plugin, workflow-durable-task-step-plugin
- label(s): performance, remoting
- resolution: Unresolved
- votes: 5
- watchers: 12
- imported: 2025-12-09
Raw content of original issue
Retroactively filing an issue originally treated as part of
JENKINS-38381, since it is independently mergeable and has some independent benefits even when using existing (per-step, master-based) log storage.Historically, when running durable tasks (sh and kin), running output was handled by having the master send a callable to the agent at intervals, initially short (¼s) but growing exponentially up to some maximum (15s) if the process seems to be idle, and resetting to short again if and when fresh output is detected.
Not only is there up to a 15s delay in displaying new output, this is wasteful of master and network resources when the process is idle for a long time; and sending a UserRequest and corresponding response involves a fair amount of Java serialization.
By contrast, non-durable processes (such as those created by Launcher in a freestyle build) use RemoteOutputStream to send content from the agent to master as soon as it is available, minimizing network traffic; Remoting is also able to optimize this kind of traffic by sending low-overhead Chunk packets of tailored sizes.
Durable tasks should switch from the policy of pulling log output to having the agent push log output. The content cannot be detected immediately, since we are effectively tailing a log file, but it can be detected quickly after the log file is updated with minimal overhead.
Activity
jenkins-infra-bot commentedon Jun 25, 2018
Remote links associated with this issue:
jenkins-infra-bot commentedon Jun 25, 2018
Raw content of original comment:
This change supersedes the code affected by JENKINS-37575">
JENKINS-37575.jenkins-infra-bot commentedon Sep 25, 2018
Raw content of original comment:
Released in Durable Task version 1.26 and Pipeline Nodes and Processes 2.22.
jenkins-infra-bot commentedon Oct 24, 2018
Raw content of original comment:
Disabling this mode in workflow-durable-task-step 2.25 until some reported issues can be tracked down and fixed.
jenkins-infra-bot commentedon Mar 18, 2022
basil:
Raw content of original comment:
The experimental (newer) push mode reliably hangs when running on Java 17 in the context of the workflow-durable-task-step-plugin test suite (see JENKINS-68080">
JENKINS-68080). The default (older) pull mode passes on Java 17 in the same test suite.jenkins-infra-bot commentedon Dec 9, 2025
[Original
depends_onfrom Jira: JENKINS-71509]jenkins-infra-bot commentedon Dec 9, 2025
[Original
blocksfrom Jira: JENKINS-38381]