Repository navigation
[JENKINS-49707] Auto retry for elastic agents after channel closure #559
Description
While my pipeline was running, the node that was executing logic terminated. I see this at the bottom of my console output:
Cannot contact ip-172-31-242-8.us-west-2.compute.internal: java.io.IOException: remote file operation failed: /ebs/jenkins/workspace/common-pipelines-nodeploy at hudson.remoting.Channel@48503f20:ip-172-31-242-8.us-west-2.compute.internal: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on ip-172-31-242-8.us-west-2.compute.internal failed. The channel is closing down or has closed down
There's a spinning arrow below it.
I have a cron script that uses the Jenkins master CLI to remove nodes which have stopped responding. When I examine this node's page in my Jenkins website, it looks like the node is still running that job and i see an orange label that says "Feb 22, 2018 5:16:02 PM Node is being removed".
I'm wondering what would be a better way to say "If the channel closes down, retry the work on another node with the same label?
Things seem stuck. Please advise.
Originally reported by 
piratejohnny, imported from: Auto retry for elastic agents after channel closure
- assignee:
jglick
- status: Resolved
- priority: Critical
- component(s): workflow-durable-task-step-plugin
- resolution: Fixed
- resolved: 2022-10-17T15:11:50+00:00
- votes: 37
- watchers: 54
- imported: 20251212-090250
Raw content of original issue
While my pipeline was running, the node that was executing logic terminated. I see this at the bottom of my console output:
Cannot contact ip-172-31-242-8.us-west-2.compute.internal: java.io.IOException: remote file operation failed: /ebs/jenkins/workspace/common-pipelines-nodeploy at hudson.remoting.Channel@48503f20:ip-172-31-242-8.us-west-2.compute.internal: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on ip-172-31-242-8.us-west-2.compute.internal failed. The channel is closing down or has closed downThere's a spinning arrow below it.
I have a cron script that uses the Jenkins master CLI to remove nodes which have stopped responding. When I examine this node's page in my Jenkins website, it looks like the node is still running that job and i see an orange label that says "Feb 22, 2018 5:16:02 PM Node is being removed".
I'm wondering what would be a better way to say "If the channel closes down, retry the work on another node with the same label?
Things seem stuck. Please advise.
19 attachments
- JavaMelodyGrubHeapDump_4_07_18.pdf
- JavaMelodyNodeGrubThreads_4_07_18.pdf
- MonitoringJavaelodyOnNodes.html
- NetworkAndMachineStats.png
- Thread dump [Jenkins].html
- grub.remoting.logs.zip
- grubSystemInformation.html
- image-2018-02-22-17-27-31-541.png
- image-2018-02-22-17-28-03-053.png
- jenkins.log
- jenkins_Agent_devbuild9_System_Information.html
- jenkins_agent_devbuild9_remoting_logs.zip
- jenkins_agents_Thread_dump.html
- jenkins_support_2018-06-29_01.14.18.zip
- jobConsoleOutput.txt
- jobConsoleOutput.txt
- slaveLogInMaster.grub.zip
- support_2018-07-04_07.35.22.zip
- threadDump.txt
Activity
jenkins-infra-bot commentedon Feb 23, 2018
Remote links associated with this issue:
jenkins-infra-bot commentedon Feb 23, 2018
Raw content of original comment:
Not sure what is the request here. You get a system message from Remoting, it's not related to Pipeline or Jobs at all in general. If you want to implement retry features or document the suggestions, IMHO it is on the Pipeline side
jenkins-infra-bot commentedon Feb 23, 2018
Raw content of original comment:
oleg_nenashev I'm not sure how to handle this situation. The problem I need to overcome is that my pipeline hangs with the error message I have screenshotted above. I would much prefer that it errors out and fails. Unfortunately, the pipeline keeps running indefinately.
Can this instead be configured to throw a catchable exception?
jenkins-infra-bot commentedon Feb 27, 2018
Raw content of original comment:
oleg_nenashev Should this be redesignated a remoting bug? I'm not sure how to unblock my pipelines that are hanging from this issue.
jenkins-infra-bot commentedon Feb 27, 2018
Raw content of original comment:
i just changed the JIRA "component" field for this to "remoting".
jenkins-infra-bot commentedon Feb 27, 2018
Raw content of original comment:
You can find some pointers here: https://speakerdeck.com/onenashev/day-of-jenkins-2017-dealing-with-agent-connectivity-issues?slide=51
Please provide the following info:
You can find some pointers here: https://speakerdeck.com/onenashev/day-of-jenkins-2017-dealing-with-agent-connectivity-issues?slide=51
jenkins-infra-bot commentedon Mar 15, 2018
iamshital:
Raw content of original comment:
I would like to increase the Priority of this issue to "Major" since this issue is affecting a lot of users.
jenkins-infra-bot commentedon Mar 30, 2018
slaughter550:
Raw content of original comment:
We have also been greatly effected by this issue. A resolution would be very nice
jenkins-infra-bot commentedon Jun 13, 2018
Raw content of original comment:
We are receiving this message sporadically in cloud nodes in Azure managed by Jenkins.
jenkins-infra-bot commentedon Jun 29, 2018
Raw content of original comment:
Hi,
We are losing at a team to TeamCity mostly for this remoting issue
Here are the logs requested, the disconnection happened at 10:21 am (agent devbuild9)
JENKINS-49707">jenkins_agent_devbuild9_remoting_logs.zip
JENKINS-49707">jenkins_agents_Thread_dump.html
JENKINS-49707">jenkins_Agent_devbuild9_System_Information.html
JENKINS-49707">jenkins_support_2018-06-29_01.14.18.zip
I will appreciate any pointer, to where I can start looking for more information. Let me know if you need more logs.
This is happening several times daily so I can provide more logs if needed
jenkins-infra-bot commentedon Jun 29, 2018
Raw content of original comment:
At least we have some data for diagnostics now
jenkins-infra-bot commentedon Jul 4, 2018
Raw content of original comment:
This is a fresher issue, with fewer things going on, this time the agent that got disconnected is called grub
Job console output shows (jobConsoleOutput.txt) show at 17:27:54
jenkins master log at that time (jenkins.log) shows the following lines:
The agent remoting log that shows the error is the file created at 5:08 pm (remoting.log.2 inside grub.remoting.logs.zip)
but it does not have a timestamp in the message. it would be handy to have one. because I can not work out if the agent or Jenkins master initiated the disconnection.
I've also included
- System Information (grubSystemInformation.html)
- Heap Dump (JavaMelodyGrubHeapDump_4_07_18.pdf)
- threads (JavaMelodyNodeGrubThreads_4_07_18.pdf)
- (MonitoringJavaelodyOnNodes.html)
A screenshot (*NetworkAndMachineStats.png) of the stats of the master (jenkinssecure1) and the agent (grub) showing the network activity, memory and CPU history. Hardly anything going on both machines.
jenkins-infra-bot commentedon Jul 4, 2018
Raw content of original comment:
Here are the files:
JENKINS-49707">jobConsoleOutput.txt
JENKINS-49707">grub.remoting.logs.zip
JENKINS-49707">JavaMelodyGrubHeapDump_4_07_18.pdf
JENKINS-49707">JavaMelodyNodeGrubThreads_4_07_18.pdf
JENKINS-49707">MonitoringJavaelodyOnNodes.html
JENKINS-49707">grubSystemInformation.html
JENKINS-49707">Thread dump [Jenkins].html
JENKINS-49707">support_2018-07-04_07.35.22.zip
JENKINS-49707">slaveLogInMaster.grub.zip
JENKINS-49707">jenkins.log
jenkins-infra-bot commentedon Jul 23, 2018
Raw content of original comment:
Hi oleg_nenashev,
Do you have any update on this?
We have seen similar issues: The Jenkins Pipeline hangs when the node becomes unreachable at some point in time.
It would be great to see this fixed. This issue sometimes blocks many jobs in the queue of our CI.
In this case it was an intermittent networking issue:
finally, we aborted the build:
Please note that in the post steps, we see the errors occur but the build no longer hangs here:
I hope this somewhat helps.
With best regards,
Tom.
jenkins-infra-bot commentedon Jul 23, 2018
Raw content of original comment:
tom_ghyselinck nope, I don't. I have requested info which is needed to diagnose the issue, but I have never reviewed it. I will unlikely have time for that in short-term, busy with other stuff in the community. jthompson is the current Remoting default assignee, so I will assign the issue to him.
28 remaining items
jenkins-infra-bot commentedon Aug 16, 2019
Raw content of original comment:
jglick Just wanted to thank you and everybody else who's been working on jenkins and to confirm that the work over on https://issues.jenkins-ci.org/browse/JENKINS-36013 appears to have handled this case in a much better way. I consider the current behavior to be a major step in the right direction for Jenkins. Here's what I noticed:
Last night, our Jenkins worker pool did its normal scheduled nightly scale down and one of the pipelines got disrupted. The message I see in my affected pipeline's console log is:
{ agent \{ label 'universal' }Agent ip-172-31-235-152.us-west-2.compute.internal was deleted; cancelling node body
The above mentioned hostname is the one that Jenkins selected at at the top of my declarative pipeline as a result of my call for a 'universal' machine (universal is how we label all of our workers):
pipeline
...
This particular declarative pipeline tries to "sh" to the console at the end inside a post{} section and clean up after itself, but since the node was lost, the next error that also appears in the Jenkins console log is:
org.jenkinsci.plugins.workflow.steps.MissingContextVariableException: Required context class hudson.FilePath is missing
This error was the result of the following code:
post {
always {
sh """|#!/bin/bash
...
Let me just point out that the recent Jenkins advancements are fantastic. Before JENKINS-36013">
JENKINS-36013, this pipeline would have just been stuck with no error messages. I'm so happy with this progress you have no idea.Now if there's any way to get this to actually retry the step it was on such that the pipeline can actually tolerate losing the node, we would have the best of all worlds. At my company, the fact that a node is deleted during a scaledown is a confusing irrelevant problem for one of my developers to grapple with. The job of my developers (the folks writing Jenkins pipelines) is to write idempotent pipeline steps and my job is make sure all of the developer's steps trigger and the pipeline concludes with a high amount of durability.
Keep up the great work you are all doing. This is great.
jenkins-infra-bot commentedon Aug 16, 2019
Raw content of original comment:
The MissingContextVariableException is tracked by JENKINS-58900">
JENKINS-58900. That is just a bad error message, though; the point is that the node is gone.Well that is the primary subject of this RFE, my “subcase #1” above. Pending a supported feature, you might be able to hack something up in a trusted Scripted library like
jenkins-infra-bot commentedon Nov 19, 2019
Raw content of original comment:
We use kubernetes plugin with our bare-metal kubernetes cluster and the problem is that pipeline can run indefinitely if agent inside pod were killed/underlying node was restarted. Is there any option to tweak such behavior, e.g. some timeout settings (except explicit timeout step)?
jenkins-infra-bot commentedon Nov 19, 2019
Raw content of original comment:
oxygenxo that should have already been fixed—see linked PRs.
jenkins-infra-bot commentedon Nov 15, 2021
Raw content of original comment:
A very limited variant of this concept (likely not compatible with Pipeline) is implemented in the EC2 Fleet plugin: https://github.com/jenkinsci/ec2-fleet-plugin/blob/2d4ed2bd0b05b1b3778ec7508923e21db0f9eb7b/src/main/java/com/amazon/jenkins/ec2fleet/EC2FleetAutoResubmitComputerLauncher.java#L87-L108
jenkins-infra-bot commentedon Dec 12, 2025
[Original
depends_onfrom Jira: JENKINS-30383]