[JENKINS-76295] SSH agents stopped working on Jenkins 2.537

[jenkins-infra-bot]

I just updated to Jenkins 2.537, and suddenly launching SSH agents started to fail:

[11/18/25 18:56:35] [SSH] Starting sftp client.
[11/18/25 18:56:35] [SSH] Copying latest remoting.jar...
java.io.IOException: Could not copy remoting.jar into '/home/jenkins/.jenkins-cd-control' on agent
    at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:738)
    at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.lambda$launch$0(SSHLauncher.java:462)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalArgumentException: invalid len argument
    at PluginClassLoader for trilead-api//com.trilead.ssh2.SFTPv3Client.read(SFTPv3Client.java:1250)
    at PluginClassLoader for trilead-api//com.trilead.ssh2.jenkins.SFTPClient$SFTPInputStream.read(SFTPClient.java:172)
    at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:337)
    at org.apache.commons.io.input.BoundedInputStream.read(BoundedInputStream.java:536)
    at org.apache.commons.io.output.AbstractByteArrayOutputStream.writeImpl(AbstractByteArrayOutputStream.java:405)
    at org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream.write(UnsynchronizedByteArrayOutputStream.java:227)
    at org.apache.commons.io.IOUtils.copyToOutputStream(IOUtils.java:1958)
    at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:2918)
    at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.readInputStreamIntoByteArrayAndClose(SSHLauncher.java:796)
    at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:705)
    ... 5 more
[11/18/25 18:56:35] Launch failed - cleaning up connection
[11/18/25 18:56:35] [SSH] Connection closed.

Reverting back to Jenkins 2.536 fixed this issue again, thus some part of the update seems to be broken.

---

Originally reported by stefan6419846, imported from: SSH agents stopped working on Jenkins 2.537

• assignee: markewaite
• status: Closed
• priority: Blocker
• component(s): core
• resolution: Fixed
• resolved: 2025-11-19T12:01:28+00:00
• votes: 3
• watchers: 7
• imported: 2025-11-24

Raw content of original issue

I just updated to Jenkins 2.537, and suddenly launching SSH agents started to fail:

[11/18/25 18:56:35] [SSH] Starting sftp client.
[11/18/25 18:56:35] [SSH] Copying latest remoting.jar...
java.io.IOException: Could not copy remoting.jar into '/home/jenkins/.jenkins-cd-control' on agent
    at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:738)
    at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.lambda$launch$0(SSHLauncher.java:462)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalArgumentException: invalid len argument
    at PluginClassLoader for trilead-api//com.trilead.ssh2.SFTPv3Client.read(SFTPv3Client.java:1250)
    at PluginClassLoader for trilead-api//com.trilead.ssh2.jenkins.SFTPClient$SFTPInputStream.read(SFTPClient.java:172)
    at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:337)
    at org.apache.commons.io.input.BoundedInputStream.read(BoundedInputStream.java:536)
    at org.apache.commons.io.output.AbstractByteArrayOutputStream.writeImpl(AbstractByteArrayOutputStream.java:405)
    at org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream.write(UnsynchronizedByteArrayOutputStream.java:227)
    at org.apache.commons.io.IOUtils.copyToOutputStream(IOUtils.java:1958)
    at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:2918)
    at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.readInputStreamIntoByteArrayAndClose(SSHLauncher.java:796)
    at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:705)
    ... 5 more
[11/18/25 18:56:35] Launch failed - cleaning up connection
[11/18/25 18:56:35] [SSH] Connection closed.

Reverting back to Jenkins 2.536 fixed this issue again, thus some part of the update seems to be broken.

environment

Jenkins 2.537
ssh-slaves 3.1085.vc64d040efa_85

1 attachment

• image-2025-11-18-16-05-47-409.png

  

[jenkins-infra-bot]

• Remote links associated with this issue:

  • PR-11310 - Allow SSH build agents to connect by reverting Apache commons-io 2.21.0
  • Jenkins developer list proposal for an unscheduled weekly release

[jenkins-infra-bot]

ifernandezcalvo:

• Original comment link
• Raw content of original comment:

  The release of SSH build agents was on Oct 18th; no new releases after that. The Jenkins Core version you use is the weekly version, not the LTS. Jenkins 2.537 was released on November 11, 2025. I understand the same version is running in Jenkins Core 2.536. This is the first issue reported after a week of release.
  The exception is a failure to copy the remote.jar file to the home folder (/home/jenkins/.jenkins-cd-control) of the user used to connect via SSH.
  Review the permissions on that folder and check that you can establish an SFTP connection between the Jenkins controller and the SSH agent. In the user groups, there is one problem reported that is exactly the same https://community.jenkins.io/t/error-with-comunication-between-master-jenkins-and-jenkins-nodes/1290/9
  
  Before report an issue please read How to report an issue
  
  Some notes about Priority in issues: 
  
  	
  The reporter does not set the priority
  	
  An error in a weekly version never has priority; it is the unstable version of the Jenkins Core
  	
  A critical/blocker issue is set depending on the number of users affected; one user affected is never critical.
  
  
  

The release of SSH build agents was on Oct 18th; no new releases after that. The Jenkins Core version you use is the weekly version, not the LTS. Jenkins 2.537 was released on November 11, 2025. I understand the same version is running in Jenkins Core 2.536. This is the first issue reported after a week of release.
The exception is a failure to copy the remote.jar file to the home folder (/home/jenkins/.jenkins-cd-control) of the user used to connect via SSH.
Review the permissions on that folder and check that you can establish an SFTP connection between the Jenkins controller and the SSH agent. In the user groups, there is one problem reported that is exactly the same https://community.jenkins.io/t/error-with-comunication-between-master-jenkins-and-jenkins-nodes/1290/9

Before report an issue please read How to report an issue

Some notes about Priority in issues:

• The reporter does not set the priority
• An error in a weekly version never has priority; it is the unstable version of the Jenkins Core
• A critical/blocker issue is set depending on the number of users affected; one user affected is never critical.

[jenkins-infra-bot]

stefan6419846:

• Original comment link
• Raw content of original comment:

  Sorry if there has been confusion about the priority: Setting this by myself has never been an issue in the past, although I never had an issue which would affect basically my whole setup. From the documentation, it is not obvious to me that the weekly releases are considered for "beta testers" - it rather sounds like they just receive updates more frequently. As with all FOSS, I am aware that there is limited liability.
  
  Regarding the versions, there seems to be some mix-up from your side: Version 2.536 has been released on November 11, 2025, while version 2.537 has just been released today (November 18, 2025). The version of SSH build agents remained unchanged for the tests.
  
  The release of previous week (version 2.536) is running completely fine and does not show this failure - it did not before and does not after rolling back to it after experiencing issues. This setup has been working for years and has remained unchanged the whole time, thus I would rule out permissions or connection issues as well. Some quick manual tests show that establishing an SSH connection still works as expected; network issues are unlikely as well due to controller and agent using the same host.
  
  Thus Jenkins 2.357 seems to somehow break this in my setup. I will try to have another look at this if possible and I am open to guidance regarding further debugging this.
  

Sorry if there has been confusion about the priority: Setting this by myself has never been an issue in the past, although I never had an issue which would affect basically my whole setup. From the documentation, it is not obvious to me that the weekly releases are considered for "beta testers" - it rather sounds like they just receive updates more frequently. As with all FOSS, I am aware that there is limited liability.

Regarding the versions, there seems to be some mix-up from your side: Version 2.536 has been released on November 11, 2025, while version 2.537 has just been released today (November 18, 2025). The version of SSH build agents remained unchanged for the tests.

The release of previous week (version 2.536) is running completely fine and does not show this failure - it did not before and does not after rolling back to it after experiencing issues. This setup has been working for years and has remained unchanged the whole time, thus I would rule out permissions or connection issues as well. Some quick manual tests show that establishing an SSH connection still works as expected; network issues are unlikely as well due to controller and agent using the same host.

Thus Jenkins 2.357 seems to somehow break this in my setup. I will try to have another look at this if possible and I am open to guidance regarding further debugging this.

[jenkins-infra-bot]

markewaite:

• Original comment link
• Raw content of original comment:

  Thanks for the report stefan6419846.
  
  I'm able to duplicate the issue as you reported it.  When I use Jenkins 2.537 to connect to my ssh agents, they all report the following in the log for the agent:
  
  
  [11/18/25 13:13:30] [SSH] Copying latest remoting.jar...
  java.io.IOException: Could not copy remoting.jar into '/home/jagent/testing-b.markwaite.net-agent' on agent
  	at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:738)
  	at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.lambda$launch$0(SSHLauncher.java:462)
  	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  	at java.base/java.lang.Thread.run(Unknown Source)
  Caused by: java.lang.IllegalArgumentException: invalid len argument
  	at PluginClassLoader for trilead-api//com.trilead.ssh2.SFTPv3Client.read(SFTPv3Client.java:1250)
  	at PluginClassLoader for trilead-api//com.trilead.ssh2.jenkins.SFTPClient$SFTPInputStream.read(SFTPClient.java:172)
  	at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:337)
  	at org.apache.commons.io.input.BoundedInputStream.read(BoundedInputStream.java:536)
  	at org.apache.commons.io.output.AbstractByteArrayOutputStream.writeImpl(AbstractByteArrayOutputStream.java:405)
  	at org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream.write(UnsynchronizedByteArrayOutputStream.java:227)
  	at org.apache.commons.io.IOUtils.copyToOutputStream(IOUtils.java:1958)
  	at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:2918)
  	at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.readInputStreamIntoByteArrayAndClose(SSHLauncher.java:796)
  	at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:705)
  	... 5 more
  [11/18/25 13:13:30] Launch failed - cleaning up connection
  
  
  Plugin versions installed on my controller are documented in my plugins.txt file.  I will reassign this issue to Jenkins core, since the harmful change seems to be in the Jenkins 2.537 release and not in the SSH build agents plugin.
  We consider weekly releases to be ready for production just as much as LTS releases are.  I disagree with ifernandezcalvo saying that weekly is the unstable version of Jenkins.  It is the version that we release every week and we actively strive to keep it ready for production at all times.  In this case, it looks like something changed that broke the weekly release.
  

Thanks for the report stefan6419846.

I'm able to duplicate the issue as you reported it. When I use Jenkins 2.537 to connect to my ssh agents, they all report the following in the log for the agent:

[11/18/25 13:13:30] [SSH] Copying latest remoting.jar...
java.io.IOException: Could not copy remoting.jar into '/home/jagent/testing-b.markwaite.net-agent' on agent
	at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:738)
	at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.lambda$launch$0(SSHLauncher.java:462)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException: invalid len argument
	at PluginClassLoader for trilead-api//com.trilead.ssh2.SFTPv3Client.read(SFTPv3Client.java:1250)
	at PluginClassLoader for trilead-api//com.trilead.ssh2.jenkins.SFTPClient$SFTPInputStream.read(SFTPClient.java:172)
	at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:337)
	at org.apache.commons.io.input.BoundedInputStream.read(BoundedInputStream.java:536)
	at org.apache.commons.io.output.AbstractByteArrayOutputStream.writeImpl(AbstractByteArrayOutputStream.java:405)
	at org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream.write(UnsynchronizedByteArrayOutputStream.java:227)
	at org.apache.commons.io.IOUtils.copyToOutputStream(IOUtils.java:1958)
	at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:2918)
	at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.readInputStreamIntoByteArrayAndClose(SSHLauncher.java:796)
	at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:705)
	... 5 more
[11/18/25 13:13:30] Launch failed - cleaning up connection

Plugin versions installed on my controller are documented in my plugins.txt file. I will reassign this issue to Jenkins core, since the harmful change seems to be in the Jenkins 2.537 release and not in the SSH build agents plugin.

We consider weekly releases to be ready for production just as much as LTS releases are. I disagree with ifernandezcalvo saying that weekly is the unstable version of Jenkins. It is the version that we release every week and we actively strive to keep it ready for production at all times. In this case, it looks like something changed that broke the weekly release.

[jenkins-infra-bot]

aoutler:

• Original comment link
• Raw content of original comment:

  Confirmed. I have the same issue.  It started today when I updated Jenkins.  All my nodes except Synology went offline
  
    
  
   
  
  my synology is using an older version of sshd and is 127.0.0.1:
  
  OpenSSH_8.2p1, OpenSSL 1.1.1u  30 May 2023 
  
   
  For reference, servers which went offline are all version:
  
  OpenSSH_9.6p1 Ubuntu-3ubuntu13.14, OpenSSL 3.0.13 30 Jan 2024
   
  
  

Confirmed. I have the same issue.  It started today when I updated Jenkins.  All my nodes except Synology went offline

 

 

my synology is using an older version of sshd and is 127.0.0.1:

OpenSSH_8.2p1, OpenSSL 1.1.1u  30 May 2023 

 

For reference, servers which went offline are all version:

OpenSSH_9.6p1 Ubuntu-3ubuntu13.14, OpenSSL 3.0.13 30 Jan 2024
 

[jenkins-infra-bot]

mawinter69:

• Original comment link
• Raw content of original comment:

  I see the same issue on a SLES12 SP5 machine that has OpenSSH_7.2p2 with the current Jenkins head repo
  The culprit is the update of commons-io to version 2.21.0 with Update dependency commons-io:commons-io to v2.21.0 #11276
  Downgrading commons-io to 2.20.0 fixes the issue
  

I see the same issue on a SLES12 SP5 machine that has OpenSSH_7.2p2 with the current Jenkins head repo
The culprit is the update of commons-io to version 2.21.0 with #11276
Downgrading commons-io to 2.20.0 fixes the issue

[jenkins-infra-bot]

markewaite:

• Original comment link
• Raw content of original comment:

  The culprit is the update of commons-io to version 2.21.0 with Update dependency commons-io:commons-io to v2.21.0 #11276
  
  Thanks very much!  That matches what I see as well.  Considering the severity of the issue, I'm recommending to timja that we create a new weekly release tomorrow so that users of the weekly release are not blocked by this issue for more than 24 hours.
  

The culprit is the update of commons-io to version 2.21.0 with #11276

Thanks very much! That matches what I see as well. Considering the severity of the issue, I'm recommending to timja that we create a new weekly release tomorrow so that users of the weekly release are not blocked by this issue for more than 24 hours.

[jenkins-infra-bot]

JIRAUSER157093:

• Original comment link
• Raw content of original comment:

  I opened [Regression] Could not copy remoting.jar into 'c:\jenkins' on agent - invalid len argument  [>2.532,<=2.537] #11312 earlier for this same issue
  Wasn't sure where jenkins issues belong now
  Good to see im not the only one
  

I opened #11312 earlier for this same issue
Wasn't sure where jenkins issues belong now
Good to see im not the only one

[jenkins-infra-bot]

markewaite:

• Original comment link
• Raw content of original comment:

  My sincere apologies to everyone.  I merged the commons-io upgrade without performing detailed checks that it did not break key areas.  I'll investigate further to understand why the automated tests passed and see if there is a reasonable way to add an automated test for this issue.
  

My sincere apologies to everyone. I merged the commons-io upgrade without performing detailed checks that it did not break key areas. I'll investigate further to understand why the automated tests passed and see if there is a reasonable way to add an automated test for this issue.