Agents can't be reused - Broken Pipe

[nussera]

Jenkins and plugins versions report

Environment

Jenkins: 2.463
OS: Linux - 4.18.0-553.5.1.el8_10.x86_64
Java: 21.0.3 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
analysis-model-api:12.3.3
ansicolor:1.0.4
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-1.0
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.113.v81215a_241826
authorize-project:1.7.2
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-233.vfdcdeb_0a_08a_a_
branch-api:2.1169.va_f810c56e895
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.0
cloud-stats:336.v788e4055508b_
cloudbees-folder:6.942.vb_43318a_156b_2
command-launcher:107.v773860566e2e
commons-compress-api:1.26.1-2
commons-lang3-api:3.14.0-76.vda_5591261cfe
commons-text-api:1.12.0-119.v73ef73f2345d
configuration-as-code:1810.v9b_c30a_249a_4c
configuration-as-code-groovy:1.1
credentials:1337.v60b_d7b_c7b_c9f
credentials-binding:677.vdc9d38cb_254d
data-tables-api:2.0.8-1
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:439.va_3cb_0a_6a_fb_29
docker-java-api:3.3.6-90.ve7c5c7535ddd
docker-plugin:1.6.2
durable-task:555.v6802fe0f0b_82
echarts-api:5.5.0-1
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1814.v404722f34263
envinject:2.908.v66a_774b_31d93
envinject-api:1.199.v3ce31253ed13
font-awesome-api:6.5.2-1
forensics-api:2.4.0
git:5.2.2
git-client:5.0.0
git-forensics:2.1.0
gitlab-api:5.3.0-91.v1f9a_fda_d654f
gitlab-branch-source:704.vc7f1202d7e14
gradle:2.12
gson-api:2.11.0-41.v019fcf6125dc
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
http_request:1.18
instance-identity:185.v303dc7c645f9
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jersey2-api:2.42-147.va_28a_44603b_d5
job-dsl:1.87
job-restrictions:0.8
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-2
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
ldap:725.v3cb_b_711b_1a_ef
mailer:472.vf7c289a_4b_420
mask-passwords:173.v6a_077a_291eb_5
material-theme:0.5.2-rc100.6121925fe229
matrix-project:832.va_66e270d2946
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.12.1-113.v4d3ea_5eb_7f72
mina-sshd-api-core:2.12.1-113.v4d3ea_5eb_7f72
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-graph-view:304.va_f2a_16b_e4964
pipeline-groovy-lib:727.ve832a_9244dfa_
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2198.v41dd8ef6dd56
pipeline-model-definition:2.2198.v41dd8ef6dd56
pipeline-model-extensions:2.2198.v41dd8ef6dd56
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56
pipeline-stage-view:2.34
pipeline-utility-steps:2.17.0
plain-credentials:182.v468b_97b_9dcb_8
plugin-util-api:4.1.0
prism-api:1.29.0-15
prometheus:773.v3b_62d8178eec
pyenv-pipeline:2.1.2
remote-file:1.24
role-strategy:727.vd344b_eec783d
scm-api:690.vfc8b_54395023
script-security:1341.va_2819b_414686
shiningpanda:0.24
simple-theme-plugin:176.v39740c03a_a_f5
snakeyaml-api:2.2-111.vc6598e30cc65
ssh-credentials:337.v395d2403ccd4
ssh-slaves:2.973.v0fa_8c0dea_f9f
sshd:3.330.vc866a_8389b_58
structs:337.v1b_04ea_4df7c8
terraform:1.0.10
theme-manager:262.vc57ee4a_eda_5d
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
variant:60.v7290fc0eb_b_cd
warnings-ng:11.3.0
workflow-aggregator:596.v8c21c963d92d
workflow-api:1316.v33eb_726c50b_a_
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3903.v48a_8836749e9
workflow-durable-task-step:1353.v1891a_b_01da_18
workflow-job:1426.v2ecb_a_a_42fd46
workflow-multibranch:791.v28fb_f74dfca_e
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:907.v6713a_ed8a_573
xml-job-to-job-dsl:0.1.13

What Operating System are you using (both controller, and any agents involved in the problem)?

The server is based on the jenkins/jenkins:2.463-jdk21 image
The agents are based on the jenkins/inbound-agent:latest-bookworm-jdk17 image
The underlying server is a RHEL9 VM
The container runtime is PODMAN

Reproduction steps

1. Define an agent template via 'configuration-as-code' like the following:

clouds:
  - docker:
      dockerApi:
        connectTimeout: 5
        dockerHost:
          uri: "unix:///run/user/10001/podman/podman.sock"
        readTimeout: 60
      errorDuration: 20
      name: "local-docker-cloud"
      templates:
      - connector: "attach"
        dockerTemplateBase:
          environment:
          - "JENKINS_WEB_SOCKET=true"
          environmentsString: |-
            JENKINS_WEB_SOCKET=true
          image: "jenkins/inbound-agent:latest-bookworm-jdk17"
        instanceCapStr: "5"
        labelString: "foo bar"
        mode: EXCLUSIVE
        name: "cloud1"
        nodeProperties:
        - jobRestrictionProperty:
            jobRestriction:
              regexNameRestriction:
                checkShortName: false
                regexExpression: "^cloud-jobs/.*$"
        pullStrategy: PULL_ALWAYS
        pullTimeout: 300
        remoteFs: "/home/jenkins/agent

2. Create a pipeline job in "/cloud-jobs" which uses a node with the label "cloud1".
3. Trigger the job - Everything works
4. Wait a couple of seconds
5. Trigger the job again - Sometimes the job is stuck waiting for the node

Expected Results

The second invokation of the job will reuse the available node and execute successfully.
If the node can't be used anymore, it should start a new one and discard the old one.

Actual Results

Sometimes the second job get stuck waiting for the existing node to be ready again. It seems to ignore any specified agent timeouts.
In the logs you will find messages like these:

java.io.IOException: Broken pipe
	at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.write(Unknown Source)
	at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
	at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
	at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
	at java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)
	at PluginClassLoader for docker-java-api//com.github.dockerjava.transport.UnixSocket$WrappedWritableByteChannel.write(UnixSocket.java:95)
	at java.base/sun.nio.ch.ChannelOutputStream.writeFully(Unknown Source)
	at java.base/sun.nio.ch.ChannelOutputStream.write(Unknown Source)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flush(SessionOutputBufferImpl.java:126)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.BHttpConnectionBase.flush(BHttpConnectionBase.java:308)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.flush(DefaultBHttpClientConnection.java:68)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:144)
	at PluginClassLoader for docker-java-api//com.github.dockerjava.httpclient5.HijackingHttpRequestExecutor.execute(HijackingHttpRequestExecutor.java:50)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:218)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:717)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:216)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:116)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:188)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
	at PluginClassLoader for apache-httpcomponents-client-5-api//org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)
	at PluginClassLoader for docker-java-api//com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:206)
Caused: java.lang.RuntimeException
	at PluginClassLoader for docker-java-api//com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:210)
	at PluginClassLoader for docker-java-api//com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9)
	at PluginClassLoader for docker-java-api//com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
	at PluginClassLoader for docker-java-api//com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
	at java.base/java.lang.Thread.run(Unknown Source)

Anything else?

No response

Are you interested in contributing a fix?

I can write a bit of java, but I'm not familiar with the development of Jenkins plugins.
If you can point me to where I need to change what, this can be done.

[MarkEWaite]

Thanks for your interest @nussera . We'd love to have more support for podman in the plugin. Unfortunately, I don't have any advice for the location to change. I'd need to duplicate the problem and then use a debugger to identify the changes that would be needed.

[nussera]

Some update to this issue: When I switched vom docker sockets to tcp and jnlp mode it started working. I still get ClosedChannelException, but it never hangs waiting for a new agent to be deployed.
Maybe this helps to somehow narrow this down.