You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What Operating System are you using (both controller, and any agents involved in the problem)?
Centos7
Reproduction steps
Jenkins pipeline jobs fail to spawn shell commands if they were previously in a suspended state when using a docker multibranch pipeline.
Please note, this only happens when the job enters a "Suspended" state and keeps building any ways.
Not sure how to provide exact details here as I am still learning a lot about jenkins, but here is the problem.
Jobs that enter a suspended state fail to execute shell commands inside the container.
From what I can tell there is some sort of IO failure with the spawning of the container.
Jenkins logs here Logs here:
Cached connection io.jenkins.docker.client.DockerAPI$SharableDockerClient@57bfaad6 to DockerClientParameters{dockerUri='tcp://localhost:7504', credentialsId='null', readTimeoutInMsOrNull=100000, connectTimeoutInMsOrNull=100000}
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerCloud canAddProvisionedAgent
Provisioning 'teldev/centos7-builder:latest' number 1 (of 2) on 'docker-uat1'; Total containers: 0 (of 2)
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerCloud provision
Will provision 'teldev/centos7-builder:latest', for label: 'null', in cloud: 'docker-uat1'
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Trying to run container for image "teldev/centos7-builder:latest"
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Trying to run container for node centos7-builder-jekins-slave1-002uuz9748r51 from image: teldev/centos7-builder:latest
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Started container ID 922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f for node centos7-builder-jekins-slave1-002uuz9748r51 from image: teldev/centos7-builder:latest
Feb 23, 2023 12:57:54 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from centos7-builder-jekins-slave1-002uuz9748r51 (922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f): Feb 23, 2023 12:57:54 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Using /var/lib/jenkins/agent.log as an agent error log destination; output log will not be generated
Feb 23, 2023 12:57:54 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from centos7-builder-jekins-slave1-002uuz9748r51 (922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f): channel started
Feb 23, 2023 12:57:56 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from centos7-builder-jekins-slave1-002uuz9748r51 (922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f): WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by jenkins.slaves.StandardOutputSwapper$ChannelSwapper to constructor java.io.FileDescriptor(int)
WARNING: Please consider reporting this to the maintainers of jenkins.slaves.StandardOutputSwapper$ChannelSwapper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Feb 23, 2023 12:58:02 AM INFO hudson.slaves.NodeProvisioner update
Image of teldev/centos7-builder:latest provisioning successfully completed. We have now 6 computer(s)
Feb 23, 2023 12:58:44 AM INFO hudson.slaves.SlaveComputer tryReconnect
Attempting to reconnect LTWC Dev Server 1
Feb 23, 2023 12:59:22 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'centos7-builder-jekins-slave1-002uuz9748r51'.
Feb 23, 2023 12:59:22 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel centos7-builder-jekins-slave1-002uuz9748r51
java.net.SocketException: Socket closed
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at io.jenkins.docker.client.DockerMultiplexedInputStream.readInternal(DockerMultiplexedInputStream.java:51)
at io.jenkins.docker.client.DockerMultiplexedInputStream.read(DockerMultiplexedInputStream.java:33)
at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:94)
at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:74)
at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:105)
at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Feb 23, 2023 12:59:22 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'centos7-builder-jekins-slave1-002uuz9748r51'.
Build log here:
[2023-02-23T00:57:51.862Z] Started by user Shipper, Michael
[2023-02-23T00:57:51.887Z] [Pipeline] Start of Pipeline
[2023-02-23T00:57:51.924Z] [Pipeline] library
[2023-02-23T00:57:51.933Z] Loading library teldev_build_tools@master
[2023-02-23T00:57:51.934Z] No credentials specified
[2023-02-23T00:57:51.934Z] > /bin/git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@libs/b874e508a693557e2c5b7ccce06797e7c000ab4bef3a2f21cb534bf208455a8e/.git # timeout=10
[2023-02-23T00:57:51.941Z] Fetching changes from the remote Git repository
[2023-02-23T00:57:51.942Z] > /bin/git config remote.origin.url ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-it-jenkins-sanebuild.git # timeout=10
[2023-02-23T00:57:51.948Z] Fetching upstream changes from ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-it-jenkins-sanebuild.git
[2023-02-23T00:57:51.948Z] > /bin/git --version # timeout=10
[2023-02-23T00:57:51.955Z] > git --version # 'git version 1.8.3.1'
[2023-02-23T00:57:51.955Z] > /bin/git fetch --tags --progress ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-it-jenkins-sanebuild.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2023-02-23T00:57:52.183Z] > /bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10
[2023-02-23T00:57:52.189Z] Checking out Revision 09d77cf0b427b671ead030a77dab2cb6b03a7b70 (refs/remotes/origin/master)
[2023-02-23T00:57:52.189Z] > /bin/git config core.sparsecheckout # timeout=10
[2023-02-23T00:57:52.205Z] > /bin/git checkout -f 09d77cf0b427b671ead030a77dab2cb6b03a7b70 # timeout=10
[2023-02-23T00:57:52.212Z] Commit message: "fixed bad var name"
[2023-02-23T00:57:52.213Z] > /bin/git rev-list --no-walk 09d77cf0b427b671ead030a77dab2cb6b03a7b70 # timeout=10
[2023-02-23T00:57:53.084Z] [Pipeline] node
[2023-02-23T00:57:57.391Z] Running on centos7-builder-jekins-slave1-002uuz9748r51 on docker-uat1 in /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools
[2023-02-23T00:57:57.394Z] [Pipeline] {
[2023-02-23T00:57:57.477Z] [Pipeline] isUnix
[2023-02-23T00:57:57.483Z] [Pipeline] withEnv
[2023-02-23T00:57:57.485Z] [Pipeline] {
[2023-02-23T00:57:57.502Z] [Pipeline] sh
[2023-02-23T00:58:01.164Z] + docker inspect -f . teldev/debian-min-build
[2023-02-23T00:58:01.164Z] .
[2023-02-23T00:58:01.681Z] [Pipeline] }
[2023-02-23T00:58:01.702Z] [Pipeline] // withEnv
[2023-02-23T00:58:01.711Z] [Pipeline] withDockerContainer
[2023-02-23T00:58:02.608Z] centos7-builder-jekins-slave1-002uuz9748r51 on docker-uat1 seems to be running inside container 922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f
[2023-02-23T00:58:02.807Z] but /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools could not be found among [/var/cache/yum, /run/docker.sock, /var/cache/mock]
[2023-02-23T00:58:02.807Z] but /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@tmp could not be found among [/var/cache/yum, /run/docker.sock, /var/cache/mock]
[2023-02-23T00:58:03.287Z] $ docker run -t -d -u 798:798 --tmpfs=/var/lib/jenkins/.gnupg:rw,exec,mode=0700,size=10m,uid=798,gid=798 -w /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools -v /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools:/var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools:rw,z -v /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@tmp:/var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** teldev/debian-min-build cat
[2023-02-23T00:58:03.781Z] $ docker top 16c37bfc43f0e057515fef5cea590671fd34577211be9c3280d9fbcec7a39719 -eo pid,comm
[2023-02-23T00:58:03.983Z] [Pipeline] {
[2023-02-23T00:58:04.036Z] [Pipeline] withEnv
[2023-02-23T00:58:04.042Z] [Pipeline] {
[2023-02-23T00:58:04.071Z] [Pipeline] stage
[2023-02-23T00:58:04.076Z] [Pipeline] { (clone git branch)
[2023-02-23T00:58:04.115Z] [Pipeline] checkout
[2023-02-23T00:58:14.600Z] No credentials specified
[2023-02-23T00:58:14.600Z] Warning: JENKINS-30600: special launcher org.jenkinsci.plugins.docker.workflow.WithContainerStep$Decorator$1@20fbff43; decorates RemoteLauncher[hudson.remoting.Channel@193b6b15:centos7-builder-jekins-slave1-002uuz9748r51] will be ignored (a typical symptom is the Git executable not being run inside a designated container)
[2023-02-23T00:58:14.643Z] Cloning the remote Git repository
[2023-02-23T00:58:14.734Z] Cloning repository ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git
[2023-02-23T00:58:14.845Z] > /bin/git init /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools # timeout=10
[2023-02-23T00:58:14.981Z] Fetching upstream changes from ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git
[2023-02-23T00:58:14.981Z] > /bin/git --version # timeout=10
[2023-02-23T00:58:14.986Z] > git --version # 'git version 1.8.3.1'
[2023-02-23T00:58:14.986Z] > /bin/git fetch --tags --progress ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2023-02-23T00:58:27.602Z] > /bin/git config remote.origin.url ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git # timeout=10
[2023-02-23T00:58:27.608Z] > /bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2023-02-23T00:58:30.912Z] Avoid second fetch
[2023-02-23T00:58:30.962Z] Checking out Revision 4b93166c6d7431c35d6a2d217016b979f9b9fc10 (refs/remotes/origin/release/dev)
[2023-02-23T00:58:30.953Z] > /bin/git rev-parse refs/remotes/origin/release/dev^{commit} # timeout=10
[2023-02-23T00:58:31.047Z] > /bin/git config core.sparsecheckout # timeout=10
[2023-02-23T00:58:31.059Z] > /bin/git checkout -f 4b93166c6d7431c35d6a2d217016b979f9b9fc10 # timeout=10
[2023-02-23T00:58:32.694Z] Commit message: "rolling in priority fix"
[2023-02-23T00:58:32.738Z] > /bin/git rev-list --no-walk 4b93166c6d7431c35d6a2d217016b979f9b9fc10 # timeout=10
[2023-02-23T00:58:32.889Z] [Pipeline] }
[2023-02-23T00:58:32.912Z] [Pipeline] // stage
[2023-02-23T00:58:32.921Z] [Pipeline] stage
[2023-02-23T00:58:32.922Z] [Pipeline] { (Build)
[2023-02-23T00:58:32.960Z] [Pipeline] sh
[2023-02-23T00:59:00.301Z] Aborted by Shipper, Michael
[2023-02-23T00:59:00.344Z] Sending interrupt signal to process
[2023-02-23T00:59:15.303Z] Click here to forcibly terminate running steps
[2023-02-23T00:59:20.345Z] After 20s process did not stop
[2023-02-23T00:59:20.352Z] [Pipeline] }
[2023-02-23T00:59:20.378Z] [Pipeline] // stage
[2023-02-23T00:59:20.388Z] [Pipeline] stage
[2023-02-23T00:59:20.389Z] [Pipeline] { (install local copy of charter-rpm-tools)
[2023-02-23T00:59:20.403Z] Stage "install local copy of charter-rpm-tools" skipped due to earlier failure(s)
[2023-02-23T00:59:20.406Z] [Pipeline] }
[2023-02-23T00:59:20.437Z] [Pipeline] // stage
[2023-02-23T00:59:20.447Z] [Pipeline] stage
[2023-02-23T00:59:20.449Z] [Pipeline] { (push package)
[2023-02-23T00:59:20.463Z] Stage "push package" skipped due to earlier failure(s)
[2023-02-23T00:59:20.466Z] [Pipeline] }
[2023-02-23T00:59:20.496Z] [Pipeline] // stage
[2023-02-23T00:59:20.506Z] [Pipeline] stage
[2023-02-23T00:59:20.508Z] [Pipeline] { (Declarative: Post Actions)
[2023-02-23T00:59:20.528Z] [Pipeline] cleanWs
[2023-02-23T00:59:20.580Z] [WS-CLEANUP] Deleting project workspace...
[2023-02-23T00:59:20.580Z] [WS-CLEANUP] Deferred wipeout is used...
[2023-02-23T00:59:20.677Z] [WS-CLEANUP] done
[2023-02-23T00:59:20.682Z] [Pipeline] }
[2023-02-23T00:59:20.706Z] [Pipeline] // stage
[2023-02-23T00:59:20.711Z] [Pipeline] }
[2023-02-23T00:59:20.734Z] [Pipeline] // withEnv
[2023-02-23T00:59:20.740Z] [Pipeline] }
[2023-02-23T00:59:20.746Z] $ docker stop --time=1 16c37bfc43f0e057515fef5cea590671fd34577211be9c3280d9fbcec7a39719
[2023-02-23T00:59:22.301Z] $ docker rm -f --volumes 16c37bfc43f0e057515fef5cea590671fd34577211be9c3280d9fbcec7a39719
[2023-02-23T00:59:22.446Z] [Pipeline] // withDockerContainer
[2023-02-23T00:59:22.452Z] [Pipeline] }
[2023-02-23T00:59:22.485Z] [Pipeline] // node
[2023-02-23T00:59:22.495Z] [Pipeline] End of Pipeline
[2023-02-23T00:59:22.504Z] Finished: ABORTED
Expected Results
I expect the shell commands to run
Actual Results
No shell command is ever run.
Anything else?
If I hop into the container jenkins spawned, the docker is not running any commands.
just "cat" and "/bin/bash"
Waiting for input.
The text was updated successfully, but these errors were encountered:
Jenkins and plugins versions report
Environment
What Operating System are you using (both controller, and any agents involved in the problem)?
Centos7
Reproduction steps
Jenkins pipeline jobs fail to spawn shell commands if they were previously in a suspended state when using a docker multibranch pipeline.
Please note, this only happens when the job enters a "Suspended" state and keeps building any ways.
Not sure how to provide exact details here as I am still learning a lot about jenkins, but here is the problem.
Jobs that enter a suspended state fail to execute shell commands inside the container.
From what I can tell there is some sort of IO failure with the spawning of the container.
Jenkins logs here Logs here:
Build log here:
Expected Results
I expect the shell commands to run
Actual Results
No shell command is ever run.
Anything else?
If I hop into the container jenkins spawned, the docker is not running any commands.
just "cat" and "/bin/bash"
Waiting for input.
The text was updated successfully, but these errors were encountered: