Add some troubleshooting docs?

[mtscout6]

I have been having some trouble getting this plugin to work, but I don't see any common troubleshooting documentation I can turn to to figure out what's going wrong.

I have been able to get this plugin to successfully start my containers, but they are not getting used to execute jobs. I'm not sure why and I don't know where to begin to look.

I am seeing this on the mange jenkins page:

I'm not sure which port setting that is referring to, and here are the log details.

Failed to listen to incoming slave connection

java.net.BindException: Failed to listen on port 50000 because it's already in use.
    at hudson.TcpSlaveAgentListener.<init>(TcpSlaveAgentListener.java:75)
    at jenkins.model.Jenkins.<init>(Jenkins.java:825)
    at hudson.model.Hudson.<init>(Hudson.java:82)
    at hudson.model.Hudson.<init>(Hudson.java:78)
    at hudson.WebAppMain$3.run(WebAppMain.java:222)
Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
    at hudson.TcpSlaveAgentListener.<init>(TcpSlaveAgentListener.java:73)
    ... 4 more

I am using the evarga/jenkins-slave container from docker hub, and I have the user jenkins:jenkins setup.

I am running the Jenkins server in a docker container as well using the image provided by the docker team: https://registry.hub.docker.com/_/jenkins/

I ran that container with run -d -P --name jenkins-server --volumes-from jenkins-data jenkins:1.565 where jenkins-data is a data volume I have setup for the server.

And here you can see that jenkins is spawning the containers:

[mtscout6]

I had a suspicion that it was because I started up my server with the -P command instead of explicit port mappings, but that didn't work either.

➜  jenkins-docker git:(master) ✗ docker ps -a
CONTAINER ID        IMAGE                          COMMAND                CREATED             STATUS                   PORTS                                              NAMES
272c1c881ac3        evarga/jenkins-slave:latest    "/usr/sbin/sshd -D"    3 days ago          Up 34 minutes            22/tcp                                             grave_hawking
adf4fa65093b        evarga/jenkins-slave:latest    "/usr/sbin/sshd -D"    3 days ago          Up 34 minutes            22/tcp                                             berserk_lovelace
90d73ba8e55f        evarga/jenkins-slave:latest    "/usr/sbin/sshd -D"    3 days ago          Up 34 minutes            22/tcp                                             furious_kirch
53ae331f4bfb        evarga/jenkins-slave:latest    "/usr/sbin/sshd -D"    3 days ago          Up 34 minutes            22/tcp                                             sick_curie
ad00996356d2        evarga/jenkins-slave:latest    "/usr/sbin/sshd -D"    3 days ago          Up 34 minutes            22/tcp                                             loving_einstein
3a12ee0f2a46        jenkins:1.565                  "/usr/local/bin/jenk   3 days ago          Up 36 minutes            0.0.0.0:3010->8080/tcp, 0.0.0.0:50000->50000/tcp   jenkins-server
96c074ee392c        mtscout6/jenkins-data:latest   "echo Data-only cont   3 days ago          Exited (0) 5 hours ago                                                      jenkins-data

[mtscout6]

Ok, looks like the port that's not working is 22 from the slave, from /var/jenkins_home/logs/slaves/../slave.log:

root@ab08811e61e5:/var/jenkins_home/logs/slaves/test# cat slave.log
[10/05/14 07:26:45] [SSH] Opening SSH connection to :22.
Connection refused
SSH Connection failed with IOException: "Connection refused".
java.io.IOException: There was a problem while connecting to :22
        at com.trilead.ssh2.Connection.connect(Connection.java:818)
        at com.trilead.ssh2.Connection.connect(Connection.java:687)
        at com.trilead.ssh2.Connection.connect(Connection.java:587)
        at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1203)
        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:696)
        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:691)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:346)
        at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:459)
        at com.trilead.ssh2.Connection.connect(Connection.java:758)
        ... 9 more
[10/05/14 07:26:45] Launch failed - cleaning up connection
[10/05/14 07:26:45] [SSH] Connection closed.

[mtscout6]

In my continued attempt to get this to work I ran

docker run -it --rm ubuntu:14.04

apt-get update
apt-get install ssh
ssh jenkins@172.17.0.7

So my presumption that the containers couldn't ssh into each other is out. What am I missing that is preventing Jenkins from connecting to these slaves?

[mtscout6]

I found some more logs:

Oct 09, 2014 2:44:43 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud provision
Excess workload after pending Spot instances: 1
Oct 09, 2014 2:44:44 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud countCurrentDockerSlaves
Images found: [Image{repository=null, tag=null, id=f3ac335af281c1ece22f2b76f79b62c0eb9dc75e27fb4bc6fba063135b20fbfc, created=1397576253, size=0, virtualSize=438036135}]
Oct 09, 2014 2:44:44 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud addProvisionedSlave
Provisioning for AMI evarga/jenkins-slave; Estimated number of total slaves: 1; Estimated number of slaves for ami evarga/jenkins-slave: 0
Oct 09, 2014 2:44:44 PM INFO hudson.slaves.NodeProvisioner update
Started provisioning Image of evarga/jenkins-slave from docker-1 with 1 executors. Remaining excess workload:0.0
Oct 09, 2014 2:44:44 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud provision
Excess workload after pending Spot instances: 1
Oct 09, 2014 2:44:44 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud countCurrentDockerSlaves
Images found: [Image{repository=null, tag=null, id=f3ac335af281c1ece22f2b76f79b62c0eb9dc75e27fb4bc6fba063135b20fbfc, created=1397576253, size=0, virtualSize=438036135}]
Oct 09, 2014 2:44:44 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud addProvisionedSlave
Provisioning for AMI evarga/jenkins-slave; Estimated number of total slaves: 2; Estimated number of slaves for ami evarga/jenkins-slave: 1
Oct 09, 2014 2:44:44 PM INFO hudson.slaves.NodeProvisioner update
Started provisioning Image of evarga/jenkins-slave from docker-1 with 1 executors. Remaining excess workload:0.0
Oct 09, 2014 2:44:44 PM SEVERE com.nirima.jenkins.plugins.docker.DockerCloud$1 call
Error in provisioning; slave=null, template=DockerTemplate{image=evarga/jenkins-slave, parent=DockerCloud{name=docker-1, serverUrl=http://192.168.59.103:2375}}
Oct 09, 2014 2:44:44 PM SEVERE com.nirima.jenkins.plugins.docker.DockerCloud$1 call
Error in provisioning; slave=null, template=DockerTemplate{image=evarga/jenkins-slave, parent=DockerCloud{name=docker-1, serverUrl=http://192.168.59.103:2375}}
Oct 09, 2014 2:44:53 PM WARNING hudson.slaves.NodeProvisioner update
Provisioned slave Image of evarga/jenkins-slave failed to launch
java.lang.RuntimeException: No mapped port 22 in host for SSL. Config=ContainerInspectResponse{id=6714df15a16c528b686aac97edcdca9ef73215765e0c71aafb7a86ad642ab0bd, created=2014-10-09T14:44:44.262529648Z, path=/usr/sbin/sshd, args=[Ljava.lang.String;@25ebba4b, config=ContainerConfig{hostName=6714df15a16c, portSpecs=null, user=, tty=false, stdinOpen=false, stdInOnce=false, memoryLimit=0, memorySwap=0, cpuShares=0, attachStdin=false, attachStdout=false, attachStderr=false, env=[Ljava.lang.String;@1713a560, cmd=[Ljava.lang.String;@ace4d76, dns=null, image=evarga/jenkins-slave, volumes=null, volumesFrom=, entrypoint=null, networkDisabled=false, privileged=false, workingDir=, domainName=, exposedPorts={22/tcp={}}, onBuild=null}, state=ContainerState{running=true, pid=941, exitCode=0, startedAt=2014-10-09T14:44:44.543010882Z, ghost=false, finishedAt=0001-01-01T00:00:00Z}, image=f3ac335af281c1ece22f2b76f79b62c0eb9dc75e27fb4bc6fba063135b20fbfc, networkSettings=NetworkSettings{ipAddress=172.17.0.3, ipPrefixLen=16, gateway=172.17.42.1, bridge=docker0, portMapping=null, ports=Ports{ports={}, mapping=null}}, sysInitPath=null, resolvConfPath=/mnt/sda1/var/lib/docker/containers/6714df15a16c528b686aac97edcdca9ef73215765e0c71aafb7a86ad642ab0bd/resolv.conf, volumes={}, volumesRW={}, hostnamePath=/mnt/sda1/var/lib/docker/containers/6714df15a16c528b686aac97edcdca9ef73215765e0c71aafb7a86ad642ab0bd/hostname, hostsPath=/mnt/sda1/var/lib/docker/containers/6714df15a16c528b686aac97edcdca9ef73215765e0c71aafb7a86ad642ab0bd/hosts, name=/lonely_darwin, driver=aufs}
    at com.nirima.jenkins.plugins.docker.DockerComputerLauncher.getSSHLauncher(DockerComputerLauncher.java:62)
    at com.nirima.jenkins.plugins.docker.DockerComputerLauncher.makeLauncher(DockerComputerLauncher.java:41)
    at com.nirima.jenkins.plugins.docker.DockerComputerLauncher.<init>(DockerComputerLauncher.java:37)
    at com.nirima.jenkins.plugins.docker.DockerTemplate.provision(DockerTemplate.java:206)
    at com.nirima.jenkins.plugins.docker.DockerCloud$1.call(DockerCloud.java:164)
    at com.nirima.jenkins.plugins.docker.DockerCloud$1.call(DockerCloud.java:159)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

[thomassuckow]

Failed to listen on port 50000 because it's already in use. Is unrelated to the docker-plugin. It probably only matters if you use windows slaves.

What version of the docker-plugin are you using? A bug in Docker 1.2.0 prevents docker-plugin 0.7 from working. A workaround is available in 0.8.

[mtscout6]

I'm using version 0.7 of the Docker Plugin and Docker 1.2.0. And I'm using Jenkins 1.565.2. I don't see an update for 0.8 of the plugin?

[mtscout6]

Ah I see I need jenkins updated to at least 1.580. I'll update and let you know what I see.

[mtscout6]

Works great with latest Jenkins! Thanks.