Running Jenkins in Docker and using docker-plugin for create slaves

[zlberto]

I'm running Jenkins on Docker, and i'm trying to use docker plugin to create slaves containers at the same host as jenkins master.

Since the docker plugin i can connect to the cloud using unix:///var/run/docker.sock as Docker url, when i hit in "Test connection" button, connects correctly.

The problem is when i try to run a job, the slave is created but jenkins says "All nodes labeled 'jenkins_slave' are offline", this are logs error from the slave:

[10/28/16 10:36:27] [SSH] Opening SSH connection to 0.0.0.0:32936.
Connection refused
SSH Connection failed with IOException: "Connection refused".
java.io.IOException: There was a problem while connecting to 0.0.0.0:32936
    at com.trilead.ssh2.Connection.connect(Connection.java:818)
    at com.trilead.ssh2.Connection.connect(Connection.java:687)
    at com.trilead.ssh2.Connection.connect(Connection.java:587)
    at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1185)
    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:711)
    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:706)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at com.trilead.ssh2.transport.TransportManager.establishConnection(TransportManager.java:354)
    at com.trilead.ssh2.transport.TransportManager.initialize(TransportManager.java:467)
    at com.trilead.ssh2.Connection.connect(Connection.java:758)
    ... 9 more
[10/28/16 10:36:27] Launch failed - cleaning up connection
[10/28/16 10:36:27] [SSH] Connection closed

My docker version is:

Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64

Thanks in advance for any help, i had spent more than 3 day trying to solve this.

[gamma]

Did you really map the unix:///var/run/docker.sock into the docker container running jenkins? Why not use a port instead?

does docker start the slaves at least? It looks like from the logs, otherwise it would not have the port. Which containers do you use as slave nodes? Did they exit unexpectedly (docker ps -a)?

[MansM]

Having the same issue.
socket is correctly mounted, docker does launch the slaves but it cant connect to 0.0.0.0 (like doh).
and no failed slaves.

[gamma]

Can you post some logs from Jenkins and your configuration please?

[MansM]

jenkins    | INFO: Will provision 'buildslave-centos6', for label: 'buildslave-centos6', in cloud: 'docker'
jenkins    | Nov 06, 2016 5:03:02 PM com.nirima.jenkins.plugins.docker.DockerCloud addProvisionedSlave
jenkins    | INFO: Not Provisioning 'buildslave-centos6'. Instance limit of '1' reached on server 'docker'
jenkins    | Nov 06, 2016 5:03:12 PM com.nirima.jenkins.plugins.docker.DockerCloud provision
jenkins    | INFO: Asked to provision 1 slave(s) for: buildslave-centos6
jenkins    | Nov 06, 2016 5:03:12 PM com.nirima.jenkins.plugins.docker.DockerCloud provision
jenkins    | INFO: Will provision 'buildslave-centos6', for label: 'buildslave-centos6', in cloud: 'docker'
jenkins    | Nov 06, 2016 5:03:12 PM com.nirima.jenkins.plugins.docker.DockerCloud addProvisionedSlave
jenkins    | INFO: Not Provisioning 'buildslave-centos6'. Instance limit of '1' reached on server 'docker'
jenkins    | Nov 06, 2016 5:03:22 PM com.nirima.jenkins.plugins.docker.DockerCloud provision
jenkins    | INFO: Asked to provision 1 slave(s) for: buildslave-centos6
jenkins    | Nov 06, 2016 5:03:22 PM com.nirima.jenkins.plugins.docker.DockerCloud provision
jenkins    | INFO: Will provision 'buildslave-centos6', for label: 'buildslave-centos6', in cloud: 'docker'
jenkins    | Nov 06, 2016 5:03:22 PM com.nirima.jenkins.plugins.docker.DockerCloud addProvisionedSlave
jenkins    | INFO: Not Provisioning 'buildslave-centos6'. Instance limit of '1' reached on server 'docker'
jenkins    | Nov 06, 2016 5:03:32 PM com.nirima.jenkins.plugins.docker.DockerCloud provision
jenkins    | INFO: Asked to provision 1 slave(s) for: buildslave-centos6

etc

[MansM]

<clouds>
    <com.nirima.jenkins.plugins.docker.DockerCloud plugin="docker-plugin@0.16.0">
      <name>docker</name>
      <templates>
        <com.nirima.jenkins.plugins.docker.DockerTemplate>
          <configVersion>2</configVersion>
          <labelString>buildslave-centos7</labelString>
          <launcher class="com.nirima.jenkins.plugins.docker.launcher.DockerComputerSSHLauncher">
            <sshConnector plugin="ssh-slaves@1.11">
              <port>22</port>
              <credentialsId>jenkins</credentialsId>
              <jvmOptions></jvmOptions>
              <javaPath></javaPath>
              <maxNumRetries>0</maxNumRetries>
              <retryWaitTime>0</retryWaitTime>
            </sshConnector>
          </launcher>
          <remoteFsMapping></remoteFsMapping>
          <remoteFs>/home/jenkins</remoteFs>
          <instanceCap>1</instanceCap>
          <mode>NORMAL</mode>
          <retentionStrategy class="com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy">
            <idleMinutes>10</idleMinutes>
            <idleMinutes defined-in="com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy">10</idleMinutes>
          </retentionStrategy>
          <numExecutors>1</numExecutors>
          <dockerTemplateBase>
            <image>buildslave-centos7</image>
            <dockerCommand></dockerCommand>
            <lxcConfString></lxcConfString>
            <hostname></hostname>
            <dnsHosts/>
            <volumes/>
            <volumesFrom2/>
            <environment/>
            <bindPorts></bindPorts>
            <bindAllPorts>false</bindAllPorts>
            <privileged>false</privileged>
            <tty>false</tty>
            <extraHosts class="java.util.Collections$UnmodifiableRandomAccessList" resolves-to="java.util.Collections$UnmodifiableList">
              <c class="list">
                <string>gitlab:172.17.0.2</string>
              </c>
              <list reference="../c"/>
            </extraHosts>
          </dockerTemplateBase>
          <removeVolumes>false</removeVolumes>
          <pullStrategy>PULL_NEVER</pullStrategy>
        </com.nirima.jenkins.plugins.docker.DockerTemplate>
        <com.nirima.jenkins.plugins.docker.DockerTemplate>
          <configVersion>2</configVersion>
          <labelString>buildslave-centos6</labelString>
          <launcher class="com.nirima.jenkins.plugins.docker.launcher.DockerComputerSSHLauncher">
            <sshConnector plugin="ssh-slaves@1.11">
              <port>22</port>
              <credentialsId>jenkins</credentialsId>
              <jvmOptions></jvmOptions>
              <javaPath></javaPath>
              <maxNumRetries>0</maxNumRetries>
              <retryWaitTime>0</retryWaitTime>
            </sshConnector>
          </launcher>
          <remoteFsMapping></remoteFsMapping>
          <remoteFs>/home/jenkins</remoteFs>
          <instanceCap>1</instanceCap>
          <mode>NORMAL</mode>
          <retentionStrategy class="com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy">
            <idleMinutes>10</idleMinutes>
            <idleMinutes defined-in="com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy">10</idleMinutes>
          </retentionStrategy>
          <numExecutors>1</numExecutors>
          <dockerTemplateBase>
            <image>buildslave-centos6</image>
            <dockerCommand></dockerCommand>
            <lxcConfString></lxcConfString>
            <hostname></hostname>
            <dnsHosts/>
            <volumes/>
            <volumesFrom2/>
            <environment/>
            <bindPorts></bindPorts>
            <bindAllPorts>false</bindAllPorts>
            <privileged>false</privileged>
            <tty>false</tty>
            <extraHosts class="java.util.Collections$UnmodifiableRandomAccessList" resolves-to="java.util.Collections$UnmodifiableList">
              <c class="list">
                <string>gitlab:172.17.0.2</string>
              </c>
              <list reference="../c"/>
            </extraHosts>
          </dockerTemplateBase>
          <removeVolumes>false</removeVolumes>
          <pullStrategy>PULL_LATEST</pullStrategy>
        </com.nirima.jenkins.plugins.docker.DockerTemplate>
      </templates>
      <serverUrl>unix:///var/run/docker.sock</serverUrl>
      <connectTimeout>0</connectTimeout>
      <readTimeout>0</readTimeout>
      <credentialsId></credentialsId>
      <containerCap>100</containerCap>
    </com.nirima.jenkins.plugins.docker.DockerCloud>
  </clouds>

[MansM]

I think the issue is within retrieving the ip of the container. for some reason it tries to connect to 0.0.0.0:port

[njtman]

@MansM
I am experiencing the same issue. Have you been able to find a solution?
Is what we are trying to do even possible?

[gamma]

I'm just guessing, but the 0.0.0.0:port connection issue has been solved in #454 - It has not been released yet. Is that the point in code you are referring to @MansM ?

[njtman]

@gamma I can test off master to see if this fixes the issue. How can I build the plugin?

[gamma]

I uploaded a gist with the hpi - it does not included all the newest stuff, just the ip:port issue (check my branch). We're using this hpi for our build system for some time now (Jenkins v1 and v2).

https://gist.github.com/gamma/0d6e646c0ad57abfc6b9e5d0e47464be

[njtman]

Thanks @gamma. I uploaded the plugin and I'm still seeing the same issue.

Could not connect to 0.0.0.0 port 32795. Are you sure this location is contactable from Jenkins?
Jan 17, 2017 7:54:16 PM INFO com.nirima.jenkins.plugins.docker.launcher.DockerComputerSSHLauncher getSSHLauncher
Creating slave SSH launcher for /0.0.0.0:32795

[gamma]

I see. It is a different code point. To build the master you'll need maven. Took me some time since I'm not familiar with it ... sry, I thought I could help.

[jaydp17]

any update on this? I'm also facing the same issue.
@Modomu did you get a solution?

[ndeloof]

as jenkins master run inside a container and agent in started as a sibling, master would need to connect to :sshport. Generally speaking sshd external IP detection is very hazardous.
For such a deployment I recommend using JNLP assuming jenkins rootUrl is well set and accessible from agent container.

[martinheg]

@jaydp17 @njtman
I have the same setup as described here and had the same issue, but setting the 'Docker Hostname' value in the Cloud settings (under Advanced) resolved it for me.

• Jenkins: 2.85
• docker-plugin 1.0.1

[matts19]

In my case I had specified a non-root user to run the container with. After clearing this field, ssh connection was successful. In hindsight this made sense because sshd has to run by root.