New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jenkins slave dind #66

Closed
t5unamie opened this Issue Apr 19, 2015 · 15 comments

Comments

Projects
None yet
8 participants
@t5unamie

t5unamie commented Apr 19, 2015

Hi There,

I have tried create a container that has dind and sshd for jenkins slave. I have tried to do this 3 different ways.

1> Create a container from the jpetazzo/dind.

{code}
Please find buildfile below.

Based on Ubuntu

Set the base image to Ubuntu

FROM jpetazzo/dind

File Author / Maintainer

MAINTAINER Johnathan Phan

Enviroment setup

############ BEGIN INSTALLATION

Update system

RUN apt-get update

Install a basic SSH and docker server

RUN apt-get install -y openjdk-7-jdk openssh-server
RUN sed -i 's|session required pam_loginuid.so|session optional pam_loginuid.so|g' /etc/pam.d/sshd
RUN mkdir -p /var/run/sshd
RUN adduser --quiet jenkins
RUN echo "jenkins:jenkins" | chpasswd
RUN usermod -a -G 0 jenkins
ADD scripts/jenkins-slave-startup.sh /
RUN chmod +x jenkins-slave-startup.sh

Standard SSH port

EXPOSE 22
{code}

2> Tried to use https://registry.hub.docker.com/u/tehranian/dind-jenkins-slave/dockerfile/ directly.

3> Tried to create my own container with build file merging commands and files from bother jpetazzo/dind and evarga/jenkins-slave.

I get continous issues concerning running docker in docker with another service.

For example in the first example where I extend dind. I get the following error inside the container.

INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
WARN[0000] WARNING: Udev sync is not supported. This will lead to unexpected behavior, data loss and errors
ERRO[0000] There are no more loopback devices available.
FATA[0000] loopback mounting failed

I don't know why this does'nt work.

Can someone help me?

@ryanwalls

This comment has been minimized.

Show comment
Hide comment
@ryanwalls

ryanwalls May 4, 2015

+1, having the same issue.

ryanwalls commented May 4, 2015

+1, having the same issue.

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Jun 15, 2015

Owner

Sorry, I hadn't seen this report earlier.

Are you still trying to use Docker-in-Docker with Jenkins?

If yes, I'd recommend to switch to a different mode: just start the Jenkins container with -v /var/run/docker.sock:/var/run/docker.sock so that it can access the Docker API from inside. That way you won't need Docker-in-Docker and everything will behave much better.

Owner

jpetazzo commented Jun 15, 2015

Sorry, I hadn't seen this report earlier.

Are you still trying to use Docker-in-Docker with Jenkins?

If yes, I'd recommend to switch to a different mode: just start the Jenkins container with -v /var/run/docker.sock:/var/run/docker.sock so that it can access the Docker API from inside. That way you won't need Docker-in-Docker and everything will behave much better.

@hayderimran7

This comment has been minimized.

Show comment
Hide comment
@hayderimran7

hayderimran7 Jun 17, 2015

Contributor

Hi , i had exact same issue, but from another docker thread, someone posted a really useful script to create a loopback device.
I add this script to my Dockerfile too, so after the container comes up with SSHD as PID 1 , and jenkins slave process (using swarm jar file) + docker daemon(which fails due to loopback mounting issue) as subsequent process run , i manually get inside the container and run the following script. After that, run the docker daemon again( remove /var/run/docker.pid if needed) and it worked :)

#!/bin/bash
ensure_loop(){
  num="$1"
  dev="/dev/loop$num"
  if test -b "$dev"; then
    echo "$dev is a usable loop device."
    return 0
  fi

  echo "Attempting to create $dev for docker ..."
  if ! mknod -m660 $dev b 7 $num; then
    echo "Failed to create $dev!" 1>&2
    return 3
  fi

  return 0
}

LOOP_A=$(losetup -f)
LOOP_A=${LOOP_A#/dev/loop}
LOOP_B=$(expr $LOOP_A + 1)

ensure_loop $LOOP_A
ensure_loop $LOOP_B
Contributor

hayderimran7 commented Jun 17, 2015

Hi , i had exact same issue, but from another docker thread, someone posted a really useful script to create a loopback device.
I add this script to my Dockerfile too, so after the container comes up with SSHD as PID 1 , and jenkins slave process (using swarm jar file) + docker daemon(which fails due to loopback mounting issue) as subsequent process run , i manually get inside the container and run the following script. After that, run the docker daemon again( remove /var/run/docker.pid if needed) and it worked :)

#!/bin/bash
ensure_loop(){
  num="$1"
  dev="/dev/loop$num"
  if test -b "$dev"; then
    echo "$dev is a usable loop device."
    return 0
  fi

  echo "Attempting to create $dev for docker ..."
  if ! mknod -m660 $dev b 7 $num; then
    echo "Failed to create $dev!" 1>&2
    return 3
  fi

  return 0
}

LOOP_A=$(losetup -f)
LOOP_A=${LOOP_A#/dev/loop}
LOOP_B=$(expr $LOOP_A + 1)

ensure_loop $LOOP_A
ensure_loop $LOOP_B
@hayderimran7

This comment has been minimized.

Show comment
Hide comment
@hayderimran7

hayderimran7 Jun 18, 2015

Contributor

@jpetazzo with your solution of -v /var/run/docker.sock:/var/run/docker.sock i get this :

----- No docker socket bind mounted in, so lets attempt to wrap it DIND style ----
time="2015-06-18T14:03:42-07:00" level="info" msg="+job serveapi(tcp://0.0.0.0:2375, unix:///var/run/docker.sock)"
time="2015-06-18T14:03:42-07:00" level="info" msg="Listening for HTTP on tcp (0.0.0.0:2375)"
time="2015-06-18T14:03:42-07:00" level="info" msg="/!\\ DON'T BIND ON ANOTHER IP ADDRESS THAN 127.0.0.1 IF YOU DON'T KNOW WHAT YOU'RE DOING /!\\"
time="2015-06-18T14:03:42-07:00" level="info" msg="Listening for HTTP on unix (/var/run/docker.sock)"
device or resource busy
time="2015-06-18T14:03:42-07:00" level="info" msg="-job serveapi(tcp://0.0.0.0:2375, unix:///var/run/docker.sock) = ERR (1)"
time="2015-06-18T14:03:42-07:00" level="fatal" msg="device or resource busy"

are you saying we should not use wrapdocker script when using jenkins and just mount the /var/run/docker.sock , not sure how to proceed with your solution ://

Contributor

hayderimran7 commented Jun 18, 2015

@jpetazzo with your solution of -v /var/run/docker.sock:/var/run/docker.sock i get this :

----- No docker socket bind mounted in, so lets attempt to wrap it DIND style ----
time="2015-06-18T14:03:42-07:00" level="info" msg="+job serveapi(tcp://0.0.0.0:2375, unix:///var/run/docker.sock)"
time="2015-06-18T14:03:42-07:00" level="info" msg="Listening for HTTP on tcp (0.0.0.0:2375)"
time="2015-06-18T14:03:42-07:00" level="info" msg="/!\\ DON'T BIND ON ANOTHER IP ADDRESS THAN 127.0.0.1 IF YOU DON'T KNOW WHAT YOU'RE DOING /!\\"
time="2015-06-18T14:03:42-07:00" level="info" msg="Listening for HTTP on unix (/var/run/docker.sock)"
device or resource busy
time="2015-06-18T14:03:42-07:00" level="info" msg="-job serveapi(tcp://0.0.0.0:2375, unix:///var/run/docker.sock) = ERR (1)"
time="2015-06-18T14:03:42-07:00" level="fatal" msg="device or resource busy"

are you saying we should not use wrapdocker script when using jenkins and just mount the /var/run/docker.sock , not sure how to proceed with your solution ://

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Jun 18, 2015

Owner

Yes, just mount the socket, and then Jenkins can talk directly talk to it,
without using wrapdocker.

On Thu, Jun 18, 2015 at 2:12 PM, Imran Hayder notifications@github.com
wrote:

@jpetazzo https://github.com/jpetazzo with your solution of -v
/var/run/docker.sock:/var/run/docker.sock i get this :

----- No docker socket bind mounted in, so lets attempt to wrap it DIND style ----
time="2015-06-18T14:03:42-07:00" level="info" msg="+job serveapi(tcp://0.0.0.0:2375, unix:///var/run/docker.sock)"
time="2015-06-18T14:03:42-07:00" level="info" msg="Listening for HTTP on tcp (0.0.0.0:2375)"
time="2015-06-18T14:03:42-07:00" level="info" msg="/!\ DON'T BIND ON ANOTHER IP ADDRESS THAN 127.0.0.1 IF YOU DON'T KNOW WHAT YOU'RE DOING /!"
time="2015-06-18T14:03:42-07:00" level="info" msg="Listening for HTTP on unix (/var/run/docker.sock)"
device or resource busy
time="2015-06-18T14:03:42-07:00" level="info" msg="-job serveapi(tcp://0.0.0.0:2375, unix:///var/run/docker.sock) = ERR (1)"
time="2015-06-18T14:03:42-07:00" level="fatal" msg="device or resource busy"

are you saying we should not use wrapdocker script when using jenkins and
just mount the /var/run/docker.sock , not sure how to proceed with your
solution ://


Reply to this email directly or view it on GitHub
#66 (comment).

@jpetazzo https://twitter.com/jpetazzo
"Is it safe to run applications into containers?"
http://www.slideshare.net/jpetazzo/docker-linux-containers-lxc-and-security

Owner

jpetazzo commented Jun 18, 2015

Yes, just mount the socket, and then Jenkins can talk directly talk to it,
without using wrapdocker.

On Thu, Jun 18, 2015 at 2:12 PM, Imran Hayder notifications@github.com
wrote:

@jpetazzo https://github.com/jpetazzo with your solution of -v
/var/run/docker.sock:/var/run/docker.sock i get this :

----- No docker socket bind mounted in, so lets attempt to wrap it DIND style ----
time="2015-06-18T14:03:42-07:00" level="info" msg="+job serveapi(tcp://0.0.0.0:2375, unix:///var/run/docker.sock)"
time="2015-06-18T14:03:42-07:00" level="info" msg="Listening for HTTP on tcp (0.0.0.0:2375)"
time="2015-06-18T14:03:42-07:00" level="info" msg="/!\ DON'T BIND ON ANOTHER IP ADDRESS THAN 127.0.0.1 IF YOU DON'T KNOW WHAT YOU'RE DOING /!"
time="2015-06-18T14:03:42-07:00" level="info" msg="Listening for HTTP on unix (/var/run/docker.sock)"
device or resource busy
time="2015-06-18T14:03:42-07:00" level="info" msg="-job serveapi(tcp://0.0.0.0:2375, unix:///var/run/docker.sock) = ERR (1)"
time="2015-06-18T14:03:42-07:00" level="fatal" msg="device or resource busy"

are you saying we should not use wrapdocker script when using jenkins and
just mount the /var/run/docker.sock , not sure how to proceed with your
solution ://


Reply to this email directly or view it on GitHub
#66 (comment).

@jpetazzo https://twitter.com/jpetazzo
"Is it safe to run applications into containers?"
http://www.slideshare.net/jpetazzo/docker-linux-containers-lxc-and-security

@dduportal

This comment has been minimized.

Show comment
Hide comment
@dduportal

dduportal Jul 2, 2015

Hello all, I have the same kind of need : running Jenkins instances (slaves at first), with docker available.

The idead I want to achieve is to use docker's lightweight isolation to "lightly" isolate docker APIs :

  • One slave has to be immutable to be dynamically started from the Jenkins master : Docker Jenkins Plugin seems perfect for that, maybe used with Jenkin's swarm plugin
  • This slave has to enable docker commands (Jenkins docker plugin build step for example)
  • I don't want a user to see what images or containers are ran by their neighbours on the Jenkins
  • I don't need a "strong" isolation. If I have a blackhat, he can jump on another docker and steal all the thing, does not matter :)

So the socket sharing is not a solution for my case, and since I don't have acces to paltform that can be managed by docker-machine, I don't know which path to start with :

  • Try to work on dind ?
  • Study new stuff like RancherOS that achieve that for me, but make me change my base OS ?
  • Use Jenkins to enforce this light isolation thru a plugin and docker namespacing ?
  • Try to write a docker-machine provider to launch a docker engine inside an existing one ?
  • Another idea ?

dduportal commented Jul 2, 2015

Hello all, I have the same kind of need : running Jenkins instances (slaves at first), with docker available.

The idead I want to achieve is to use docker's lightweight isolation to "lightly" isolate docker APIs :

  • One slave has to be immutable to be dynamically started from the Jenkins master : Docker Jenkins Plugin seems perfect for that, maybe used with Jenkin's swarm plugin
  • This slave has to enable docker commands (Jenkins docker plugin build step for example)
  • I don't want a user to see what images or containers are ran by their neighbours on the Jenkins
  • I don't need a "strong" isolation. If I have a blackhat, he can jump on another docker and steal all the thing, does not matter :)

So the socket sharing is not a solution for my case, and since I don't have acces to paltform that can be managed by docker-machine, I don't know which path to start with :

  • Try to work on dind ?
  • Study new stuff like RancherOS that achieve that for me, but make me change my base OS ?
  • Use Jenkins to enforce this light isolation thru a plugin and docker namespacing ?
  • Try to write a docker-machine provider to launch a docker engine inside an existing one ?
  • Another idea ?
@dduportal

This comment has been minimized.

Show comment
Hide comment

menski added a commit to camunda-ci/camunda-docker-dind that referenced this issue Aug 13, 2015

menski added a commit to camunda-ci/camunda-docker-dind that referenced this issue Aug 13, 2015

menski added a commit to camunda-ci/camunda-docker-dind that referenced this issue Aug 13, 2015

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Sep 3, 2015

Owner

After all that time, I finally could take a few hours to write a blog post about this:

http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

I'll close the issue here, thanks all!

Owner

jpetazzo commented Sep 3, 2015

After all that time, I finally could take a few hours to write a blog post about this:

http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

I'll close the issue here, thanks all!

@jpetazzo jpetazzo closed this Sep 3, 2015

@phillipgreenii

This comment has been minimized.

Show comment
Hide comment
@phillipgreenii

phillipgreenii Sep 14, 2015

@jpetazzo I read your article and I completely agree. Sibling containers is just fine and it keeps things much easier. With regards to your suggested solution, do you have it working? I tried making a jenkins slave using your suggestion using volumes for docker and docker.sock, but I am met with

docker: error while loading shared libraries: libdevmapper.so.1.02: cannot open shared object file: No such file or directory

Is that a artifact of my particular install, or is there more details than what was put into the article?

phillipgreenii commented Sep 14, 2015

@jpetazzo I read your article and I completely agree. Sibling containers is just fine and it keeps things much easier. With regards to your suggested solution, do you have it working? I tried making a jenkins slave using your suggestion using volumes for docker and docker.sock, but I am met with

docker: error while loading shared libraries: libdevmapper.so.1.02: cannot open shared object file: No such file or directory

Is that a artifact of my particular install, or is there more details than what was put into the article?

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Sep 16, 2015

Owner

Right, that happened when switching to dynamic binaries...

There is now an official docker:dind image upstream! I invite you to test it, since it is actively maintained. Thank you!

Owner

jpetazzo commented Sep 16, 2015

Right, that happened when switching to dynamic binaries...

There is now an official docker:dind image upstream! I invite you to test it, since it is actively maintained. Thank you!

@mattfysh

This comment has been minimized.

Show comment
Hide comment
@mattfysh

mattfysh Nov 23, 2015

@jpetazzo - thanks for the blog post! Very helpful :)
I also got the error @phillipgreenii mentioned... and fixed it by adding this to the docker run command:

-v /lib64/libdevmapper.so.1.02:/usr/lib/libdevmapper.so.1.02 \
-v /lib64/libsystemd-journal.so.0:/usr/lib/libsystemd-journal.so.0 \
-v /lib64/libsystemd-id128.so.0:/usr/lib/libsystemd-id128.so.0 \

I'm on redhat and installed docker with yum - how do I switch back to static binaries so I can remove those 3 volumes?

mattfysh commented Nov 23, 2015

@jpetazzo - thanks for the blog post! Very helpful :)
I also got the error @phillipgreenii mentioned... and fixed it by adding this to the docker run command:

-v /lib64/libdevmapper.so.1.02:/usr/lib/libdevmapper.so.1.02 \
-v /lib64/libsystemd-journal.so.0:/usr/lib/libsystemd-journal.so.0 \
-v /lib64/libsystemd-id128.so.0:/usr/lib/libsystemd-id128.so.0 \

I'm on redhat and installed docker with yum - how do I switch back to static binaries so I can remove those 3 volumes?

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Dec 2, 2015

Owner

@mattfysh, I'd have two ideas:

  1. If you're building a full-fledged container and need the Docker CLI in it, just install it with the normal package manager inside that container (yum, apt...) and it will take care of everything.
  2. If you just need a simple container with Docker in it and not much else, use the docker:dind image.

HTH!

Owner

jpetazzo commented Dec 2, 2015

@mattfysh, I'd have two ideas:

  1. If you're building a full-fledged container and need the Docker CLI in it, just install it with the normal package manager inside that container (yum, apt...) and it will take care of everything.
  2. If you just need a simple container with Docker in it and not much else, use the docker:dind image.

HTH!

@mattfysh

This comment has been minimized.

Show comment
Hide comment
@mattfysh

mattfysh Dec 3, 2015

awesome, thanks @jpetazzo - will go with option #1 :)

mattfysh commented Dec 3, 2015

awesome, thanks @jpetazzo - will go with option #1 :)

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Dec 18, 2015

I had a need to run a Jenkins CI slave w/ docker. I settled on following @jpetazzo advice here and setup a Jenkins slave container that uses docker on the CoreOS host. The trick is getting the library path correct as the host docker dynamically searches for libs at run time inside the Jenkins slave container. Note, you'll have to tweak LD_LIBRARY_PATH relative to the OS in use for your container. Also, more then likely /var/run/docker.sock on the host is docker:docker. If the container user executing docker CLI is not root be sure to create a docker user in the image with same uid:gid as host and add the non-root user to group in Dockerfile.

jenkins-slave-docker:
  build: .
  restart: always
  container_name: jenkins-slave-docker
  environment:
     - LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu:/opt/lib64
  ports:
     - 0.0.0.0:49255:22
  volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /usr/bin/docker:/usr/bin/docker
     - /usr/lib64/:/opt/lib64/

ghost commented Dec 18, 2015

I had a need to run a Jenkins CI slave w/ docker. I settled on following @jpetazzo advice here and setup a Jenkins slave container that uses docker on the CoreOS host. The trick is getting the library path correct as the host docker dynamically searches for libs at run time inside the Jenkins slave container. Note, you'll have to tweak LD_LIBRARY_PATH relative to the OS in use for your container. Also, more then likely /var/run/docker.sock on the host is docker:docker. If the container user executing docker CLI is not root be sure to create a docker user in the image with same uid:gid as host and add the non-root user to group in Dockerfile.

jenkins-slave-docker:
  build: .
  restart: always
  container_name: jenkins-slave-docker
  environment:
     - LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu:/opt/lib64
  ports:
     - 0.0.0.0:49255:22
  volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /usr/bin/docker:/usr/bin/docker
     - /usr/lib64/:/opt/lib64/
@jessesanford

This comment has been minimized.

Show comment
Hide comment
@jessesanford

jessesanford Mar 4, 2016

@ghost your suggestion worked for me.

jessesanford commented Mar 4, 2016

@ghost your suggestion worked for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment