Fixed the services function so that Drone can launch containers on custom images.

[yosssi]

No description provided.

[sjhewitt]

this would be very useful! I want to run a postgres 9.3 container, and a few other services so it'd be great to be able to specify them by image name

[yosssi]

@sjhewitt Thank you very much for your comment!

[rgarcia]

👍 just found myself looking for this. Although taking a look at this PR I think I'd prefer the service config to be more expressive, maybe even a full Config object. Then you could specify alternative commands (instead of the image default), additional env variables, etc.

Use case for me is I want to start up a mongodb service, but with an additional flag (--replSet) that the default bradrydzewski/mongodb image does not have.

[bradrydzewski]

I'd like to propose some modifications to this PR

Instead of hard-coding the port(s) in the yaml file we could query the ports using the Docker api:

[{
    "id": "9f0d1136eb8581941dbab39940f8c68106123e060808a202603acf397d79c44b",
    ...
    "container_config": {
        ...
        "ExposedPorts": {
            "6379/tcp": {}
        },

This means we wouldn't need to hard-code ports in the Go code either:
https://github.com/drone/drone/blob/master/pkg/build/images.go#L8

The yaml could look like this:

services:
  - mongo           # standard mongo image
  - mongo:2.6       # standard mongo image, version 2.6
  - foo/mongo       # custom mongo image
  - foo/mongo:2.6   # custom mongo image, version 2.6

@rgarcia the startup commands for mongo are actually hard-coded into the Dockerfile. If you wanted to specify additional startup parameters (like --replSet) you would need to create your own mongo Docker image. You could then specify it in the yaml like this:

services:
  - rgarcia/mongo

[rgarcia]

@bradrydzewski even for images with entrypoint set you can customize the entrypoint arguments:

docker run -P bradrydzewski/mongodb:2.4 --noprealloc --replSet myreplset

It'd be nice to have a service format flexible enough to do this.

[bradrydzewski]

cool. didn't know that was possible!

we could do this:

services:
  - mongo
  - mongo --noprealloc --replSet myreplset

and just parse any trailing items and send to the Docker API as command args. Let's add this as a separate PR once the other changes are merged.

thanks for the feedback!

[yosssi]

@bradrydzewski Thanks for your reviewing. I checked the role of image.Ports in build package.
https://github.com/drone/drone/blob/master/pkg/build/images.go#L8

As a conclusion, I think this Ports filed is not necessary and we should remove it.

This Ports value is set to ExposedPorts of Docker's Create a container API and these ports are exposed.
http://docs.docker.io/en/latest/reference/api/docker_remote_api_v1.9/#create-a-container

But normally, exposed ports have been defined already in the Dockerfile's EXPOSE instruction like this and we don't need to set these ports when calling Docker's Create a container API.

I'll show you a few examples.

mongodb service has Ports field of []string{"27017"}. When this container launches, port 27017 is exposed, but this is because of the Dockerfile's EXPOSE instruction.

$ root@dev:~# docker ps
CONTAINER ID        IMAGE                       COMMAND                CREATED             STATUS              PORTS                        NAMES
960bcf36d862        bradrydzewski/mongodb:2.4   usr/bin/mongod --nop   8 minutes ago       Up 8 minutes        127.0.0.1:49156->27017/tcp   grave_babbage

Even if I changed this Ports field value to "", the container's exposed port was still 27017.

$ root@dev:~# docker ps
CONTAINER ID        IMAGE                       COMMAND                CREATED             STATUS              PORTS                        NAMES
960bcf36d862        bradrydzewski/mongodb:2.4   usr/bin/mongod --nop   8 minutes ago       Up 8 minutes        127.0.0.1:49156->27017/tcp   grave_babbage

When I changed this Ports field value to "27011", the container's exposed ports became 27011, 27017.

$ root@dev:~# docker ps
CONTAINER ID        IMAGE                       COMMAND                CREATED             STATUS              PORTS                                   NAMES
597d9b601686        bradrydzewski/mongodb:2.4   usr/bin/mongod --nop   2 seconds ago       Up 1 seconds        127.0.0.1:49158->27011/tcp, 27017/tcp   tender_nobel

From the above, we don't need to specify the exposed ports when calling Docker's Create a container API because the Dockerfile has already specified these ports. So, I think image.Ports in build package is not necessary and we should remove it.

What do you think?

[bradrydzewski]

we use the ports here:
https://github.com/drone/drone/blob/master/pkg/build/build.go#L449

databases are exposed to the docker container using a local IP address (not localhost). The above code will proxy the databases (or services) over localhost to the external container and IP address. This is important because it lets your write your unit tests and database tests using localhost, even though the database isn't running on localhost :)

[yosssi]

Thanks for your reply. I understood. I'll try to fix later.

[xpepper]

@yosssi I also found your pull request very useful: I merged on my drone fork (https://github.com/xpepper/drone), deployed on my drone installation and it works!

This was particularly useful for running a build on a project which needed the elasticsearch service and since the default service seems to be broken (see #89) I used the official build from dockerfile (https://index.docker.io/u/dockerfile/elasticsearch/) which (thanks to your patch which enables custom services) works great.

...
services:
  - elasticsearch dockerfile/elasticsearch 9200
...

I think this pull request is worth merging.
Maybe some tests should be added though.

[ctavan]

@xpepper I also built drone based on your current master to try out my postgres 9.3 image (see drone/images#9) and it works quite nice.

However I also think it would be even cleaner if the port would be extracted from the database container. After all we're not defining any ports for the default services neither.

So @yosssi if you find the time to fix that it would be super-awesome!

[xpepper]

@ctavan I agree with you, unfortunately I've not (still) enough skills in go to apply the fix myself, so it would be fantastic if @yosssi could improve its pull request with this more feature.

[yosssi]

@xpepper @ctavan I'm so sorry for the late response. I'll improve this PR at the beginning of next month. Please wait a little longer.

[ctavan]

Great @yosssi , looking forward to it!

[bradrydzewski]

Given the popularity of this request I went ahead and implemented the change:
https://github.com/bradrydzewski/drone/tree/custom-services

Here is the specific code if anyone is interested:
https://github.com/bradrydzewski/drone/blob/custom-services/pkg/build/build.go#L176

You can use the following yaml syntax:

services:
  - elasticsearch   # this gets converted to bradrydzewski/elasticsearch:0.9
  - johnsmith/redis # this gets converted to johnsmith/redis:latest
  - johnsmith/postgres:9.3

The mock unit tests are all passing, however, I haven't tested this against a real Docker client yet. I'll get to it some time this week. With some help testing we can expedite this.

[ctavan]

@bradrydzewski awesome, thanks a lot! I just built and deployed your branch and tested it with this .drone.yml:

image: node0.10
script:
  # workaround for cache, see https://github.com/drone/drone/issues/147
  - mkdir -p /tmp/npm
  - sudo chown -R ubuntu:ubuntu /tmp/npm
  - npm config set cache /tmp/npm
  # actual script:
  - npm test
cache:
  - /tmp/npm
services:
  - bradrydzewski/postgres:9.3

Works perfectly!

Not that I built the postgres 9.3 image from drone/images#9 as bradrydzewski/postgres:9.3 locally.

[bradrydzewski]

@ctavan thanks for testing. I just merged the change (pull request #217). If anyone finds bugs please create a new issue in the tracker. thanks!