docs for running NSQ inside linux containers (docker)

[lgs]

I think we're actually missing NSQ on Docker continers. I mean an official Dockerfile from bitly, to have the environment replicated within docker containers.

I also post a Q. on stackoverflow for something similar ( see http://stackoverflow.com/questions/20326416/distributed-systems-nsq-topology-pattern-on-docker-continers ).

What do you say? Anyone got any experience with it?

[mreiferson]

I had replied to a similar question on the NSQ user group with a barebones example of a dockerfile for nsqd.

I agree that we should publish an official Dockerfile for nsqd, nsqlookupd, and nsqadmin...

Are you familiar with the publishing process? Are you interested in facilitating this? I can help structure them.

As far as implementing the topology that you referenced, I think it can be done. Once we publish the docker containers I can elaborate on the stackoverflow question.

[lgs]

Yes, I saw it and tried the gist. To evoid /bin/sh: 1: wget: not found error, I had to add

RUN sed 's/main$/main universe/' -i /etc/apt/sources.list
RUN apt-get update
RUN apt-get install wget -y

just before RUN wget, but then I get ERROR: cannot verify s3.amazonaws.com's certificate :

Step 6 : RUN wget https://s3.amazonaws.com/bitly-downloads/nsq/nsq-0.2.23.linux-amd64.go1.1.2.tar.gz
 ---> Running in b94905ad95ee
--2013-12-05 15:29:26--  https://s3.amazonaws.com/bitly-downloads/nsq/nsq-0.2.23.linux-amd64.go1.1.2.tar.gz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 205.251.243.186
Connecting to s3.amazonaws.com (s3.amazonaws.com)|205.251.243.186|:443... connected.
ERROR: cannot verify s3.amazonaws.com's certificate, issued by `/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3':
  Unable to locally verify the issuer's authority.
To connect to s3.amazonaws.com insecurely, use `--no-check-certificate'.
Error build: The command [/bin/sh -c wget https://s3.amazonaws.com/bitly-downloads/nsq/nsq-0.2.23.linux-amd64.go1.1.2.tar.gz] returned a non-zero code: 255

... and yes, I'd be interested in facilitating this, eventually.

[mreiferson]

Updated the gist to add the apt-get package dependencies

[mreiferson]

FYI, #279 is going to make this easier because you'll be able to import a config file on the host spinning up the docker container rather than having to perform magic to get all the command line options right.

It will still work as is, but it won't be as flexible.

[mreiferson]

Alternatively, if you look at the change I made to the gist, you can add ca-certificates to the apt-get...

[lgs]

got this now:

cp: cannot stat `nsq-0.2.23.linux-amd64/bin/nsqd': No such file or directory
Error build: The command [/bin/sh -c cp nsq-0.2.23.linux-amd64/bin/nsqd /usr/local/bin] returned a non-zero code: 255
... I gonna try `ADD` instead od `RUN cp` ... back in a min

[lgs]

nop,

it was just the wrong path for nsqd copy. The correct one is:

RUN cp /nsq-0.2.23.linux-amd64.go1.1.2/bin/nsqd /usr/local/bin

now it works:

FROM ubuntu
RUN apt-get -y install wget tar ca-certificates
RUN wget https://s3.amazonaws.com/bitly-downloads/nsq/nsq-0.2.23.linux-amd64.go1.1.2.tar.gz
RUN tar zxvf nsq-0.2.23.linux-amd64.go1.1.2.tar.gz
RUN mkdir -p /usr/local/bin
RUN cp /nsq-0.2.23.linux-amd64.go1.1.2/bin/nsqd /usr/local/bin
RUN mkdir -p /data
VOLUME ["/data"]
EXPOSE 4150 4151
CMD ["/usr/local/bin/nsqd", "--data-path=/data"]

[mreiferson]

Nice, thanks for working through that.

The last bit is going to be the broadcast address... I'm not sure how docker handles IP addresses?

[danielhfrank]

Regarding topology, there are some interesting considerations here. Note that in the topology example that you linked to, we've co-located the API service and the nsqd service on the same host. This was done in order to ensure that, as long as the API service is running, nsqd will be available to receive messages.

Now, contrast this with the Docker approach. As I understand it, Docker encourages the developer to run each service in a separate container, and be agnostic about what physical hosts these containers are run on. I might have that wrong, and I'm not exactly sure where the (relatively new) linked containers feature fits into that.

If you wanted to closely mimic the topology in the diagram, you might wish to simply swap out the diagram's "hosts" with containers. In that case, you would want to run your API process and nsqd side by side in the same container under supervisord or something - you can see an example of how to do this here. Alternatively, you could go ahead and run each service in a separate container, and make it a deployment decision to ensure that each physical host with container(s) running the api process has an nsqd container as well. This would probably allow you to better take advantage of the linked containers feature. Furthermore, we are practically only ever going to publish a Dockerfile that runs nsqd in isolation on the container, so you'll be able to use that as well 😉

Now, the role of lookupd in all of this further complicates things. I don't know enough about what a multi-host production Docker setup might look like, but ... well, as @mreiferson said, good luck with that broadcast address 😈

[lgs]

@mreiferson another issue is that it seems nsqd wasn't able to start like that. I put another change at CMD/EXPOSE to make it works:

FROM ubuntu
RUN apt-get -y install wget tar ca-certificates
RUN wget https://s3.amazonaws.com/bitly-downloads/nsq/nsq-0.2.23.linux-amd64.go1.1.2.tar.gz
RUN tar zxvf nsq-0.2.23.linux-amd64.go1.1.2.tar.gz
RUN mkdir -p /usr/local/bin
RUN cp /nsq-0.2.23.linux-amd64.go1.1.2/bin/nsqd /usr/local/bin
RUN mkdir -p /data
VOLUME ["/data"]
EXPOSE 4150
EXPOSE 4151
ENTRYPOINT ["/usr/local/bin/nsqd"]
CMD ["--data-path=/data"]

now when you fire up docker run, it start automatically on designated ports, while before just didn't :

lsoave@basenode:~/Docker/experiments$ docker run -i -t trying-nsq bash
2013/12/05 17:14:09 nsqd v0.2.23 (built w/go1.1.2)
2013/12/05 17:14:09 worker id 560
2013/12/05 17:14:09 NSQ: persisting topic/channel metadata to nsqd.560.dat
2013/12/05 17:14:09 TCP: listening on [::]:4150
2013/12/05 17:14:09 HTTP: listening on [::]:4151

[lgs]

@danielhfrank thanks for clarifying Daniel. I appreciate your pretty extensive advices and suggestions. I'm at the very beginning with Docker and NSQ too. We'll see what can come out of it.

Great project anyway,
thanks for going open source with NSQ and keep on the good job.

[mreiferson]

ok, so after reading some docs it seems like two things need to happen to properly expose the nsqd running inside the container to the outside world...

1. the docker run command you use to launch the container needs to bind the host port to the container port
2. an environment variable that specifies broadcast IP needs to be passed into the container when launching (I think this is only currently possible from the host). Also, the Dockerfile needs to be updated to read the environment variable (I've updated it here)

The run command would look something like:

docker run -p 4150:4150 -p 4151:4151 -e BROADCAST_ADDRESS=my.public.ip.domain.com <image> <cmd>

Now, the only problem left is how to handle the other port, 4151.

[mreiferson]

FWIW, I got this to work in an ubuntu VM:

## VM

Successfully built 8a85ee04b4de
mreiferson@ubuntu:~/nsq-dockerfile$ sudo docker run -p 4150:4150 -p 4151:4151 -e BROADCAST_ADDRESS=192.168.56.21 8a85ee04b4de
2013/12/05 20:42:07 nsqd v0.2.23 (built w/go1.1.2)
2013/12/05 20:42:07 worker id 905
2013/12/05 20:42:07 NSQ: persisting topic/channel metadata to /data/nsqd.905.dat
2013/12/05 20:42:07 TCP: listening on [::]:4150
2013/12/05 20:42:07 HTTP: listening on [::]:4151

## HOST

[mreiferson@AIRSNAKES ~]$ curl 'http://192.168.56.21:4151/stats'
nsqd v0.2.23 (built w/go1.1.2)

NO_TOPICS

I'll try to setup a "trusted" docker build for this, I think it's a sufficient start.

[mreiferson]

While waiting for some answers from docker on trusted builds, I've published:

https://index.docker.io/u/mreiferson/nsqd/
https://index.docker.io/u/mreiferson/nsqlookupd/

Since these support essentially identical operation to how they would run outside docker, I consider the deployment of the topology mentioned in the OP's stackoverflow link an exercise of configuration management...

[mreiferson]

updating this to reflect the need for documentation on running NSQ inside containers

cc @danielhfrank

[crosbymichael]

Do you need any help with this? Docs or working on a trusted build?

[mreiferson]

@crosbymichael I was told that there was an internal issue for re-evaluating the github permissions for trusted builds, see https://twitter.com/imsnakes/status/412379117481242624. I don't think I'm going to produce a trusted build until that changes.

As far as docs, if you're interested in contributing that would be fantastic! Are you familiar enough with NSQ? I can fill in the details obviously...

[mreiferson]

@crosbymichael has published trusted builds, I've published builds, I'm going to consider this sufficient for getting it up and running inside docker.

If anyone wants to contribute longer form documentation please feel free.