Connecting from outside docker #5

Closed
kellabyte opened this Issue Jan 9, 2014 · 8 comments

Comments

Projects
None yet
4 participants
@kellabyte

I'm new to docker and can't seem to figure this out. Is it possible to connect externally instead of using client.sh with nodetool and with CQL client libraries to docker cluster nodes?

Thanks for the help!

@nicolasff

This comment has been minimized.

Show comment
Hide comment
@nicolasff

nicolasff Jan 10, 2014

Owner

It is possible to connect to the first node in the cluster as ports 9160 and 9042 are exposed to the docker host (the one running the containers) on localhost:19160 and localhost:19042, respectively.

Example:

$ ./start-cluster.sh 1.2.13 3
Starting node 1
Starting node 2
Starting node 3

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                                                  NAMES
12aef8d5ba27        cassandra:1.2.13    /usr/bin/start-cassa   8 seconds ago       Up 8 seconds        9042/tcp, 9160/tcp                                     focused_albattani
213061052ffd        cassandra:1.2.13    /usr/bin/start-cassa   9 seconds ago       Up 9 seconds        9042/tcp, 9160/tcp                                     hopeful_pasteur
e218d6959179        cassandra:1.2.13    /usr/bin/start-cassa   10 seconds ago      Up 10 seconds       127.0.0.1:19042->9042/tcp, 127.0.0.1:19160->9160/tcp   hopeful_shockley

$ netstat -atn | grep -E "19042|19160"
tcp        0      0 127.0.0.1:19042         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:19160         0.0.0.0:*               LISTEN

$ telnet localhost 9160
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

$ telnet localhost 19160
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]

telnet> close

$ cqlsh
Connection error: Could not connect to localhost:9160

$ cqlsh localhost 19160
Connected to Test Cluster at localhost:19160.
[cqlsh 3.1.8 | Cassandra 1.2.13 | CQL spec 3.0.0 | Thrift protocol 19.36.2]
Use HELP for help.
cqlsh>

The other nodes cannot be reached directly from outside a container – this is why client.sh is provided.

You'll notice that JMX is not exposed, since the protocol requires that you reconnect to a single-use port and I don't know how this could work with docker...

I hope this helps! If you have any suggestions of how you think it could work better, I'd be happy to hear them.

Nicolas

Owner

nicolasff commented Jan 10, 2014

It is possible to connect to the first node in the cluster as ports 9160 and 9042 are exposed to the docker host (the one running the containers) on localhost:19160 and localhost:19042, respectively.

Example:

$ ./start-cluster.sh 1.2.13 3
Starting node 1
Starting node 2
Starting node 3

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                                                  NAMES
12aef8d5ba27        cassandra:1.2.13    /usr/bin/start-cassa   8 seconds ago       Up 8 seconds        9042/tcp, 9160/tcp                                     focused_albattani
213061052ffd        cassandra:1.2.13    /usr/bin/start-cassa   9 seconds ago       Up 9 seconds        9042/tcp, 9160/tcp                                     hopeful_pasteur
e218d6959179        cassandra:1.2.13    /usr/bin/start-cassa   10 seconds ago      Up 10 seconds       127.0.0.1:19042->9042/tcp, 127.0.0.1:19160->9160/tcp   hopeful_shockley

$ netstat -atn | grep -E "19042|19160"
tcp        0      0 127.0.0.1:19042         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:19160         0.0.0.0:*               LISTEN

$ telnet localhost 9160
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

$ telnet localhost 19160
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]

telnet> close

$ cqlsh
Connection error: Could not connect to localhost:9160

$ cqlsh localhost 19160
Connected to Test Cluster at localhost:19160.
[cqlsh 3.1.8 | Cassandra 1.2.13 | CQL spec 3.0.0 | Thrift protocol 19.36.2]
Use HELP for help.
cqlsh>

The other nodes cannot be reached directly from outside a container – this is why client.sh is provided.

You'll notice that JMX is not exposed, since the protocol requires that you reconnect to a single-use port and I don't know how this could work with docker...

I hope this helps! If you have any suggestions of how you think it could work better, I'd be happy to hear them.

Nicolas

@nicolasff

This comment has been minimized.

Show comment
Hide comment
@nicolasff

nicolasff Jan 21, 2014

Owner

Hey @kellabyte,

Did this answer your question about connecting to Cassandra instances running inside of Docker containers?

Cheers,
Nicolas

Owner

nicolasff commented Jan 21, 2014

Hey @kellabyte,

Did this answer your question about connecting to Cassandra instances running inside of Docker containers?

Cheers,
Nicolas

@aspyker

This comment has been minimized.

Show comment
Hide comment
@aspyker

aspyker Jan 24, 2014

you could also do something like I did for the web app using this cassandra cluster.

http://ispyker.blogspot.com/2014/01/experiments-with-docker-for-acme-air-dev.html

I'd love to get to a point where the vagrant started on an guest only interface on a subnet and then all docker containers started on the same subnet with routing from vagrant guest to container directly. Then we'd have docker for a more full IaaS cloud without the container abstraction getting in the way (as opposed to mapping ports). I ran into other posts of people trying to do the same thing, but I'm not that good of a networking guru.

aspyker commented Jan 24, 2014

you could also do something like I did for the web app using this cassandra cluster.

http://ispyker.blogspot.com/2014/01/experiments-with-docker-for-acme-air-dev.html

I'd love to get to a point where the vagrant started on an guest only interface on a subnet and then all docker containers started on the same subnet with routing from vagrant guest to container directly. Then we'd have docker for a more full IaaS cloud without the container abstraction getting in the way (as opposed to mapping ports). I ran into other posts of people trying to do the same thing, but I'm not that good of a networking guru.

@nicolasff

This comment has been minimized.

Show comment
Hide comment
@nicolasff

nicolasff Jan 24, 2014

Owner

Hi @aspyker,

That's a great diagram! I'd also love to have a better solution for this, and would prefer Docker to handle it rather than have to set this up with pipework and dnsmasq by hand. There's also the issue of using JMX over this kind of setup, something for which I don't have any solution just yet.

I'd say that the challenges here are due to issues with the discoverability of containers. I'd love to be able to set up "groups" in Docker with clear networking rules between them, like EC2 security groups for example.

Owner

nicolasff commented Jan 24, 2014

Hi @aspyker,

That's a great diagram! I'd also love to have a better solution for this, and would prefer Docker to handle it rather than have to set this up with pipework and dnsmasq by hand. There's also the issue of using JMX over this kind of setup, something for which I don't have any solution just yet.

I'd say that the challenges here are due to issues with the discoverability of containers. I'd love to be able to set up "groups" in Docker with clear networking rules between them, like EC2 security groups for example.

@tsindot

This comment has been minimized.

Show comment
Hide comment
@tsindot

tsindot Jan 24, 2014

Contributor

First, great project.

I am trying to spin this up via vagrant and then leverage the scripts provided as part of the initial provision process to get the 1..n nodes up and running. I think I have most of it working, but do not seem to be able to connect to the cluster from the host. I followed the steps outlined above with the following results:

$ netstat -atn | grep -E "19042|19160"
tcp4 0 0 .19042 *. LISTEN
tcp4 0 0 .19160 *. LISTEN
$ telnet localhost 19160
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
t$ cqlsh localhost 19160
Connection error: TSocket read 0 bytes

Form the VM, here is what I see from a docker ps:

vagrant@ubuntu-12:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
32d7f7fb2c5a cassandra:2.0.3 /usr/bin/start-cassa About an hour ago Up 17 minutes 9042/tcp, 9160/tcp loving_wright
aa110c2d5060 cassandra:2.0.3 /usr/bin/start-cassa About an hour ago Up 17 minutes 9042/tcp, 9160/tcp sick_engelbart
c61371de7b15 cassandra:2.0.3 /usr/bin/start-cassa About an hour ago Up 17 minutes 127.0.0.1:19042->9042/tcp, 127.0.0.1:19160->9160/tcp nostalgic_turing

Any thoughts on what I may be missing?

TIA,

Contributor

tsindot commented Jan 24, 2014

First, great project.

I am trying to spin this up via vagrant and then leverage the scripts provided as part of the initial provision process to get the 1..n nodes up and running. I think I have most of it working, but do not seem to be able to connect to the cluster from the host. I followed the steps outlined above with the following results:

$ netstat -atn | grep -E "19042|19160"
tcp4 0 0 .19042 *. LISTEN
tcp4 0 0 .19160 *. LISTEN
$ telnet localhost 19160
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
t$ cqlsh localhost 19160
Connection error: TSocket read 0 bytes

Form the VM, here is what I see from a docker ps:

vagrant@ubuntu-12:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
32d7f7fb2c5a cassandra:2.0.3 /usr/bin/start-cassa About an hour ago Up 17 minutes 9042/tcp, 9160/tcp loving_wright
aa110c2d5060 cassandra:2.0.3 /usr/bin/start-cassa About an hour ago Up 17 minutes 9042/tcp, 9160/tcp sick_engelbart
c61371de7b15 cassandra:2.0.3 /usr/bin/start-cassa About an hour ago Up 17 minutes 127.0.0.1:19042->9042/tcp, 127.0.0.1:19160->9160/tcp nostalgic_turing

Any thoughts on what I may be missing?

TIA,

@nicolasff

This comment has been minimized.

Show comment
Hide comment
@nicolasff

nicolasff Jan 24, 2014

Owner

Hi @tsindot,

Can you confirm that all the shell commands you quoted are from within vagrant, and not from the VM host?
Also, could you please post here the version of Vagrant and Docker that you're using, so that I can try to reproduce this issue? As you can see above, I was able to connect with no problem, but that was from a Linux desktop.

Thanks!

Owner

nicolasff commented Jan 24, 2014

Hi @tsindot,

Can you confirm that all the shell commands you quoted are from within vagrant, and not from the VM host?
Also, could you please post here the version of Vagrant and Docker that you're using, so that I can try to reproduce this issue? As you can see above, I was able to connect with no problem, but that was from a Linux desktop.

Thanks!

@tsindot

This comment has been minimized.

Show comment
Hide comment
@tsindot

tsindot Jan 25, 2014

Contributor

Hi @nicolasff

Yes, I'm pretty sure it is something I am doing wrong especially based on your above example, the post by @aspyker showing it working in his dev environment, and the fact the I'm fairly new to docker.

So the first set are from the HOST (netstat, telnet, cqlsh) not the VM. The second set of commands, docker ps, is from within VM. My host os is Mac OSX 10.9.1.

The version are as follows:

Vagrant : $ vagrant -v
              Vagrant 1.4.3

 Docker:  vagrant@ubuntu-12:~$ docker version
              Client version: 0.7.6
              Go version (client): go1.2
              Git commit (client): bc3b2ec
              Server version: 0.7.6
              Git commit (server): bc3b2ec
              Go version (server): go1.2
              Last stable version: 0.7.6

Virtualbox: 4.3.6 r91406
                 Network: NAT
                 Port Forwarding:  19160 -> 9160

From Vagrantfile:

Vagrant::Config.run do |config|

Setup virtual machine box. This VM configuration code is always executed.

config.vm.box = BOX_NAME
config.vm.box_url = BOX_URI

#cassandra
config.vm.forward_port 7000, 17000
config.vm.forward_port 7001, 17001
config.vm.forward_port 7199, 17199
config.vm.forward_port 9160, 19160
config.vm.forward_port 9042, 19042

...
config.vm.provision "shell", inline: $script
config.vm.provision :shell, :path => 'git.sh'
config.vm.provision :shell, :path => 'docker.sh'
end

Again, I'm fairly confident it is something I am doing wrong based on your above post that shows it working fine. The git.sh, really bad name as of now, installs git and a few other packages, then clones your repo. From there the docker.sh issues the commands to create the image, and startup the docker containers, basically they just invoke your scripts, perhaps this is just wrong on my part.

Thanks again for the assistance.

Contributor

tsindot commented Jan 25, 2014

Hi @nicolasff

Yes, I'm pretty sure it is something I am doing wrong especially based on your above example, the post by @aspyker showing it working in his dev environment, and the fact the I'm fairly new to docker.

So the first set are from the HOST (netstat, telnet, cqlsh) not the VM. The second set of commands, docker ps, is from within VM. My host os is Mac OSX 10.9.1.

The version are as follows:

Vagrant : $ vagrant -v
              Vagrant 1.4.3

 Docker:  vagrant@ubuntu-12:~$ docker version
              Client version: 0.7.6
              Go version (client): go1.2
              Git commit (client): bc3b2ec
              Server version: 0.7.6
              Git commit (server): bc3b2ec
              Go version (server): go1.2
              Last stable version: 0.7.6

Virtualbox: 4.3.6 r91406
                 Network: NAT
                 Port Forwarding:  19160 -> 9160

From Vagrantfile:

Vagrant::Config.run do |config|

Setup virtual machine box. This VM configuration code is always executed.

config.vm.box = BOX_NAME
config.vm.box_url = BOX_URI

#cassandra
config.vm.forward_port 7000, 17000
config.vm.forward_port 7001, 17001
config.vm.forward_port 7199, 17199
config.vm.forward_port 9160, 19160
config.vm.forward_port 9042, 19042

...
config.vm.provision "shell", inline: $script
config.vm.provision :shell, :path => 'git.sh'
config.vm.provision :shell, :path => 'docker.sh'
end

Again, I'm fairly confident it is something I am doing wrong based on your above post that shows it working fine. The git.sh, really bad name as of now, installs git and a few other packages, then clones your repo. From there the docker.sh issues the commands to create the image, and startup the docker containers, basically they just invoke your scripts, perhaps this is just wrong on my part.

Thanks again for the assistance.

nicolasff added a commit that referenced this issue Feb 2, 2014

@nicolasff

This comment has been minimized.

Show comment
Hide comment
@nicolasff

nicolasff Feb 2, 2014

Owner

Hello all,

I've changed the way that ports are exposed on the host. Ports 9160 and 9042 are now exposed as 9160 and 9042, without the prefix "1".
To expose these ports to OSX with vagrant, add the following lines under Vagrant::Config.run do |config|:

config.vm.forward_port 9160, 9160
config.vm.forward_port 9042, 9042

You should then be able to run cqlsh from OSX to connect directly to a Cassandra instance running in docker.

Owner

nicolasff commented Feb 2, 2014

Hello all,

I've changed the way that ports are exposed on the host. Ports 9160 and 9042 are now exposed as 9160 and 9042, without the prefix "1".
To expose these ports to OSX with vagrant, add the following lines under Vagrant::Config.run do |config|:

config.vm.forward_port 9160, 9160
config.vm.forward_port 9042, 9042

You should then be able to run cqlsh from OSX to connect directly to a Cassandra instance running in docker.

@nicolasff nicolasff closed this Feb 2, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment