-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can not start-up the docker with the access-address #1
Comments
Thanks for reporting this issue. Looks like an enhancement to access-address setting in our latest build (version 3.3.21) affected clustering in a docker environment. We are working on a fix in the next release. A current work-around is to use release 3.3.17 After cloning the repo, you can checkout 3.3.17 :
|
I tried 3.3.17 again, it can start success. However, it seems not aware of the other node in the cluster. In the amc dashboard, I can see only 1 node in the cluster.
on node 10.10.99.128
|
Would it be possible for you to post your config file? Also do you have a container per host sever? Meaning do you have a different access-address for each of the containers? Also verify that the two container servers can communicate over port 3002. You could try telnet from one host to the other to verify connectivity.. Also where you able to use the tip command. asinfo -h Node_IP_ADDR -v 'tip:host=Docker_host_ADDR;port=3002' |
config on 10.10.99.128 is:
config on 10.10.99.129 is the same except the access-address is setting to 10.10.99.129. I am running a container on each host server and I am using the docker "host" network mode. So the IP is just the docker host machine IP. |
Hi Vincent, Issue may be with your mesh config. For Aerospike server version 3.3.17 try using:
When using Please see following link for info on heartbeat configuration: in version greater then Aerospike version 3.3.19 a new setting was added to support multiple seed servers. When using mesh-address you do not need to specify the port. best, Lucien |
Hi Vincent, I was able to bring up a Aerospike 3.3.17 mesh cluster on EC2 by running the following:
My host IP address was : 172.31.12.80 Here is the config file I used:
And ifconfig of host server:
Running docker with Cluster setup can be verified from command line by installing aerospike-tools on one of your hosts:
We have a fix that will soon be released for the access-address restriction for NAT ip addresses issue affecting version 3.3.21. More info to come soon. Hope this helps. best, |
Still no luck. The mesh-address config is OK and I can see the establishing on the 3002 port by using the tcpdump.
|
Could you send us your exact command for running docker? Is this running on AWS or are you using bare metal? Also please double check the IP address of the container by running:
The error you sent would be the sign of IP address changing within the container which should not be the case if using --Lucien |
I am running on my own server machines and they are in the same subset. (which mean only 1 hop.)
docker inspect showed:
Since the network mode is host, I think it is normal the network is missing. |
and the starting log on 10.10.99.128:
|
starting on 10.10.99.129:
|
By the way, I can set the elasticsearch cluster in the docker success on the same machine and the elasticsearch is using the unicast which I think should be work the same as aerospike. |
There seems to still be something still preventing communication from the containers. Nov 25 2014 10:59:24 GMT: INFO (info): (thr_info.c::4473) heartbeat_received: self 186 : foreign 0 heartbeat received shows 0 foreign heartbeat. Any firewall enabled on these hosts? or routing rules affecting containers? Please see section on Firewalld: Could you also provide your docker run command line ? and also try running the "tip" command to inform a node of the other:
I have successfully tested the config on EC2, ubuntu and centos7 with latest docker. |
firewall has been disabled on both machines and the docker daemon has already been restarted after the firewalld stopped.
the iptables --list showed on 128 is:
on 129 is:
ifconfig on 128 is:
on 129 is:
docker run command is:
I tried the tip command and seems no help. From logs It indeed seems the heartbeat has some problem. However, I can not find out why this happen. |
Could you provide your Dockerfile and your aerospike.conf file so I can try to replicate. Thanks Lucien |
Vincent, One more change on centos7 you may need to specify the interface name.
--Lucien |
Oh, it is working like a charm after adding
in the configure. But why would this matter? Or could it auto check this by comparing with the access-address? Anyway, Thank you very much for the greate help ! |
Happy to be of service. This is a bug that I'm going to follow up on. best, |
Ok, got it. I'd like to do the test again after the fixing. |
We'll update once fixed. best, |
To allow the access from outside and allow the server in docker on different hosts to communicate with each other, I set the access-address to the Docker host ip. However, it failed to start. The config and logs as below:
,
the failed log is:
Nov 18 2014 08:38:09 GMT: INFO (as): (as.c::408) initializing services...
Nov 18 2014 08:38:09 GMT: INFO (drv_ssd): (drv_ssd.c::1666) ns test starting write worker threads
Nov 18 2014 08:38:09 GMT: INFO (tsvc): (thr_tsvc.c::998) shared queues: 4 queues with 4 threads each
Nov 18 2014 08:38:09 GMT: INFO (drv_ssd): (drv_ssd.c::811) ns test starting defrag threads
Nov 18 2014 08:38:09 GMT: INFO (hb): (hb.c::2434) heartbeat socket initialization
Nov 18 2014 08:38:09 GMT: INFO (hb): (hb.c::2448) initializing mesh heartbeat socket : 0.0.0.0:3002
Nov 18 2014 08:38:09 GMT: INFO (info): (thr_info.c::4940) static external network definition
Nov 18 2014 08:38:09 GMT: CRITICAL (info): (thr_info.c:info_interfaces_static_fn:4954) external address:10.10.99.128 is not matching with any of service addresses:172.17.0.7:3000
Nov 18 2014 08:38:09 GMT: WARNING (as): (signal.c::70) SIGABRT received, aborting Aerospike Community Edition build 3.3.21
The text was updated successfully, but these errors were encountered: