Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
GitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
[docker] Docker: "This node is not requested endpoint" issues #9219
Since the original issue is closed, I'm re-posting this as we are re-visiting this now and still having issues: #4537 (comment)
Perhaps hazelcast can re-open the old issue or look into addressing it as it is sort of a blocker at this point (still need to try testing w/ overlay network), but regardless:
What is described below is reproducible with this sample project:
a) clone, build and create a docker image for the sample project above (see the README)
b) Edit the
c) In terminal one, launch node1 (mapped to 40001)
d) In terminal two, launch node2 (mapped to port 40002)
One both up you will see these kind of messages and each container trys to connect to the other to form the cluster
Again due to this:
The only way this will work is if the user wires is if the user specifies the public-address in the network configuration as such:
The problem with this is that in a totally dynamic environment the docker container does not know the docker host nor mapped port it is provisioned to by whatever clustering framework you are launching containers with (swarm etc) without using a tool like https://github.com/bitsofinfo/docker-discovery-registrator-consul which pre-determines this information in the java process BEFORE the HZ instance is started (and dynamically configures this public-address on the fly).... but this MUST happen before HZ starts and you have to launch your containers in a way that may not be universally acceptable/compatible for all folks use-cases of how they are launching containers.
This is also contingent on if you can set the network dynamically and if that will be obeyed if using a discovery SPI (like consul)
Seems like hazelcast should have some more flexibility in this area/scenario to maybe optionally enforce this rule that is currently throwing that exception
Hello @bitsofinfo ,
I tried this on AWS, when i use this xml config and
Putting the docker host subnet in there yields
Putting the docker
Note my members are spread across many different docker hosts and I have no idea what their ips might be, (hence using the SPI)
Either way we are able to use https://github.com/bitsofinfo/docker-discovery-registrator-consul to dynamically determine each nodes docker host ip and mapped HZ port, to collect this information and dynamically seed the
Again if this "check" below could just be a configurable option to disable it would be great. If I configure hazelcast to listen on 5701, and 5701 is mapped to a high port on the dockerhost, and I give HZ a list of valid members that live wherever (however discovered... hardwired or via SPI), it should just work.
@bitsofinfo hi. Been trying to run hazelcast on docker and your post saved the day, was finally able to understand whats happening and am now able to connect multiple hazelcast containers amongst themselves.
I agree with the point you make about dynamic ports. --net=host is just bad, it won't scale.
I ended up parameterizing the public-address and tcp member list so that you can pass them as arguments to docker run command:
For example, on my machine:
Here's my run.sh script inside Dockerfile:
This makes a 2 node cluster and both hazelcast are able to talk to each other because of public address.
This could be simplified if we were able to access the port mapping from within docker container. It seems its an in-progress feature request on docker which might take a while: moby/moby#26331
With moby's change, we would not fix a port, instead let docker use a dynamic port mapping and construct the public ip from within the container itself.
Am still at a loss on how to construct the member list though. It probably has to be done via a discovery service.