Unable to get concensus using Ubuntu18 container build

[m-madore]

Good Morning,

In our lab, we have 4 ubuntu linux machines, each that represent an partner organization that we are attempting to model. These virtual machines are behind a load balancer that has a single IP address where we select a node by varying the port number. Inside the Linux server we run docker and the ubuntu18 container version of the hyperledger node services.

In order to get the node to bind to the container interface and allow the port to be exposed I modified the genesis file for each node to set the node and client ip address to 0.0.0.0 and the local port numbers for which the load balancer is mapped. The client appears to be connecting to all the nodes, but it appears that I'm not getting consensus.

I can't seem to locate a point in the logs that would indicate the source of the consensus issue, have I upset the configuration by modifying the endpoint data in the genesis file. How would you suggest that I reconcile what endpoints the node has to bind to locally versus what is visible to the outside world?

indy_cli.log
IndyNode1.log
IndyNode2.log
IndyNode3.log
IndyNode4.log
validator.IndyNode1.out.txt
validator.IndyNode2.out.txt
validator.IndyNode3.out.txt
validator.IndyNode4.out.txt

[m-madore]

@mgmgwi any thoughts?

[mgmgwi]

Sorry I'm currently offline, without access to my notebook. Setting 0.0.0.0 in the genesis file seems strange, but I don't get your setup. Do you have 4 machines with docker or just one? The file should contain the IPs which are reachable from each node. If you are using a loadbalancer then you should use it's IP.
I'm back at the office in about 2 weeks, then I can have a closer look.

Maybe @Echsecutor or @solidnerd can help in the meantime.

[solidnerd]

Hey @m-madore,

mapping to 0.0.0.0 means that this will be every IP Address. Hyperledger Indy Node requires a fixed set of IP addresses (this should never change) or a single one with a port variation.

So you can set the Address to the externally visible IP Adress of the LoadBalancer.

In our approach for generating the genesis file, we have a script to generate all the things that are needed

https://github.com/hyperledger/indy-node-container/blob/main/test/init-test-network.sh

All Docker nodes get a fixed IP address to be readdressable these can be seen in:

https://github.com/hyperledger/indy-node-container/blob/main/test/docker-compose.yaml

For example, you can also do the same without docker-compose

docker network create --subnet "10.133.133.0/24" --gateway "10.133.133.254" indy-network

[m-madore]

@solidnerd - our test implementation is slightly different to mimic a docker install in multiple physical locations, hence the there are four hosts with one docker each, the 10.x.x.x ip addresses are the host addresses on port 9701 and represent of an internal network within the partner organization and the client addresses and ports are on the load balancer that map individual port numbers back to the client port 9702 at each host inside the private network.

I only set the host address to 0.0.0.0 for each container for the respective node in order to control the bind on the listening port for the current node and used the docker compose file to map the docker port to the host port. From within the docker I was not able to bind directly to host address, so I cheated and bound the docker's listening port to 0.0.0.0 and relied on port parameter in docker-compose to expose the listener on the host VM.

So given this use case, I'm trying to demonstrate how to deploy 4 geographically separate nodes that are connected through an internal network with the node wrapped up in a docker deployment. I think what you are suggesting is that I create an private address range inside the docker and define routing at the hosts to direct traffic to appropriate docker instance??

[m-madore]

@solidnerd @mgmgwi any thoughts on how to have the nodes work behind a load balancer??

[mgmgwi]

@m-madore sorry for the late answer. Can you please post your config files? If you have used the init-test-network.sh script to init the nodes, then your pool_transaction_genesis file is probably wrong. You may have to change this line:

docker run -v "${PWD}"/etc_indy:/etc/indy -v "${PWD}"/lib_indy:/var/lib/indy "$IMAGE_NAME_NODE" \
    /bin/bash -c "rm -rf /var/lib/indy/* && generate_indy_pool_transactions --nodes ${NODES} --clients 0 --nodeNum $(seq -s ' ' $NODES) --ips=\"$(seq -f '10.133.133.%g' -s ',' $NODES)

Setting --ips="ENTER_YOUR_LOADBALANCER_IP,ENTER_YOUR_LOADBALANCER_IP,ENTER_YOUR_LOADBALANCER_IP,ENTER_YOUR_LOADBALANCER_IP" may fix this, although the port may be wrong too.

[Echsecutor]

Hi @m-madore !
Sorry to join this discussion quite late. Many helpful things have already been said. Let me recapitulate few points:

I modified the genesis file for each node to set the node and client ip address to 0.0.0.0 and the local port numbers for which the load balancer is mapped.

0.0.0.0 is used in our default setup ( https://github.com/hyperledger/indy-node-container/blob/main/run/.env ) in order to tell the indy nodes to listen for incoming connections targeting any destination IP. This is not a valid IP Adress where you can reach anyone, as is needed for the genesis file.

As mentioned by the others: the nodes need to be able to reach each other via the IP Adresses + ports specified in the genesis file. I.e. you need to set up your local routing such as to enable that.

I only set the host address to 0.0.0.0 for each container for the respective node in order to control the bind on the listening port for the current node and used the docker compose file to map the docker port to the host port.

Very good, that's the idea.

any thoughts on how to have the nodes work behind a load balancer??

Say you have an nginx load ballancer like e.g.

server {
    server_name YOUR_REACHABLE_HOST_NAME;
    listen 8030;

location / {
proxy_pass http://192.168.20.36:8032; 
include proxy_params;
}

}

you would put YOUR_REACHABLE_HOST_NAME and 8030 into the genesis file and map the internal indy node port to 8032 on the VM which your node balancer can reach under the local ip 192.168.20.36. the docker container should internally bind to 0.0.0.0 (for simplicity) and any port, as long as the latter is mapped to 8032 via e.g. the docker compose file.

I hope that clarifies a littlebit about the setup. Your local routing might be more or less complex, depending on your setting (node balancer on same host/VM as indy container or not, etc.)

[Echsecutor]

@m-madore I hope you could get your setup up and running? If we can provide further advice, please re-open the ticket